JP5901780B2 - Image processing apparatus, imaging apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and an image processing program.

  In addition to autofocus using a phase difference detection method and a contrast detection method, digital cameras having a so-called manual focus mode in which a user can manually perform focus adjustment are widely known.

  As a digital camera having a manual focus mode, a method using a split micro prism screen that displays a phase difference visually by providing a reflex mirror so that focus adjustment can be performed while checking a subject is known. ing. Also known is a method employing a method for visually confirming contrast.

  By the way, in a digital camera that omits a reflex mirror that has been widely used in recent years, there is no method for checking a subject image while displaying a phase difference because there is no reflex mirror, and it has been necessary to rely on a contrast detection method. However, in this case, it is impossible to display a contrast higher than the resolution of a display device such as a (liquid crystal display), and it has been necessary to take a method of displaying it by partially enlarging it.

  Therefore, in recent years, a split image is displayed in a live view image (also referred to as a through image) in order to make it easier for the operator to focus on the subject in the manual focus mode. Here, the split image is, for example, a divided image divided into two (for example, images divided in the vertical direction), and is shifted in the parallax generation direction (for example, the horizontal direction) according to the shift of the focus, and is in focus. In this state, it indicates a divided image in which the shift in the parallax generation direction is eliminated. An operator (for example, a photographer) adjusts the focus by operating the manual focus ring so that the split image (for example, each image divided in the vertical direction) is not displaced.

  An imaging apparatus described in Japanese Patent Laying-Open No. 2009-147665 (hereinafter referred to as Patent Document 1) includes a first subject image and a first object image formed by a light beam divided by a pupil division unit among light beams from an imaging optical system. A first image and a second image are generated by photoelectrically converting the two subject images. Then, a split image is generated using the first and second images, and a third image is generated by photoelectric conversion of a third subject image formed by a light beam that is not divided by the pupil dividing unit. The third image is displayed on the display unit, the generated split image is displayed in the third image, and the color information extracted from the third image is added to the split image. In this way, by adding the color information extracted from the third image to the split image, the visibility of the split image can be improved.

  An image processing apparatus described in Japanese Patent Application Laid-Open No. 2000-078607 (hereinafter referred to as Patent Document 2) includes color balance adjustment processing means for performing color balance adjustment on an image signal input from the image input apparatus. The color balance adjustment processing unit estimates the chromaticity of the illumination light source when the image signal is acquired from the image signal, and performs color balance adjustment on the image signal based on the estimated chromaticity of the illumination light source.

  However, in the imaging device described in Patent Document 1, the color characteristics of the display color of the third image and the color characteristics of the display color of the split image may be approximated. In this case, there is a problem that it becomes difficult to visually recognize the boundary line between the third image and the split image.

  Further, even if the color balance adjustment is performed on the split image or an image other than the split image (for example, the background image) using the image processing device described in Patent Document 2, the image other than the split image, the split image, There is a problem that it becomes difficult to visually recognize the boundary line.

  The present invention has been proposed in view of such a situation, and can improve the visibility of the boundary line between the first display image and the second display image used for focusing confirmation. An object is to provide an image processing apparatus, an imaging apparatus, an image processing method, and an image processing program.

An image processing apparatus according to a first aspect of the present invention includes an imaging device including first and second pixel groups in which a subject image that has passed through first and second regions in a photographic lens is divided into pupils, respectively. the first display image generating unit, the first image and the second image output from the first and second pixel group for generating a first display image based on the image output from the device A second display image generation unit that generates a second display image to be used for focus confirmation based on the first display image, and a color of the first display image generated by the first display image generation unit An acquisition unit that acquires information, and a color having a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition unit is a display color of the second display image A determination unit, a display unit for displaying an image, and the display unit The first display image generated by the first display image generation unit is displayed, and is generated by the second display image generation unit within a display area of the first display image. And a display control unit for displaying the second display image. Thereby, the visibility of the boundary line between the first display image and the second display image used for focusing confirmation can be improved as compared with the case where this configuration is not provided.

  According to a second aspect of the present invention, in the first aspect of the present invention, it is possible to visually recognize that the color having the different color characteristic is a color characteristic different from the color characteristic of the first display image. It is good also as what. Accordingly, the first display image and the second display image can be visually identified as compared with the case where the present configuration is not provided.

  According to a third aspect of the present invention, in the first aspect or the second aspect of the present invention, the acquisition unit may acquire the color information from a part or all of the first display image. Good. Thereby, compared with the case where this configuration is not provided, the color characteristics of the second display image and the color characteristics of the first display image can be differentiated with high accuracy with a simple configuration.

  According to a fourth aspect of the present invention, in the first aspect or the second aspect of the present invention, the acquisition unit includes the first display image, the first image, the second image, or the first image. The color information may be acquired based on two display images. As a result, the color characteristics of the second display image and the color characteristics of the first display image can be made to differ with high accuracy compared to the case without this configuration. According to the fourth aspect of the present invention, even if the first display image is overexposed, the first and second pixel groups have low sensitivity, so that color information can be acquired.

  According to a fifth aspect of the present invention, in any one of the first aspect to the fourth aspect of the present invention, the imaging element includes a normal imaging region used for imaging the first display image. An in-focus confirmation imaging area that is used for imaging the second display image and is adjacent to the normal imaging area, and the acquisition unit includes the first display image in the normal imaging area. The color information may be acquired from an image output from an edge portion on the boundary side with the focus confirmation imaging region. Accordingly, the boundary between the first display image and the second display image can be made clearer than in the case where the present configuration is not provided.

  According to a sixth aspect of the present invention, in any one of the first to fifth aspects of the present invention, the color information indicates color information indicating a color characteristic of an object color of the first display image. It is good also as what. As a result, the boundary between the first display image and the second display image is obtained even when the object color is more dominant than the light source color in the first display image as compared with the case without this configuration. The visibility of the line can be improved.

  According to a seventh aspect of the present invention, in the sixth aspect of the present invention, the color characteristic may be a hue. Thereby, the visibility of the boundary line between the first display image and the second display image can be further improved with a simple configuration as compared with the case where the present configuration is not provided.

  According to an eighth aspect of the present invention, in the sixth aspect or the seventh aspect of the present invention, the color having the different color characteristic is predetermined as an object color of the first display image in a hue circle. It is good also as what made the color away from the angle. As a result, an object color suitable as a color having a color characteristic different from the color characteristic of the first display image can be easily determined as compared with the case without this configuration.

  According to a ninth aspect of the present invention, in any one of the sixth to eighth aspects of the present invention, the color characteristic of the object color of the first display image is the first display image. The composition ratio of the primary color components may be used. As a result, the visibility of the boundary line between the first display image and the second display image is improved as the color characteristic of the object color of the first display image, compared with the case where the present configuration is not provided. It is possible to easily identify appropriate color characteristics.

  According to a tenth aspect of the present invention, in the ninth aspect of the present invention, the converted composition ratio obtained by converting the composition ratio by an input / output function including a decreasing function is the different color characteristic. Also good. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved with a simple configuration as compared with the case where the present configuration is not provided.

  In an eleventh aspect of the present invention, in the ninth aspect of the present invention, the converted composition ratio obtained by converting the composition ratio by color conversion data may be the different color characteristics. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved with a simple configuration as compared with the case where the present configuration is not provided.

  According to a twelfth aspect of the present invention, in the eleventh aspect of the present invention, the color conversion data may be color conversion data created based on an input / output function including a reduction function. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved with a simple configuration as compared with the case where the present configuration is not provided.

  According to a thirteenth aspect of the present invention, in any one of the first to fifth aspects of the present invention, the color information is color information indicating a color characteristic of a light source color of the first display image. It is good also as what. As a result, the boundary between the first display image and the second display image is obtained even when the light source color is more dominant than the object color in the first display image as compared with the case without this configuration. The visibility of the line can be improved.

According to a fourteenth aspect of the present invention, in any one of the first to thirteenth aspects of the present invention, each of the first and second pixel groups is a single color pixel group, and the first display image generating section 2 is, on the basis of the first and second said output from the pixel group of the first image and the second image, the achromatic image as the second display image It may be generated. Thereby, it is possible to obtain an achromatic second display image with a simple configuration as compared with the case where this configuration is not provided.

  According to a fifteenth aspect of the present invention, in the fourteenth aspect of the present invention, in the case where the second display image is an achromatic image, the determination unit is based on the color information acquired by the acquisition unit. A chromatic color having a color characteristic different from the color characteristic of the first display image may be determined as the display color of the second display image. Thereby, it is possible to determine the chromatic color as the display color of the second display image with a simple configuration as compared with the case where this configuration is not provided.

  According to a sixteenth aspect of the present invention, in any one of the first to fifteenth aspects of the present invention, the determination unit is configured such that the first display image and the second display image are achromatic. In the case of an image, based on the color information acquired by the acquisition unit, a chromatic color having a color characteristic different from the color characteristic of the first display image is determined as the display color of the second display image. It may be a thing. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved as compared with the case where this configuration is not provided.

  According to a seventeenth aspect of the present invention, in any one of the first to sixteenth aspects of the present invention, the determining unit is configured to perform the first based on the color information acquired by the acquiring unit. A chromatic color having a color characteristic different from the color characteristic of the display image, wherein the chromatic color is different between the region based on the first image and the region based on the second image in the second display image. The display color of the second display image may be determined. This makes it easier to visually identify the image based on the first image and the image based on the second image in the second display image as compared with the case without this configuration.

  According to an eighteenth aspect of the present invention, in any one of the first to seventeenth aspects of the present invention, the determination unit is configured to display the color characteristics of the first display image and the second display image. When a predetermined condition for approaching the color characteristic of the image is satisfied, a color that approaches the color characteristic of the first display image by a predetermined degree based on the color information acquired by the acquisition unit A color having characteristics may be further determined as the display color of the second display image. As a result, it is possible to make it difficult to visually recognize the boundary between the first display image and the second display image as compared with the case where the present configuration is not provided.

Nineteenth aspect of the present invention, in any one of the eighteenth aspect of the first aspect of the present invention, the image pickup element, the third image are imaged object image without being pupil split The first display image generation unit generates the first display image based on the third image output from the third pixel group. It is also good. Thereby, compared with the case where it does not have this structure, the image quality of the 1st image for a display can be improved.

  An image pickup apparatus according to a twentieth aspect of the present invention includes an image processing apparatus according to any one of the first to nineteenth aspects of the present invention, a storage unit that stores an image output from the image sensor, including. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved as compared with the case where this configuration is not provided.

An image processing method according to a twenty-first aspect of the present invention is an image processing method including first and second pixel groups in which a subject image that has passed through first and second regions in a photographic lens is divided into pupils and formed. the first display image generating step, the first image and the second image output from the first and second pixel group for generating a first display image based on the image output from the device the second display image generating step of, said first acquisition step of acquiring color information of the generated by the display image generation process first display image to generate a second display image based on the And a determination step of determining a color having a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition step as a display color of the second display image. , For the first display with respect to the display unit for displaying an image The first display image generated by the image generation step is displayed, and the second display generated by the second display image generation step is displayed in the display area of the first display image. And a display control step for displaying an image for use. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved as compared with the case where this configuration is not provided.

  In order to achieve the above object, a program according to a twenty-second aspect of the present invention provides the first display image generating unit in the image processing apparatus according to any one of the first to nineteenth aspects of the present invention. The computer functions as the second display image generation unit, the acquisition unit, the determination unit, and the display control unit. Thereby, the visibility of the boundary line between the first display image and the second display image can be improved as compared with the case where this configuration is not provided.

  According to the present invention, it is possible to improve the visibility of the boundary line between the first display image and the second display image.

It is a perspective view which shows an example of the external appearance of the imaging device which is a lens interchangeable camera which concerns on 1st Embodiment. It is a rear view which shows the back side of the imaging device shown in FIG. It is a block diagram which shows an example of a structure of the electric system of the imaging device which concerns on 1st Embodiment. FIG. 2 is a schematic layout diagram illustrating an example of a layout of color filters provided in an image sensor included in the imaging device according to the first embodiment. FIG. 5 is a diagram for explaining a method of determining a correlation direction from pixel values of 2 × 2 G pixels included in the color filter illustrated in FIG. 4. It is a figure for demonstrating the concept of a basic sequence pattern. It is a figure for demonstrating the concept of the basic sequence pattern contained in the color filter shown in FIG. FIG. 3 is a schematic arrangement diagram illustrating an example of an arrangement of light shielding members provided in an imaging element included in the imaging apparatus according to the first embodiment. It is a schematic block diagram which shows an example of a structure of the phase difference pixel (1st pixel and 2nd pixel) contained in the image pick-up element of the imaging device which concerns on 1st Embodiment. It is a block diagram which shows an example of the principal part function of the imaging device which concerns on 1st Embodiment. It is a schematic diagram which shows an example of each display area of the normal image and split image in the display apparatus of the imaging device which concerns on 1st Embodiment. It is a screen mode figure which shows an example of the normal image and split image in the live view image displayed on the display apparatus as a comparative example with respect to 1st Embodiment. It is a graph which shows an example of the 1st input / output function used with the imaging device concerning a 1st embodiment. It is a graph which shows an example of the 2nd input / output function used with the imaging device concerning a 1st embodiment. 4 is a flowchart illustrating an example of a flow of image output processing performed by an image processing unit included in the imaging apparatus according to the first embodiment. It is a flowchart which shows an example of the flow of WB processing performed by the image process part contained in the imaging device which concerns on 1st Embodiment. It is a flowchart which shows an example of the flow of the chromatic color provision process performed by the image process part contained in the imaging device which concerns on 1st Embodiment. It is a flowchart which shows an example of the flow of the display color differentiation process performed by the image process part contained in the imaging device which concerns on 1st Embodiment. It is a flowchart which shows an example of the flow of the display color assimilation process performed by the image process part contained in the imaging device which concerns on 1st Embodiment. It is a screen mode figure which shows an example of the normal image and split image which are the live view images displayed on the display apparatus of the imaging device in 1st Embodiment, and a display color differentiation process is performed. FIG. 3 is a screen mode diagram illustrating an example of a normal image and a color-divided split image when a display color differentiation process is performed on a live view image displayed on the display device of the imaging device according to the first embodiment. It is a screen mode figure which shows an example of the achromatic normal image and split image which are the live view images displayed on the display apparatus of the imaging device according to the first embodiment. FIG. 3 is a screen view illustrating an example of an achromatic normal image and a chromatic split image that are live view images displayed on the display device of the imaging apparatus according to the first embodiment. It is a screen mode figure which shows an example of the normal image and split image which are the live view images displayed on the display apparatus of the imaging device in 1st Embodiment, and the color characteristic is approximated. It is a perspective view which shows an example of the external appearance of the smart phone which concerns on 2nd Embodiment. It is a block diagram which shows an example of the principal part structure of the electrical system of the smart phone which concerns on 2nd Embodiment.

  Hereinafter, an example of an embodiment of an imaging apparatus according to the present invention will be described with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a perspective view illustrating an example of an appearance of the imaging apparatus 100 according to the first embodiment, and FIG. 2 is a rear view of the imaging apparatus 100 illustrated in FIG.

  The imaging apparatus 100 is an interchangeable lens camera, and includes a camera body 200 and an interchangeable lens 300 (photographing lens, focus lens 302 (manual operation unit)) that is attached to the camera body 200 in an interchangeable manner. This is a digital camera with a mirror omitted. The camera body 200 is provided with a hybrid finder (registered trademark) 220. The hybrid viewfinder 220 here refers to a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as “OVF”) and an electronic viewfinder (hereinafter referred to as “EVF”) are selectively used.

  The camera body 200 and the interchangeable lens 300 are interchangeably mounted by combining a mount 256 provided in the camera body 200 and a mount 346 (see FIG. 3) on the interchangeable lens 300 side corresponding to the mount 256. The

  An OVF viewfinder window 241 included in the hybrid viewfinder 220 is provided on the front surface of the camera body 200. A finder switching lever (finder switching unit) 214 is provided on the front surface of the camera body 200. When the viewfinder switching lever 214 is rotated in the direction of the arrow SW, it switches between an optical image that can be viewed with OVF and an electronic image (live view image) that can be viewed with EVF (described later). The optical axis L2 of the OVF is an optical axis different from the optical axis L1 of the interchangeable lens 300. On the upper surface of the camera body 200, a release button 211 and a dial 212 for setting a shooting mode, a playback mode, and the like are mainly provided.

  An OVF viewfinder eyepiece 242, a display unit 213, a cross key 222, a MENU / OK key 224, and a BACK / DISP button 225 are provided on the rear surface of the camera body 200.

  The cross key 222 functions as a multifunction key for outputting various command signals such as menu selection, zoom, frame advance, and the like. The MENU / OK key 224 has both a function as a menu button for instructing to display a menu on the screen of the display unit 213 and an function as an OK button for instructing confirmation and execution of selection contents. Key. The BACK / DISP button 225 is used for deleting a desired object such as a selection item, canceling a designated content, or returning to the previous operation state.

  The display unit 213 is realized by, for example, an LCD, and is used to display a live view image (through image) that is an example of a continuous frame image obtained by capturing images in a continuous frame in the shooting mode. The display unit 213 is also used to display a still image that is an example of a single frame image obtained by capturing a single frame when a still image shooting instruction is given. Furthermore, the display unit 213 is also used for displaying a playback image and a menu screen in the playback mode.

  FIG. 3 is a block diagram illustrating an example of the configuration (internal configuration) of the electrical system of the imaging apparatus 100 according to the first embodiment.

  The imaging device 100 is a digital camera that records captured still images and moving images, and the overall operation of the camera is controlled by a CPU (central processing unit) 12. In addition to the CPU 12, the imaging apparatus 100 includes an operation unit 14, an interface unit 24, a memory 26, an encoder 34, a display control unit 36, and an eyepiece detection unit 37. In addition, the imaging apparatus 100 includes an image processing unit 28 that is an example of a first display image generation unit, a second display image generation unit, an acquisition unit, a determination unit, and an estimation unit according to the present invention. The CPU 12, the operation unit 14, the interface unit 24, the memory 26 as an example of a storage unit, the image processing unit 28, the encoder 34, the display control unit 36, the eyepiece detection unit 37, and the external interface (I / F) 39 are connected to the bus 40. Are connected to each other. The memory 26 includes a non-volatile storage area (such as an EEPROM) that stores parameters, programs, and the like, and a volatile storage area (such as an SDRAM) that temporarily stores various information such as images. Have

  The operation unit 14 includes a release button 211, a dial (focus mode switching unit) 212 for selecting a shooting mode, a display unit 213, a viewfinder switching lever 214, a cross key 222, a MENU / OK key 224, and a BACK / DISP button 225. . The operation unit 14 also includes a touch panel that accepts various types of information. This touch panel is overlaid on the display screen of the display unit 213, for example. Various operation signals output from the operation unit 14 are input to the CPU 12.

  When the photographing mode is set, the image light indicating the subject is coupled to the light receiving surface of the color image sensor (for example, a CMOS sensor) 20 via the photographing lens 16 including the focus lens that can be moved manually and the shutter 18. Imaged. The signal charge accumulated in the image sensor 20 is sequentially read out as a digital signal corresponding to the signal charge (voltage) by a read signal applied from the device control unit 22. The imaging element 20 has a so-called electronic shutter function, and controls the charge accumulation time (shutter speed) of each photosensor according to the timing of the readout signal by using the electronic shutter function. The image sensor 20 according to the first embodiment is a CMOS image sensor, but is not limited thereto, and may be a CCD image sensor.

  FIG. 4 schematically shows an example of the arrangement of the color filters 21 provided in the image sensor 20. In the example shown in FIG. 4, (4896 × 3264) pixels are adopted as an example of the number of pixels and 3: 2 is adopted as the aspect ratio. However, the number of pixels and the aspect ratio are not limited thereto. . As an example, as illustrated in FIG. 4, the color filter 21 includes a first filter G corresponding to G (green) that contributes most to obtain a luminance signal, and second filters R and B corresponding to R (red). A third filter B corresponding to (blue) is included. The arrangement pattern of the first filter G (hereinafter referred to as G filter), the second filter R (hereinafter referred to as R filter), and the third filter B (hereinafter referred to as B filter) is the first arrangement. It is classified into a pattern A and a second array pattern B.

  In the first arrangement pattern A, the G filters are arranged on the four corners and the center pixel of the 3 × 3 pixel square arrangement. In the first arrangement pattern A, the R filter is arranged on the central vertical line in the horizontal direction of the square arrangement. In the first arrangement pattern A, the B filter is arranged on the central horizontal line in the vertical direction of the square arrangement. The second arrangement pattern B is a pattern in which the arrangement of the filter G is the same as that of the first basic arrangement pattern A and the arrangement of the filter R and the arrangement of the filter B are interchanged. The color filter 21 includes a basic array pattern C composed of a square array pattern corresponding to 6 × 6 pixels. The basic array pattern C is a 6 × 6 pixel pattern in which the first array pattern A and the second array pattern B are arranged point-symmetrically, and the basic array pattern C is repeatedly arranged in the horizontal direction and the vertical direction. ing. In other words, in the color filter 21, filters of R, G, and B colors (R filter, G filter, and B filter) are arranged with a predetermined periodicity. Therefore, when performing synchronization (interpolation) processing of R, G, and B signals read from the color image sensor, processing can be performed according to a repetitive pattern.

  When the image is reduced by thinning out in units of the basic arrangement pattern C, the color filter array of the reduced image after the thinning process can be the same as the color filter array before the thinning process, and a common processing circuit is provided. Can be used.

  The color filter 21 has a G filter corresponding to the color that contributes most to obtain a luminance signal (G color in the first embodiment) in each horizontal, vertical, and diagonal line of the color filter array. Has been placed. Therefore, it is possible to improve the reproduction accuracy of the synchronization process in the high frequency region regardless of the direction of high frequency.

  The color filter 21 includes an R filter and a B filter corresponding to two or more other colors (R and B colors in the first embodiment) other than the G color. Located within each line of direction. For this reason, the occurrence of color moire (false color) is suppressed, whereby an optical low-pass filter for suppressing the occurrence of false color can be prevented from being arranged in the optical path from the incident surface of the optical system to the imaging surface. . Even when an optical low-pass filter is applied, it is possible to apply a filter having a weak function of cutting a high-frequency component for preventing the occurrence of false colors, so that the resolution is not impaired.

  The basic array pattern C includes a 3 × 3 pixel first array pattern A surrounded by a broken line frame and a 3 × 3 pixel second array pattern B surrounded by a one-dot chain line. It can also be understood that the array is arranged alternately.

  In the first array pattern A and the second array pattern B, G filters, which are luminance system pixels, are arranged at the four corners and the center, and are arranged on both diagonal lines. In the first arrangement pattern A, the B filter is arranged in the horizontal direction and the R filter is arranged in the vertical direction with the central G filter interposed therebetween. On the other hand, in the second arrangement pattern B, the R filters are arranged in the horizontal direction and the B filters are arranged in the vertical direction across the center G filter. That is, in the first arrangement pattern A and the second arrangement pattern B, the positional relationship between the R filter and the B filter is reversed, but the other arrangements are the same.

  The G filters at the four corners of the first array pattern A and the second array pattern B are shown in FIG. 5 as an example, in which the first array pattern A and the second array pattern B are horizontal and vertical. Are alternately arranged, a square array of G filters corresponding to 2 × 2 pixels is formed. As an example, with respect to 2 × 2 pixels composed of G filters extracted as shown in FIG. 5, the difference absolute value of the pixel value of the G pixel in the horizontal direction, the difference absolute value of the pixel value of the G pixel in the vertical direction, and the diagonal direction The difference absolute value of the pixel values of the G pixels (upper right diagonal, upper left diagonal) is calculated. Thereby, it can be determined that there is a correlation in a direction having a small difference absolute value among the horizontal direction, the vertical direction, and the diagonal direction. That is, the direction with the highest correlation among the horizontal direction, the vertical direction, and the diagonal direction is determined using the information of the G pixels having the minimum pixel interval. This discrimination result can be used for the process of interpolating from surrounding pixels (synchronization process).

  The basic array pattern C of the color filter 21 is arranged point-symmetrically with respect to the center of the basic array pattern C (the centers of the four G filters). In addition, the first array pattern A and the second array pattern B in the basic array pattern C are also arranged point-symmetrically with respect to the central G filter, so that the circuit scale of the subsequent processing circuit is reduced. Or can be simplified.

  As an example, in the basic array pattern C as shown in FIG. 6, the color filter array of the first and third lines of the first to sixth lines in the horizontal direction is GRGGBG. The color filter array of the second line is BGBGR. The color filter array of the fourth and sixth lines is GBGGRG. The color filter array of the fifth line is RGRBGB. In the example shown in FIG. 6, basic array patterns C, C ′, and C ″ are shown. The basic array pattern C ′ indicates a pattern obtained by shifting the basic array pattern C by one pixel in the horizontal direction and the vertical direction, The basic array pattern C ″ indicates a pattern obtained by shifting the basic array pattern C by two pixels each in the horizontal direction and the vertical direction. As described above, the color filter 21 has the same color filter array even when the basic array patterns C ′ and C ″ are repeatedly arranged in the horizontal direction and the vertical direction. In the first embodiment, the basic array pattern C Is referred to as a basic array pattern for convenience.

  The imaging apparatus 100 has a phase difference AF function. The image sensor 20 includes a plurality of phase difference detection pixels used when the phase difference AF function is activated. The plurality of phase difference detection pixels are arranged in a predetermined pattern. As shown in FIG. 7 as an example, a light shielding member 20A for shielding the left half pixel in the horizontal direction and a light shielding member 20B for shielding the right half pixel in the horizontal direction are provided on the phase difference detection pixels. Yes. In the first embodiment, for convenience of explanation, the phase difference detection pixel provided with the light shielding member 20A is referred to as a “first pixel”, and the phase difference pixel provided with the light shielding member 20B is represented by “ This is referred to as a “second pixel”. Further, when there is no need to distinguish between the first pixel and the second pixel, they are referred to as “phase difference pixels”.

  FIG. 8 shows an example of the first pixel and the second pixel arranged in the image sensor 20. The first pixel shown in FIG. 8 has a light shielding member 20A, and the second pixel has a light shielding member 20B. The light shielding member 20A is provided on the front side (microlens L side) of the photodiode PD, and shields the left half of the light receiving surface. On the other hand, the light shielding member 20B is provided on the front side of the photodiode PD and shields the right half of the light receiving surface.

  The microlens L and the light shielding members 20A and 20B function as a pupil dividing unit, the first pixel receives only the left side of the optical axis of the light beam passing through the exit pupil of the photographing lens 16, and the second pixel is photographed. Only the right side of the optical axis of the light beam passing through the exit pupil of the lens 16 is received. In this way, the light beam passing through the exit pupil is divided into left and right by the microlens L and the light shielding members 20A and 20B, which are pupil dividing portions, and enter the first pixel and the second pixel, respectively.

  In addition, the subject image corresponding to the left half of the light beam passing through the exit pupil of the photographing lens 16 and the subject image corresponding to the right half of the light beam are in focus (in focus). The portion forms an image at the same position on the image sensor 20. On the other hand, the front pin portion or the rear pin portion is incident on different positions on the image sensor 20 (the phase is shifted). Thereby, the subject image corresponding to the left half light beam and the subject image corresponding to the right half light beam can be acquired as parallax images (left eye image and right eye image) having different parallaxes.

  The imaging apparatus 100 detects a phase shift amount based on the pixel value of the first pixel and the pixel value of the second pixel by using the phase difference AF function. Then, the focal position of the photographing lens is adjusted based on the detected phase shift amount. Hereinafter, when it is not necessary to distinguish between the light shielding members 20A and 20B, the light shielding members 20A and 20B are referred to as “light shielding members” without reference numerals.

  In the first embodiment, as shown in FIG. 7 as an example, the light shielding member is a G filter at the upper left corner of two sets of the first array pattern A and the second array pattern B included in the basic array pattern C. For the other pixels. That is, in the example shown in FIG. 7, when an integer greater than or equal to 0 is “n”, the light shielding member 20A is arranged on the (6n + 1) th line in the vertical direction, and the light shielding member 20B is arranged on the (6n + 4) th line. ing. In the example shown in FIG. 7, the light shielding members are provided for all the basic array patterns C. However, the present invention is not limited to this, and only the basic array patterns C in a predetermined region of the image sensor 20 are provided. It may be.

  As described above, in the color filter 21, the light shielding member is provided for the pixels of the G filter at the upper left corner of all the first array pattern A and the second array pattern B, and both the vertical direction and the horizontal direction are provided. One phase difference pixel is regularly arranged for every three pixels. For this reason, since a relatively large number of normal pixels are arranged around the phase difference pixels, the interpolation accuracy in the case of interpolating the pixel values of the phase difference pixels from the pixel values of the normal pixels can be improved. The “normal pixel” refers to a pixel other than the phase difference pixel (for example, a pixel that does not have the light shielding members 20A and 20B).

  The image sensor 20 is classified into a first pixel group, a second pixel group, and a third pixel group. The first pixel group refers to a plurality of first pixels, for example. The second pixel group refers to, for example, a plurality of second pixels. The third pixel group refers to a plurality of normal pixels, for example. In the following, the RAW image output from the first pixel group is referred to as a “first image”, the RAW image output from the second pixel group is referred to as a “second image”, and the third image The RAW image output from the pixel group is referred to as a “third image”.

  Returning to FIG. 3, the imaging device 20 outputs a first image (digital signal indicating the pixel value of each first pixel) from the first pixel group, and outputs a second image (from the second pixel group). A digital signal indicating the pixel value of each second pixel). Further, the image sensor 20 outputs a third image (a digital signal indicating the pixel value of each normal pixel) from the third pixel group. Note that the third image output from the third pixel group is a chromatic image, for example, a color image having the same color array as the normal pixel array. The first image, the second image, and the third image output from the image sensor 20 are temporarily stored in a volatile storage area in the memory 26 via the interface unit 24.

  The image processing unit 28 includes a normal processing unit 30. The normal processing unit 30 generates a chromatic color normal image (an example of a first display image) by processing the R, G, and B signals corresponding to the third pixel group. Further, the image processing unit 28 includes a split image processing unit 32. The split image processing unit 32 generates an achromatic split image by processing G signals corresponding to the first pixel group and the second pixel group. Note that the image processing unit 28 according to the first embodiment is realized by an application specific integrated circuit (ASIC) that is an integrated circuit in which a plurality of functions related to image processing are integrated into one. However, the hardware configuration is not limited to this, and may be another hardware configuration such as a programmable logic device, a computer including a CPU, a ROM, and a RAM.

  The encoder 34 converts the input signal into a signal of another format and outputs it. The hybrid viewfinder 220 has an LCD 247 that displays an electronic image. The number of pixels in a predetermined direction on the LCD 247 (for example, the number of pixels in the horizontal direction, which is a parallax generation direction) is smaller than the number of pixels in the same direction on the display unit 213. The display control unit 36 is connected to the display unit 213 and the LCD 247, and selectively displays the image on the LCD 247 or the display unit 213 by selectively controlling the LCD 247 and the display unit 213. Hereinafter, the display unit 213 and the LCD 247 are referred to as “display devices” when it is not necessary to distinguish between them.

  Note that the imaging apparatus 100 according to the first embodiment is configured to be able to switch between a manual focus mode and an autofocus mode with a dial 212 (focus mode switching unit). When the manual focus mode is selected, the display control unit 36 causes the display device to display a live view image obtained by combining the split images. On the other hand, when the autofocus mode is selected by the dial 212, the CPU 12 operates as a phase difference detection unit and an automatic focus adjustment unit. The phase difference detection unit detects a phase difference between the first image output from the first pixel group and the second image output from the second pixel group. The automatic focus adjustment unit controls the lens driving unit (not shown) from the device control unit 22 via the mounts 256 and 346 so that the defocus amount of the photographing lens 16 is zero based on the detected phase difference. Then, the photographing lens 16 is moved to the in-focus position. Note that the above “defocus amount” refers to, for example, the amount of phase shift between the first image and the second image.

  The eyepiece detection unit 37 detects that a person (for example, a photographer) looks into the viewfinder eyepiece unit 242 and outputs the detection result to the CPU 12. Therefore, the CPU 12 can grasp whether or not the finder eyepiece unit 242 is used based on the detection result of the eyepiece detection unit 37.

  The external I / F 39 is connected to a communication network such as a LAN (Local Area Network) or the Internet, and controls transmission / reception of various types of information between the external device (for example, a printer) and the CPU 12 via the communication network. Therefore, when a printer is connected as an external device, the imaging apparatus 100 can output a captured still image to the printer for printing. Further, when a display is connected as an external device, the imaging apparatus 100 can output and display a captured still image or live view image on the display.

  FIG. 9 is a functional block diagram illustrating an example of main functions of the imaging apparatus 100 according to the first embodiment. In addition, the same code | symbol is attached | subjected to the part which is common in the block diagram shown in FIG.

  The normal processing unit 30 and the split image processing unit 32 each have a WB gain unit, a gamma correction unit, and a synchronization processing unit (not shown), and with respect to the original digital signal (RAW image) temporarily stored in the memory 26. Each processing unit sequentially performs signal processing. That is, the WB gain unit executes white balance (WB) by adjusting the gains of the R, G, and B signals. The gamma correction unit performs gamma correction on each of the R, G, and B signals that have been subjected to WB by the WB gain unit. The synchronization processing unit performs color interpolation processing corresponding to the color filter array (here, Bayer array as an example) of the image sensor 20, and generates synchronized R, G, and B signals. The normal processing unit 30 and the split image processing unit 32 perform image processing on the RAW image in parallel every time a RAW image for one screen is acquired by the image sensor 20.

  The normal processing unit 30 receives R, G, and B raw images from the interface unit 24, and converts the R, G, and B pixels of the third pixel group into the first pixel group and the second pixel group. It is generated by interpolating with surrounding pixels of the same color (for example, G pixel). Thereby, a normal image for recording can be generated based on the third image output from the third pixel group.

  Further, the normal processing unit 30 outputs the generated image data of the normal image for recording to the encoder 34. The R, G, B signals processed by the normal processing unit 30 are converted (encoded) into recording signals by the encoder 34 and recorded in the recording unit 40 (see FIG. 7). In addition, a normal image for display that is an image based on the third image processed by the normal processing unit 30 is output to the display control unit 36. In the following description, for convenience of explanation, when it is not necessary to distinguish between the above-mentioned “normal image for recording” and “normal image for display”, the term “for recording” and “for display” are used. Is referred to as a “normal image”.

  The image sensor 20 can change the exposure conditions (shutter speed by an electronic shutter as an example) of each of the first pixel group and the second pixel group, and thereby can simultaneously acquire images having different exposure conditions. . Therefore, the image processing unit 28 can generate an image with a wide dynamic range based on images with different exposure conditions. In addition, a plurality of images can be simultaneously acquired under the same exposure condition, and by adding these images, a high-sensitivity image with little noise can be generated, or a high-resolution image can be generated.

  On the other hand, the split image processing unit 32 extracts the G signal of the first pixel group and the second pixel group from the RAW image once stored in the memory 26, and the G of the first pixel group and the second pixel group. An achromatic split image is generated based on the signal. Each of the first pixel group and the second pixel group extracted from the RAW image is a pixel group including G filter pixels as described above. Therefore, the split image processing unit 32 can generate an achromatic left parallax image and an achromatic right parallax image based on the G signals of the first pixel group and the second pixel group. In the following, for convenience of explanation, the above “achromatic left parallax image” is referred to as a “left eye image”, and the above “achromatic right parallax image” is referred to as a “right eye image”.

  The split image processing unit 32 combines the left-eye image based on the first image output from the first pixel group and the right-eye image based on the second image output from the second pixel group to split the image. Generate an image. The generated split image data is output to the display controller 36.

  The display control unit 36 records image data corresponding to the third pixel group input from the normal processing unit 30 and the split corresponding to the first and second pixel groups input from the split image processing unit 32. Display image data is generated based on the image data of the image. For example, the display control unit 36 displays the image input from the split image processing unit 32 in the display area of the normal image indicated by the recording image data corresponding to the third pixel group input from the normal processing unit 30. The split image indicated by the data is synthesized. Then, the image data obtained by the synthesis is output to the display unit 213. Here, the split image generated by the split image processing unit 32 is a multi-divided image obtained by combining a part of the left eye image and a part of the right eye image. As the “multiple divided image” mentioned here, for example, a split image shown in FIG. The split image shown in FIG. 11 is an image obtained by synthesizing a part of the left eye image and a part of the right eye image at a position corresponding to the position of the predetermined area of the display unit 213, and between the images divided into four in the vertical direction. Is an image shifted in a predetermined direction (for example, a parallax generation direction) according to the in-focus state. The aspect of the split image is not limited to the example shown in FIG. 11 and may be an image obtained by combining the upper half image of the left eye image and the lower half image of the right eye image. In this case, the upper and lower images divided into two are shifted in a predetermined direction (for example, the parallax generation direction) according to the in-focus state.

  The method of combining the split image with the normal image is not limited to the combining method of fitting the split image in place of a part of the normal image. For example, a synthesis method in which a split image is superimposed on a normal image may be used. In addition, when the split image is superimposed, a combining method may be used in which the transmittance of a part of the normal image on which the split image is superimposed and the split image are appropriately adjusted and superimposed. As a result, a live view image showing a subject image continuously shot is displayed on the screen of the display unit 213, but the displayed live view image has a split image displayed in the normal image display area. Image.

  Hybrid viewfinder 220 includes OVF 240 and EVF 248. The OVF 240 is an inverse Galileo finder having an objective lens 244 and an eyepiece 246, and the EVF 248 has an LCD 247, a prism 245, and an eyepiece 246.

  Further, a liquid crystal shutter 243 is disposed in front of the objective lens 244, and the liquid crystal shutter 243 shields the optical image from being incident on the objective lens 244 when the EVF 248 is used.

  The prism 245 reflects the electronic image or various information displayed on the LCD 247 and guides it to the eyepiece lens 246, and combines the optical image and information (electronic image and various information) displayed on the LCD 247.

  Here, when the viewfinder switching lever 214 is rotated in the direction of the arrow SW shown in FIG. 1, an OVF mode in which an optical image can be visually recognized by the OVF 240 and an electronic image can be visually recognized by the EVF 248 each time it is rotated. The EVF mode is switched alternately.

  In the OVF mode, the display control unit 36 controls the liquid crystal shutter 243 to be in a non-light-shielding state so that an optical image can be visually recognized from the eyepiece unit. Further, only the split image is displayed on the LCD 247. Thereby, a finder image in which a split image is superimposed on a part of the optical image can be displayed.

  On the other hand, in the EVF mode, the display control unit 36 controls the liquid crystal shutter 243 to be in a light shielding state so that only an electronic image displayed on the LCD 247 can be visually recognized from the eyepiece unit. Note that the image data equivalent to the image data obtained by combining the split image output to the display unit 213 is input to the LCD 247, whereby the split image is combined with a part of the normal image in the same manner as the display unit 213. Electronic images can be displayed.

  FIG. 10 shows an example of display areas of the normal image and the split image in the display device. As an example, as shown in FIG. 10, the split image is displayed in a rectangular frame at the center of the screen of the display device, and a normal image is displayed in the outer peripheral area of the split image. Note that the edge line representing the rectangular frame shown in FIG. 10 is not actually displayed, but is shown in FIG. 10 for convenience of explanation.

  FIG. 11 schematically shows an example of a normal image and a split image displayed on the same screen of the display device as a comparative example with respect to the first embodiment. As an example, as shown in FIG. 11, the split image is displayed in an achromatic color, and the normal image is displayed in a chromatic color. However, depending on the white balance of the entire screen, the color of the split image does not stand out from the color of the normal image, and the boundary between the split image and the normal image becomes difficult to visually recognize.

  Therefore, in the imaging apparatus 100 according to the first embodiment, the image processing unit 28 visually differentiates the display color of the normal image from the display color of the split image (hereinafter, “display color differentiation process”). Image output processing including The display color differentiation process includes a process of acquiring color information of a normal image and determining a color having a color characteristic different from the color characteristic of the normal image as a display color of the split image based on the acquired color information. The “color information” refers to color information indicating the color characteristics of the object color of a normal image, for example. Further, the “color characteristic different from the color characteristic of the normal image” refers to a color characteristic different from the color characteristic of the object color of the normal image, for example. The “color characteristic different from the color characteristic of the normal image” refers to, for example, a color separated from the object color of the normal image by a predetermined angle in the hue circle. The “predetermined angle” here refers to an angle that is predetermined as an angle at which a color that is visually recognizable as having a color characteristic different from the color characteristic of the object color of the normal image is obtained. In this case, the “predetermined angle” is preferably 160 degrees or more and 200 degrees or less, more preferably 170 degrees or more and 190 degrees or less, and most preferably 180 degrees (an angle for specifying a complementary color). It is. In addition, the angle is not limited to this, and may be an angle designated by the user via the operation unit 14 (for example, an angle at which a user can visually recognize a color). Further, the present invention is not limited thereto, and an angle determined based on an evaluation result obtained by a sensory test by a plurality of subjects may be adopted as the “predetermined angle”. In this case, the majority of subjects among the plurality of subjects are evaluated to have a color characteristic different from the color characteristic of the object color of the normal image (sample image) (for example, a hue different from the hue of the object color of the sample image). The example which employ | adopts as a "predetermined angle" is given.

  In the first embodiment, as an example of the “color characteristic of the object color of the normal image”, the composition ratio of the primary color components in the normal image after the WB is adopted. The “component ratio of primary color components” here refers to a ratio that is normalized such that the sum of the R, G, and B pixel values is “1”.

  In the display color differentiation process, the first input / output function F (W) shown in FIG. 12 is used as an example. The horizontal axis shown in FIG. 12 is an input value of the first input / output function F (W), and shows one component of the R, G, and B composition ratio in the normal image after WB. That is, the horizontal axis indicates the average value of the R, G, and B pixel values in the normal image after WB. Here, a normalized value is applied so that (R average value Wr) + (G average value Wg) + (B average value Wb) = 1. The vertical axis shown in FIG. 12 is the output value of the first input / output function F (W) and indicates R: G: B of each pixel of the split image. According to the first input / output function F (W), assuming that the red color is dominant, for example, the input is Wr: Wg: Wb = 0.4: 0.3: 0.3, the output is R : G: B = 0.8: 1: 1 and the split image is blue-green gray. Thereby, the visibility of the boundary line between the normal image and the split image is improved.

  The first input / output function F (W) is a function for converting the composition ratio of the primary color components in the normal image after the WB to the composition ratio of the primary color components of the split image. The first input / output function F (W) includes a function f1 that is a decreasing function (a function having a characteristic that the output value decreases as the input value increases). The first input / output function F (W) shown in FIG. 12 includes a linear function having a negative slope as an example of the function f1, but has a characteristic that the output value decreases as the input value increases instead of the linear function. You may employ | adopt the nonlinear function which has. Further, the first input / output function F (W) shown in FIG. 12 includes functions f2 and f3 each having a slope of “0” that is a linear function continuous to the function f1, and these functions f2 and f3. Alternatively, a non-linear function may be used, or a linear function having a negative slope may be used. The first input / output function F (W) is a bent function (a function combining the functions f1 to f3). However, the first input / output function F (W) is not limited to this, and is a monotonically decreasing function (for example, y = −ax + b). There may be.

  In the imaging apparatus 100 according to the first embodiment, the image processing unit 28 assimilates the display color of the normal image and the display color of the split image by bringing the color characteristic of the normal image close to the color characteristic of the split image. (Hereinafter referred to as “display color assimilation process”). In the display color assimilation process, the second input / output function G (W) shown in FIG. 13 is used as an example. The horizontal axis shown in FIG. 13 is the input value of the second input / output function G (W), and the configurations of R, G, and B in the normal image after the WB are implemented as in the horizontal axis shown in FIG. One component of the ratio is shown. The vertical axis shown in FIG. 13 is the output value of the second input / output function G (W), and shows one component of the composition ratio of R, G, and B in the split image, similar to the vertical axis shown in FIG.

  Similarly to the first input / output function G (W), the second input / output function G (W) also converts the composition ratio of the primary color components in the normal image after WB to the composition ratio of the primary color components of the split image. It is a function. However, the second input / output function G (W) is a function symmetrical to the first input / output function F (W) shown in FIG. 12, and includes functions g1, g2, and g3.

  The function g1 is an increase function (a function in which the output value increases as the input value increases). The second input / output function G (W) shown in FIG. 13 includes a linear function having a positive slope as an example of the function g1, but instead of the linear function, the second input / output function G (W) has a characteristic that the output value increases as the input value increases. You may employ | adopt the nonlinear function which has. Further, the second input / output function G (W) shown in FIG. 13 is a linear function continuous to the function g1 and includes functions g2 and g3 having a slope of “0”. These functions g2 and g3 Instead of this, a nonlinear function may be used, or a linear function having a positive slope may be used. The second input / output function G (W) is a bent function (a function combining the functions g1 to g3). However, the second input / output function G (W) is not limited to this, and is a monotonically increasing function (for example, y = ax + b). May be.

  The first input / output function F (W) and the second input / output function G (W) are stored in a predetermined storage area (for example, the memory 28).

  Next, image output processing performed by the image processing unit 28 as an operation of the first embodiment will be described with reference to FIG. Here, the case where the image output process is performed by the image processing unit 28 is illustrated, but the present invention is not limited to this. For example, the CPU 12 executes the image output process program and the imaging apparatus 100 executes the image output process program. Image output processing may be performed.

  In FIG. 14, first, in step 400, a normal image is acquired by the image processing unit 28. The normal image acquired in step 400 is stored in a predetermined storage area (for example, the memory 26) by the image processing unit 28. In the next step 402, the image processing unit 28 acquires the left eye image and the right eye image, and generates a split image based on the acquired left eye image and right eye image.

  In the next step 404, the image processing unit 28 executes WB processing. FIG. 15 shows an example of the flow of WB processing. In step 404A shown in FIG. 15, the image processing unit 28 acquires color information from the normal image, and estimates a light source color when imaging is performed based on the acquired color information. In the next step 404B, the image processor 28 acquires a gain corresponding to the light source color estimated in step 404A as a gain for each of R, G, and B colors used in WB. In the first embodiment, for example, gains are determined for each of the sun on a clear day, the sun on a cloudy day, a fluorescent lamp, and a tungsten light source (for example, stored in the memory 26). The gain of the light source corresponding to the estimated light source color is acquired.

  In the next step 404C, the image processing unit 28 executes white balance with the gain acquired in step 404B, and then ends the white balance process.

  Returning to FIG. 14, in step 406, the image processing unit 28 sets a pixel to be processed (target pixel (i, j)) in the split image generated in step 402. In this case, for example, when the size of the split image is m × n pixels, the target pixel is sequentially moved from the pixel (1, 1) to the pixel (m, n) every time this step 406 is performed.

  In the next step 408, the image processing unit 28 determines whether or not the normal image obtained by executing the white balance process in step 404 is not an achromatic image. The “achromatic image” referred to here is, for example, as a component ratio of R, G, and B, | R−0.33 | ≦ 0.05, | G−0.33 | ≦ 0.05, and | B-0. An image satisfying 33 | ≦ 0.05. If it is determined in step 408 that the normal image is not an achromatic image, the determination is affirmed and the process proceeds to step 410. If it is determined in step 408 that the normal image is an achromatic image, the determination is negative and the process proceeds to step 412.

  Whether or not the normal image is an achromatic image is determined by determining whether the occupancy of the achromatic region (the number of pixels of the achromatic pixel) in the entire region (the total number of pixels) of the normal image is a predetermined occupancy rate ( For example, it may be performed by determining whether or not it is 95% or more. Further, when the predetermined area in the normal image is an achromatic color area, it may be determined that the normal image is an achromatic color image. In addition, the normal image may be determined to be an achromatic image when the achromatic region is equal to or larger than a predetermined area (a predetermined number of pixels) in a region adjacent to the boundary with the split image in the normal image. In addition, since the area adjacent to the boundary with the split image in the normal image is not an achromatic color, the other area is an achromatic color area, so the occupation ratio of the achromatic color area exceeds the predetermined occupation ratio. If it has become, it may be determined that the normal image is not an achromatic image.

  In step 412, the image processing unit 28 executes chromatic color addition processing. FIG. 16 shows an example of the flow of chromatic color imparting processing. In step 412A shown in FIG. 16, the image processing unit 28 determines whether or not the target pixel set in step 406 is the first processing target pixel. If the target pixel is the first pixel to be processed in step 412A, the determination is affirmed and the process proceeds to step 412B. If the target pixel is not the first pixel to be processed in step 412A, the determination is negative and the process proceeds to step 412C.

  By the way, in a predetermined storage area (for example, the memory 26), predetermined R, G, and B values that can give a chromatic color to the target pixel by multiplying the R, G, and B pixel values of the target pixel. The component ratio (hereinafter referred to as “predetermined component ratio”) is stored.

  Therefore, in step 412B, the image processing unit 28 acquires a predetermined composition ratio from a predetermined storage area. In the next step 412C, the image processing unit 28 multiplies the pixel value of the same color of the target pixel by the component for each color having the predetermined composition ratio acquired in step 412B, so that the R, G, B of the target pixel is obtained. The pixel values of each color are corrected. For example, when “R: G: B = 0.2: 0.2: 0.6” is acquired as the predetermined configuration ratio in step 412B, the image processing unit 28 determines that “R: G: B = 0.2. : 0.2: 0.6 "is multiplied by the pixel value of the same color of the target pixel. As a result, the target pixel is corrected to a bluish pixel. If, for example, “R: G: B = 0.6: 0.2: 0.2” is acquired as the above-described predetermined configuration ratio in step 412B, the image processing unit 28 determines that “R: G: B”. = 0.6: 0.2: 0.2 "is multiplied by the pixel value of the same color of the target pixel. As a result, the target pixel is corrected to a reddish pixel. As a result of this step 412C being executed by the image processor 28, a chromatic color having a color characteristic different from the color characteristic of the normal image is determined as the display color of the target pixel.

  In the next step 412D, the image processing unit 28 stores the R, G, and B pixel values of the target pixel corrected in step 412C in a predetermined storage area (for example, the memory 26), thereby storing the image. Thereafter, the process proceeds to step 418.

  In step 410, the image processing unit 28 determines whether or not a display color assimilation condition that is a condition for executing the display color assimilation process is satisfied. The “display color assimilation condition” mentioned here includes, for example, a condition that an instruction to start execution of the display color assimilation process is input via the operation unit 14. If the display color assimilation condition is not satisfied in step 410, the determination is affirmed and the routine proceeds to step 414. If the display color assimilation condition is satisfied in step 410, the determination is negative and the routine proceeds to step 416.

  Here, an example in which the process proceeds to step 414 when an affirmative determination is made in step 410 is given, but does the condition for executing the display color differentiation process be satisfied between step 410 and step 414? A step of performing a process of determining whether or not may be inserted. In this case, if the condition for executing the display color differentiation process is satisfied, the determination is affirmed and the process proceeds to step 414. If the condition for executing the display color differentiation process is not satisfied, the determination is negative. Then, the process proceeds to step 418. Examples of the condition for executing the display color differentiation process include a condition that an instruction to start execution of the display color differentiation process is input via the operation unit 14. In addition, the average value (or composition ratio) of the R, G, and B pixel values of the normal image and the average value (or composition ratio) of the R, G, and B pixel values of the split image are R, G, An example of the condition is that the colors of B match within a predetermined error.

  In step 414, the image processing unit 28 executes display color differentiation processing. FIG. 17 shows an example of the flow of display color differentiation processing. In step 414A shown in FIG. 17, it is determined whether or not the pixel of interest set in step 406 is the first pixel to be processed. If the target pixel is the first pixel to be processed in step 414A, the determination is affirmed and the routine proceeds to step 414B. If the target pixel is not the first pixel to be processed in this step 414A, the determination is negative and the routine proceeds to step 414C.

  In step 414B, the image processing unit 28 acquires R, G, and B pixel values (for example, pixel values for all pixels included in the specific image) from the specific image. Here, a part of the normal image generated in step 400 is adopted as the “specific image”. However, the present invention is not limited to this, and the normal image and the left-eye image and right-eye image generated in step 402 are used. May be. In Step 414B, as the “part of the normal image”, the image output from the edge on the boundary side with the split image imaging area in the normal imaging area among the normal images generated in Step 400 is used. Although adopted, the size of the “edge portion” here refers to, for example, the size of 1/3 of the normal image area width. “The size of 1/3 of the normal image area width” is merely an example, and it may be an area sufficient to obtain effective pixel values (an example of color information). The size of the normal image area and the split image may be used. It goes without saying that the size of the “edge” required to acquire an effective pixel value differs depending on the size of the pixel. Moreover, not only an edge part but the other area | region in a normal image may be sufficient. The other region in the normal image refers to a region other than the “edge” in the normal image. Further, the “specific image” is not necessarily a part of the normal image, and may be an image of the entire area of the normal image. In addition, the “normal imaging area” is an area used for imaging a normal image in the image sensor 20, and is, for example, an area of the imaging device 20 corresponding to the “normal image display area” illustrated in FIG. 10. Point to. The “split image capturing area” described above refers to an area that is used for capturing a split image in the image sensor 20 and is adjacent to the normal image capturing area, and corresponds to, for example, the “split image display area” illustrated in FIG. This refers to the area of the imaging device 20 that performs.

  In the next step 414D, the image processing unit 28 calculates an average value for each color of the R, G, and B pixel values acquired in step 414B. In the next step 414E, the composition ratio of the R, G, B pixel values is calculated by the image processing unit 28 based on the average value for each color of the R, G, B pixel values calculated in step 414D. The Here, the image processing unit 28 normalizes the average value for each color of the R, G, and B pixel values calculated in step 414D, for example, so that the sum is “1”. The composition ratio is calculated.

  In the next step 414F, the image processing unit 28 acquires the first input / output function F (W) from a predetermined storage area. In the next step 414G, the image processing unit 28 calculates the composition ratio of the R, G, and B pixel values in the split image using the first input / output function F (W). That is, the composition ratio of the R, G, B pixel values calculated in step 414E is converted into the composition ratio of the R, G, B pixel values in the split image by the first input / output function F (W).

  In the next step 414H, the image processing unit 28 determines whether or not the different color application condition is satisfied. The “different color application condition” here refers to a condition for executing a process (different color application process) of applying different chromatic colors to each of the image by the left eye image and the image by the right eye image in the split image. As the different color imparting condition, for example, a condition that an instruction to start execution of the different color imparting process is made through the operation unit 14 can be cited. Further, the “different color imparting process” referred to here indicates, for example, steps 414I, 414K, 414L, 414M, and 414N described later. If the different color application condition is satisfied in step 414H, the determination is affirmed and the routine proceeds to step 414I. If it is determined in step 414H that the different color imparting conditions are not satisfied, the determination is negative and the routine proceeds to step 414J.

  In step 414I, the image processing unit 28 determines whether the target pixel set in step 406 is a pixel included in the right-eye image. If the target pixel is not a pixel included in the right-eye image in this step 414J (if the target pixel is a pixel included in the left-eye image), the determination is negative and the process proceeds to step 414L. If the target pixel is a pixel included in the right-eye image in step 414I, the determination is affirmed and the process proceeds to step 414K.

  In step 414K, the right-eye image composition ratio is calculated by the image processing unit 28 adding a correction term for the right-eye image to the composition ratio calculated in step 414G. For example, if the component ratio of the R, G, and B pixel values calculated in step 414G is F (Wr), F (Wg), and F (Wb), a correction term ΔR is added to F (Wr). As a result, “F (Wr) + ΔR”, which is the R component of the right-eye image composition ratio, is calculated. Also, by adding the correction term ΔG to F (Wg), “F (Wg) + ΔG”, which is the G component of the right-eye image composition ratio, is calculated. Further, by adding the correction term ΔB to F (Wb), “F (Wb) + ΔB”, which is the B component of the right-eye image composition ratio, is calculated.

  In the next step 414M, the image processing unit 28 multiplies the component value for each color of the right-eye image composition ratio calculated in step 414K by the pixel value of the same color of the pixel of interest, so that R, G, The pixel value of each color of B is corrected, and then the process proceeds to step 414P. As a result of this step 414M being executed by the image processing unit 28, a color having a color characteristic different from the color characteristic of the normal image is determined as the display color of the target pixel.

  In step 414L, the left eye image composition ratio is calculated by adding a correction term for the left eye image to the composition ratio calculated in step 414G by the image processing unit 28. For example, if the component ratio of the R, G, and B pixel values calculated in step 414G is F (Wr), F (Wg), and F (Wb), the correction term ΔR is subtracted from F (Wr). Thus, “F (Wr) −ΔR” which is the R component of the left-eye image composition ratio is calculated. Further, by subtracting the correction term ΔG from F (Wg), “F (Wg) −ΔG”, which is the G component of the left-eye image composition ratio, is calculated. Further, by subtracting the correction term ΔB from F (Wb), “F (Wb) −ΔB”, which is the B component of the left-eye image composition ratio, is calculated. The correction terms ΔR, ΔG, and ΔB may be values that allow the difference to be visually recognized. For example, F (Wr), ΔR = ρF (Wr), ΔG = ρF (Wg), ΔB = ρF (Wb) A value obtained by simply linearly multiplying F (Wg) and (Wb) may be adopted.

  In the next step 414N, the image processing unit 28 multiplies the component value for each color of the left-eye image component ratio calculated in step 414L by the pixel value of the same color of the target pixel, so that R, G, The pixel value of each color of B is corrected, and then the process proceeds to step 414P. As a result of this step 414N being executed by the image processing unit 28 in this way, a color having a color characteristic different from the color characteristic of the normal image is determined as the display color of the target pixel. Further, by executing the step 414N and the above step 414 by the image processing unit 28, different chromatic colors in the split-eye left-eye image and right-eye image are determined as the split-image display color.

  In step 414J, the image processing unit 28 multiplies the component value for each color of the composition ratio calculated in step 414G by the pixel value of the same color of the target pixel, so that each color of R, G, B of the target pixel is changed. The pixel value is corrected, and then the process proceeds to step 414P. As a result of this step 414J being executed by the image processing unit 28 in this way, a color having a color characteristic different from the color characteristic of the normal image is determined as the display color of the target pixel.

  In step 414P, the image processing unit 28 stores the R, G, and B pixel values of the target pixel corrected in any of the above steps 414J, 414M, and 414N in a predetermined storage area (for example, the memory 26). To store the image, and then the process proceeds to step 418.

  On the other hand, in step 414C, it is determined whether or not a different color imparting process has been executed for the previous process target pixel. If a different color imparting process is executed for the previous pixel to be processed in step 414C, the determination is affirmed and the process proceeds to step 414Q. If the different color imparting process has not been executed for the previous pixel to be processed in step 414C, the determination is negative and the process proceeds to step 414J.

  In step 414Q, the image processing unit 28 determines whether the target pixel set in step 406 is a pixel included in the right-eye image. In step 414Q, when the target pixel is not a pixel included in the right-eye image (when the target pixel is a pixel included in the left-eye image), the determination is negative and the process proceeds to step 414N. If it is determined in step 414Q that the pixel of interest is a pixel included in the right-eye image, the determination is affirmed and the process proceeds to step 414M.

  Returning to FIG. 14, in step 416, the display color assimilation process is executed by the image processing unit 28. FIG. 18 shows an example of the flow of display color assimilation processing. In step 416A shown in FIG. 18, it is determined whether or not the target pixel set in step 406 is the first processing target pixel. If the target pixel is the first pixel to be processed in step 416A, the determination is affirmed and the routine proceeds to step 416B. If the target pixel is not the first pixel to be processed in step 416A, the determination is negative and the process proceeds to step 416G.

  In step 416B, the image processing unit 28 acquires R, G, and B pixel values (for example, pixel values for all pixels included in the specific image) from the specific image. Here, the “specific image” refers to an image corresponding to the “specific image” used in step 414B, for example.

  In the next step 416C, the image processing unit 28 calculates an average value for each color of the R, G, and B pixel values acquired in step 416B. In the next step 416D, the image processing unit 28 calculates the composition ratio of the R, G, and B pixel values based on the average value for each color of the R, G, and B pixel values calculated in Step 416C. The Here, the image processing unit 28 normalizes the average value for each color of the R, G, and B pixel values calculated in step 416C, for example, so that the sum is “1”, and thereby R, G, B The composition ratio is calculated.

  In the next step 416E, the image processing unit 28 acquires the second input / output function G (W) from a predetermined storage area. In the next step 416F, the image processing unit 28 calculates the composition ratio of the R, G, and B pixel values in the split image using the second input / output function G (W). That is, the composition ratio of the R, G, and B pixel values calculated in step 416D is converted into the composition ratio of the R, G, and B pixel values in the split image by the second input / output function G (W).

  In the next step 416G, the image processing unit 28 multiplies the component value for each color of the composition ratio calculated in step 416F by the pixel value of the same color of the target pixel, so that R, G, B of the target pixel is obtained. The pixel value of each color is corrected, and then the process proceeds to step 416H. As a result of the execution of step 416G by the image processing unit 28 in this manner, a color having a color characteristic approaching the color characteristic of the normal image by a predetermined degree is determined as the display color of the target pixel.

  In the next step 416H, the image processing unit 28 stores the R, G, B pixel values of the target pixel corrected in the above step 416G in a predetermined storage area (for example, the memory 26), thereby storing the image. Thereafter, the process proceeds to step 418.

  Returning to FIG. 14, in step 418, image processing other than WB processing, chromatic color addition processing, display color differentiation processing, and display color assimilation processing is performed on the target pixel (i, j) set in step 406. (For example, gamma correction and synchronization processing) are performed.

  In the next step 420, the image processing unit 28 stores the image by storing the pixel value obtained by the image processing performed in step 418 in a predetermined storage area (for example, the memory 26) for each pixel. Thereafter, the process proceeds to step 422.

  In the next step 422, the image processing unit 28 determines whether or not the chromatic color providing process, the display color differentiation process, or the display color assimilation process for all the pixels has been completed. If the chromatic color providing process, the display color differentiation process, or the display color assimilation process for all the pixels is completed in step 422, the determination is affirmed and the process proceeds to step 424. If the chromatic color providing process, the display color differentiation process, or the display color assimilation process for all the pixels is not completed in step 422, the determination is negative and the process proceeds to step 406.

  In step 424, the image processing unit 28 outputs the image stored in the predetermined storage area in steps 400 and 420 to a predetermined output destination, and then ends the image processing. The above-mentioned “predetermined output destination” includes, for example, the display control unit 36. In addition, when the output to the external apparatus is instructed by the photographer or the like via the operation unit 14, The external I / F 40 is applied as the “predetermined output destination”.

  When the image stored in the predetermined storage area in steps 400 and 420 is output to the display control unit 36, as an example, a live view image is displayed on the display device as shown in FIG. 19, FIG. 20, FIG. Is displayed. The live view image includes a normal image and a split image. Split images are roughly classified into left-eye images and right-eye images (see, for example, FIG. 20).

  Here, when the photographing lens 16 is not focused on the subject, the left-eye image and the right-eye image in the split image are shifted in the parallax generation direction (for example, the horizontal direction). In addition, the boundary image between the normal image and the split image is also shifted in the parallax generation direction. This indicates that a phase difference has occurred, and the photographer can visually recognize the phase difference and the parallax generation direction through the split image.

  On the other hand, when the photographing lens 16 is in focus with respect to the subject, the left-eye image and the right-eye image in the split image do not shift. The boundary image between the normal image and the split image is not shifted. This indicates that no phase difference has occurred, and the photographer can recognize that no phase difference has occurred visually through the split image.

  In this way, the photographer can confirm the in-focus state of the photographing lens 16 by the split image displayed on the display device. In the manual focus mode, the focus shift amount (defocus amount) can be reduced to zero by manually operating the focus ring 302 of the photographing lens 16. Further, the normal image and the split image can be displayed as color images without color misregistration, and the manual focus adjustment by the photographer can be supported by the color split image.

  In the display color differentiation process shown in FIG. 17, when the process of step 414J is executed by the image processing unit 28, as shown in FIG. 19, for example, a normal image is displayed on the display device and the color characteristics of the normal image are displayed. The split image is displayed in a color having a color characteristic different from the above. In the example shown in FIG. 19, the display color of the normal image is red (in this case, orange), and the display color of the split image is blue. In this case, for example, “R: G: B = 0.35: 0.35: 0.3” as the component ratio of the R, G, B pixel values of the normal image, and the R, G, B pixel values of the split image As the component ratio, “R: G: B = 0.31: 0.31: 0.38” can be given. Thus, when the color ratio of the entire scene (normal image as an example) is strongly influenced by the red object color, the blue color is adopted as the split image display color. Conversely, when the color ratio of the entire scene is strongly influenced by a blue object color, a red color is employed as the split image display color. Therefore, it is possible to make the split image stand out from the normal image.

  In addition, in the display color differentiation process shown in FIG. 17, when the image processing unit 28 executes the processes of steps 414K, 414L, 414M, and 414N, as shown in FIG. 20 as an example, the colors of the left eye image and the right eye image Split images with different characteristics are displayed. In the example shown in FIG. 20, the left-eye image in the split image has a blue color, and the right-eye image in the split image has a green color. Note that the display color of the normal image is a red color (here, orange) as in the example shown in FIG. As a result, the split image can be emphasized with respect to the normal image, and the left-eye image and the right-eye image can be easily identified in the split image.

  If the normal image is an achromatic image (step 408: N), the split image is also an achromatic color image, so that it becomes difficult to visually recognize the boundary between the normal image and the split image as shown in FIG. Therefore, the image processing unit 28 according to the first embodiment applies a chromatic color to the split image as illustrated in FIG. 22 by executing the chromatic color applying process shown in FIG. In the example shown in FIG. 22, a blue color is adopted as the display color of the split image. As a result, the split image can be enhanced with respect to the achromatic normal image. Note that if the light source color is estimated by the image processing unit 28 and the red color is dominant in the estimated light source color, the display color of the split image is set to the blue color, and the blue color is used in the estimated light source color. If is dominant, the display color of the split image may be a red color.

  When the display color assimilation process shown in FIG. 18 is executed by the image processing unit 28, as shown in FIG. 23 as an example, a normal image and a split image in which the display colors are assimilated are displayed. In the example shown in FIG. 23, the red color (an orange color as an example) is dominant as the object color of the normal image. Therefore, the red color is adopted as the display color of the split image. It is displayed to melt into. Therefore, it is possible to make it difficult to visually recognize the boundary between the normal image and the split image.

  As described above, the imaging apparatus 100 according to the first embodiment has the left and right sides of the optical axis of the light beam passing through the exit pupil of the photographing lens 16 (the subject image that has passed through the first and second regions). 1) includes first and second pixel groups that are divided into pupils and imaged respectively. The image processing unit 28, which is an example of a normal image generation unit, generates a normal image, which is an example of a first display image, based on an image output from the image sensor 20 having the first and second pixel groups. Generate (step 400). The image processing unit 28, which is an example of a second display image generation unit, is an example of a second display image based on the first and second images output from the first and second pixel groups. A split image is generated (step 402). In addition, the image processing unit 28, which is an example of an acquisition unit, acquires color information of the generated normal image (Step 414B). The image processing unit 28, which is an example of a determination unit, determines a color having a color characteristic different from the color characteristic of the normal image as a split image display color based on the acquired color information (steps 414J, 414M, 414N). Thereby, the visibility of the boundary line between the normal image and the split image can be improved as compared with the case where this configuration is not provided.

  In addition, the imaging apparatus 100 according to the first embodiment employs a color that can be visually recognized as having a color characteristic different from that of the normal image as a color having a color characteristic different from that of the normal image. Yes. Thereby, compared with the case where it does not have this structure, a normal image and a split image can be visually identified.

  In the imaging device 100 according to the first embodiment, the image processing unit 28 acquires color information from the normal image. Thereby, compared with the case where this configuration is not provided, the color characteristics of the split image and the color characteristics of the normal image can be differentiated with high accuracy with a simple configuration.

  In the imaging device 100 according to the first embodiment, the image processing unit 28 acquires color information based on the normal image and the split image. Thereby, compared with the case where this configuration is not provided, the color characteristics of the split image and the color characteristics of the normal image can be differentiated with high accuracy. Even if the normal image is overexposed, the first and second pixel groups have low sensitivity, so that color information can be acquired. Note that the color information may be acquired based on the normal image and the first image, the color information may be acquired based on the normal image and the second image, the normal image, the first image, and the second image. Color information may be obtained for images

  Further, in the imaging apparatus 100 according to the first embodiment, the imaging element 20 is used for the normal imaging area used for imaging the normal image and the split image imaging area used for imaging the split image and adjacent to the normal imaging area. And have. Then, the image processing unit 28 acquires color information from an image output from an edge portion of the normal image on the boundary side with the split image imaging region in the normal imaging region. As a result, the boundary between the normal image and the split image can be made clearer than in the case without this configuration.

  In addition, the imaging apparatus 100 according to the first embodiment employs color information indicating the color characteristics of the object color of the normal image as color information. Thereby, the visibility of the boundary line between the normal image and the split image can be improved even when the object color is more dominant than the light source color in the normal image, compared to the case without this configuration.

  In addition, the imaging apparatus 100 according to the first embodiment employs a color characteristic different from the color characteristic of the normal image as the color characteristic different from the color characteristic of the normal image. Thereby, the visibility of the boundary line between the normal image and the split image can be improved even when the object color is more dominant than the light source color in the normal image, compared to the case without this configuration.

  Further, the imaging device 100 according to the first embodiment applies hue as the color characteristic. Accordingly, the visibility of the boundary line between the normal image and the split image can be further improved with a simple configuration as compared with the case where the present configuration is not provided.

  In addition, the imaging apparatus 100 according to the first embodiment employs a color that has a color characteristic different from the color characteristic of the normal image and that is separated from the object color of the normal image by a predetermined angle in the hue circle. Yes. Accordingly, an object color suitable as a color having a color characteristic different from the color characteristic of the normal image can be easily determined as compared with the case without this configuration.

  In addition, the imaging apparatus 100 according to the first embodiment employs an angle at which a color that can be visually recognized as having a color characteristic different from the color characteristic of the object color of the normal image is obtained as the predetermined angle. ing. Thereby, compared with the case where this structure is not provided, a split image can be clearly shown with respect to a normal image.

  In addition, the imaging apparatus 100 according to the first embodiment employs the composition ratio of the primary color components in the normal image as the color characteristic of the object color of the normal image. As a result, it is possible to easily identify an appropriate color characteristic for improving the visibility of the boundary line between the normal image and the split image as the color characteristic of the object color of the normal image, as compared with the case without this configuration. it can.

  In addition, the imaging apparatus 100 according to the first embodiment employs the composition ratio of the primary color components in the normal image after the white balance is implemented as the composition ratio of the primary color components in the normal image. Thereby, compared with the case where this configuration is not provided, the boundary line between the normal image and the split image in which the white balance is performed can be made clear.

  In the imaging apparatus 100 according to the first embodiment, the image processing unit 28, which is an example of an estimation unit, estimates a light source color based on a normal image. Then, a gain for the primary color component used in white balance is determined according to the light source color estimated by the image processing unit 28. Thereby, the visibility of the boundary line between the normal image and the split image on which the white balance is performed can be improved in consideration of the light source color as compared with the case where the present configuration is not provided.

  In the imaging apparatus 100 according to the first embodiment, the composition ratio of the primary color components in the normal image is determined based on the average value of the pixel values for each of the primary color components of the object color in the normal image. As a result, as compared with the case without this configuration, it is possible to easily obtain an appropriate configuration ratio for improving the visibility of the boundary line between the normal image and the split image as the configuration ratio of the primary color components in the normal image. .

  In the imaging apparatus 100 according to the first embodiment, the converted composition ratio obtained by converting the composition ratio of the primary color components in the normal image using the input / output function including the decrease function is the color characteristic of the normal image. Adopted as different color characteristics. Thereby, the visibility of the boundary line between the normal image and the split image can be improved with a simple configuration as compared with the case where the present configuration is not provided.

  In the imaging device 100 according to the first embodiment, each of the first and second pixel groups is a single color pixel group, and the image processing unit 28 is output from the first and second pixel groups. An achromatic image is generated as a split image based on the first and second images. Thereby, it is possible to obtain an achromatic split image with a simple configuration as compared with the case without this configuration.

  In the imaging apparatus 100 according to the first embodiment, a green pixel group is employed as the single color pixel group. Thereby, compared with the case where this configuration is not provided, it is possible to obtain a highly accurate achromatic image as an achromatic split image.

  In the imaging apparatus 100 according to the first embodiment, when the split image is an achromatic image, the image processing unit 28 has a color characteristic different from that of the normal image based on the acquired color information. The coloring is determined as the display color of the split image. Thereby, it is possible to determine the chromatic color as the display color of the split image with a simple configuration as compared with the case without this configuration.

  Further, in the imaging apparatus 100 according to the first embodiment, when the normal image and the split image are achromatic images, the image processing unit 28 has color characteristics different from the color characteristics of the normal image based on the acquired color information. The chromatic color having the is determined as the display color of the split image. Thereby, the visibility of the boundary line between the normal image and the split image can be improved as compared with the case where this configuration is not provided.

  In the imaging apparatus 100 according to the first embodiment, the image processing unit 28 is a chromatic color having a color characteristic different from the color characteristic of the normal image, and is different for the left-eye image and the right-eye image in the split image. Is determined as the split image display color. This makes it easier to visually distinguish the left-eye image and the right-eye image in the split image compared to the case without this configuration.

  In the imaging apparatus 100 according to the first embodiment, as the split image, an image in which the left eye image and the right eye image are alternately arranged in a direction intersecting the parallax generation direction (an example of a predetermined direction) is employed. doing. This makes it easier to visually distinguish the left eye image and the right eye image in the split image.

  In the imaging apparatus 100 according to the first embodiment, when the image processing unit 28 satisfies the display color assimilation condition, the image processing unit 28 approaches the color characteristics of the normal image with a predetermined degree based on the acquired color information. Further, the color having the above color characteristic is further determined as the display color of the split image. As a result, it is possible to make it difficult to visually recognize the boundary between the normal image and the split image as compared with the case where this configuration is not provided.

  In addition, the imaging apparatus 100 according to the first embodiment includes a display control unit that performs control to display a normal image and a split image whose display color is corrected by the image processing unit 28 side by side on the same screen on the display device. 36 is further included. Thereby, it is possible to visually recognize the boundary line between the normal image and the split image with a simple configuration as compared with the case where this configuration is not provided.

  In the imaging device 100 according to the first embodiment, the imaging element 20 includes the third pixel group, and the image processing unit 28 is based on the third image output from the third pixel group. A normal image is generated. Thereby, compared with the case where this structure is not provided, the image quality of a normal image can be improved.

  In the first embodiment, the composition ratio of the primary color components in the normal image after the implementation of WB is adopted. However, the configuration ratio is not limited to this, and the composition ratio of the primary color components in the normal image before implementation of the WB. Alternatively, it may be a composition ratio of primary color components in a normal image after gamma correction.

  Further, in the first embodiment described above, an example in which the R, G, and B pixel values of the object color of the normal image are employed as the pixel value that is an example of the color information acquired from the normal image has been described. However, the present invention is not limited to this, and R, G, and B pixel values of the light source color may be adopted. Thereby, even when the light source color is more dominant than the object color in the normal image, the visibility of the boundary line between the normal image and the split image can be improved.

  In the first embodiment, the example in which the pixel value is corrected with a configuration ratio that does not depend on the type of the display device in the chromatic color imparting process and the display color discrimination process has been described. However, it is determined according to the type of the display device. The pixel value may be corrected with the above-described composition ratio. For example, when the use of the EVF 248 is detected by the eyepiece detection unit 37, a further correction term (for example, a positive correction term) is added to the component ratio used in the pixel value correction described in the first embodiment. The pixel value is corrected using the obtained composition ratio. Thereby, even when the EVF 248 having a display area smaller than that of the display unit 213 is used, the boundary between the normal image and the split image can be visually recognized.

  In the first embodiment, the maximum output value of the first input / output function F (W) is “1”, but may be a value exceeding “1”. When the output value of the first input / output function F (W) exceeds “1”, the value may be corrected to “1” or less by performing a normalization process thereafter. The same applies to the output value of the second input / output function G (W).

  In the first embodiment, the example in which the composition ratio of the R, G, and B pixel values is calculated from the average value of the R, G, and B pixel values acquired from the specific image has been described. The component ratio of the R, G, and B pixel values may be calculated from the representative values of the R, G, and B pixel values acquired from the image. The representative value mentioned here can be exemplified by the mode, median, and the like of R, G, and B pixel values acquired from a specific image.

  In the first embodiment, the G pixel is exemplified as the phase difference pixel. However, the present invention is not limited to this, and any of R, G, and B pixels may be used. In the first embodiment, a monochromatic pixel is exemplified as the phase difference pixel. However, the present invention is not limited to this, and two or more of the R, G, and B pixels may be used as the phase difference pixel. In this case, a color image is displayed as a split image. Even in this case, the present invention is established. It should be noted that, as described in the first embodiment, the time required for the pixel value correction processing is shorter and the visual processing is shorter when the single-color phase-difference pixels are used, as compared with the case where the phase-difference pixels of a plurality of colors are employed. In addition, it is easy to distinguish from the background color (for example, the display color of the normal image).

  In the first embodiment, the interchangeable lens type digital camera in which the reflex mirror is omitted is illustrated. However, the present invention is not limited to this, and the present invention can also be applied to an imaging apparatus having an integrated lens. it can.

  Moreover, in the said 1st Embodiment, although the imaging device 100 which has the display part 213 and EVF248 was illustrated, it is not restricted to this, You may have any one of the display part 213 and EVF248.

  In the first embodiment, the split image divided into a plurality of parts in the vertical direction is illustrated. However, the present invention is not limited to this, and an image divided into a plurality of parts in the left-right direction or the oblique direction may be applied as the split image.

  In the first embodiment, the example of displaying the split image has been described. However, the present invention is not limited to this. Other focus confirmation images are generated from the left eye image and the right eye image, and the focus confirmation image is displayed. You may make it do. For example, the left-eye image and the right-eye image may be superimposed and displayed as a composite image. If the image is out of focus, the image may be displayed as a double image, and the image may be clearly displayed when the image is in focus.

  In the first embodiment, the example in which the display color for output is determined according to the input / output function F (W) shown in FIG. 12 has been described. However, the present invention is not limited to this. The display color for output may be determined using a color conversion table stored in advance in (1). In this case, an example of the color conversion table is a color conversion table in which the output value decreases as the input value increases. Further, the data relating to the input and output of the color conversion table may be data created based on the input / output function F (W) shown in FIG. 12, or data created based on the hue circle. May be. Thereby, the visibility of the boundary line between the normal image and the split image can be improved with a simple configuration. The same applies to the case where the display color for output is determined according to the input / output function G (W) shown in FIG.

  Also, the image output process flow (see FIG. 14), the white balance process flow (see FIG. 15), the chromatic color application process flow (see FIG. 16), and the display color differentiation process described in the first embodiment. The flow (see FIG. 17) and the display color assimilation process (see FIG. 18) are examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, and the processing order may be changed within a range not departing from the spirit. Each process included in the image output process may be realized by a software configuration using a computer by executing a program, or may be realized by a hardware configuration. Further, it may be realized by a combination of a hardware configuration and a software configuration. The same applies to the white balance processing, chromatic color imparting processing, display color differentiation processing, and display color assimilation processing.

  When the image output process is realized by executing a program by a computer, the program may be stored in a predetermined storage area (for example, the memory 26) in advance. The same applies to the white balance processing, chromatic color imparting processing, display color differentiation processing, and display color assimilation processing. It is not always necessary to store in the memory 26 from the beginning. For example, a program is first stored in an arbitrary “portable storage medium” such as a flexible disk connected to a computer, so-called FD, CD-ROM, DVD disk, magneto-optical disk, IC card or the like. Also good. Then, the computer may acquire the program from these portable storage media and execute it. Also, each program may be stored in another computer or server device connected to the computer via the Internet, a LAN (Local Area Network), etc., and the computer may acquire and execute the program from these. Good.

[Second Embodiment]
In the first embodiment, the imaging device 100 is illustrated. However, as a mobile terminal device that is a modification of the imaging device 100, for example, a mobile phone or a smartphone having a camera function, a PDA (Personal Digital Assistants), a portable game machine Etc. Hereinafter, a smartphone will be described as an example, and will be described in detail with reference to the drawings.

  FIG. 24 is a perspective view illustrating an example of the appearance of the smartphone 500. A smartphone 500 illustrated in FIG. 24 includes a flat housing 502, and a display input in which a display panel 521 serving as a display unit and an operation panel 522 serving as an input unit are integrated on one surface of the housing 502. Part 520. The housing 502 includes a speaker 531, a microphone 532, an operation unit 540, and a camera unit 541. Note that the configuration of the housing 502 is not limited thereto, and for example, a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a slide structure may be employed.

  FIG. 25 is a block diagram showing an example of the configuration of the smartphone 500 shown in FIG. As shown in FIG. 25, the main components of the smartphone 500 include a wireless communication unit 510, a display input unit 520, a call unit 530, an operation unit 540, a camera unit 541, a storage unit 550, and an external input / output. Part 560. In addition, as a main component of the smartphone 500, a GPS (Global Positioning System) receiving unit 570, a motion sensor unit 580, a power supply unit 590, and a main control unit 501 are provided. In addition, as a main function of the smartphone 500, a wireless communication function for performing mobile wireless communication via the base station device BS and the mobile communication network NW is provided.

  The radio communication unit 510 performs radio communication with the base station apparatus BS accommodated in the mobile communication network NW according to an instruction from the main control unit 501. Using this wireless communication, transmission and reception of various file data such as audio data and image data, e-mail data, and reception of Web data and streaming data are performed.

  The display input unit 520 is a so-called touch panel, and includes a display panel 521 and an operation panel 522. For this reason, the display input unit 520 displays images (still images and moving images), character information, and the like visually by controlling the main control unit 501, and visually transmits information to the user, and performs user operations on the displayed information. To detect. Note that when viewing the generated 3D, the display panel 521 is preferably a 3D display panel.

  The display panel 521 uses an LCD, an OELD (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 522 is a device that is placed so that an image displayed on the display surface of the display panel 521 is visible and detects one or a plurality of coordinates operated by a user's finger or stylus. When such a device is operated by a user's finger or stylus, a detection signal generated due to the operation is output to the main control unit 501. Next, the main control unit 501 detects an operation position (coordinates) on the display panel 521 based on the received detection signal.

  As shown in FIG. 24, the display panel 521 and the operation panel 522 of the smartphone 500 integrally form the display input unit 520, but the operation panel 522 is disposed so as to completely cover the display panel 521. ing. When this arrangement is adopted, the operation panel 522 may have a function of detecting a user operation even in an area outside the display panel 521. In other words, the operation panel 522 includes a detection area (hereinafter referred to as a display area) for an overlapping portion that overlaps the display panel 521 and a detection area (hereinafter, a non-display area) for an outer edge portion that does not overlap the other display panel 521. May be included).

  Note that the size of the display area and the size of the display panel 521 may be completely matched, but it is not always necessary to match the two. In addition, the operation panel 522 may include two sensitive regions of the outer edge portion and the other inner portion. Further, the width of the outer edge portion is appropriately designed according to the size of the housing 502 and the like. Furthermore, examples of the position detection method employed in the operation panel 522 include a matrix switch method, a resistance film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, and a capacitance method. You can also

  The call unit 530 includes a speaker 531 and a microphone 532. The call unit 530 converts the user's voice input through the microphone 532 into voice data that can be processed by the main control unit 501 and outputs the voice data to the main control unit 501. Further, the call unit 530 decodes the audio data received by the wireless communication unit 510 or the external input / output unit 560 and outputs it from the speaker 531. As shown in FIG. 24, for example, the speaker 531 can be mounted on the same surface as the display input unit 520 and the microphone 532 can be mounted on the side surface of the housing 502.

  The operation unit 540 is a hardware key using a key switch or the like, and receives an instruction from the user. For example, as illustrated in FIG. 24, the operation unit 540 is mounted on the side surface of the housing 502 of the smartphone 500 and is turned on when pressed with a finger or the like, and is turned off by a restoring force such as a spring when the finger is released. It is a push button type switch.

  The storage unit 550 stores the control program and control data of the main control unit 501, application software, address data that associates the name and telephone number of the communication partner, and transmitted / received e-mail data. In addition, the storage unit 550 stores Web data downloaded by Web browsing and downloaded content data. The storage unit 550 temporarily stores streaming data and the like. In addition, the storage unit 550 includes an external storage unit 552 having an internal storage unit 551 built in the smartphone and a removable external memory slot. Each of the internal storage unit 551 and the external storage unit 552 constituting the storage unit 550 is realized using a storage medium such as a flash memory type or a hard disk type. Other storage media include multimedia card micro type, card type memory (for example, MicroSD (registered trademark) memory, etc.), RAM (Random Access Memory), ROM (Read Only Memory). ) Can be exemplified.

  The external input / output unit 560 serves as an interface with all external devices connected to the smartphone 500, and is used to connect directly or indirectly to other external devices through communication or the like or a network. is there. Examples of communication with other external devices include universal serial bus (USB), IEEE 1394, and the like. Examples of the network include the Internet, wireless LAN, Bluetooth (Bluetooth (registered trademark)), RFID (Radio Frequency Identification), and infrared communication (Infrared Data Association: IrDA (registered trademark)). Other examples of the network include UWB (Ultra Wideband (registered trademark)) and ZigBee (registered trademark).

  Examples of the external device connected to the smartphone 500 include a memory card connected via a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, and a card socket. Other examples of the external device include a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, and an external audio / video device connected via an audio / video I / O (Input / Output) terminal. Can be mentioned. In addition to the external audio / video device, an external audio / video device that is wirelessly connected can be used. Also, instead of external audio / video devices, for example, smartphones with wired / wireless connection, personal computers with wired / wireless connection, PDAs with wired / wireless connection, personal computers with wired / wireless connection, earphones, etc. also apply Is possible.

  The external input / output unit may transmit data received from such an external device to each component inside the smartphone 500, or may allow data inside the smartphone 500 to be transmitted to the external device. it can.

  The GPS receiving unit 570 receives GPS signals transmitted from the GPS satellites ST1 to STn in accordance with instructions from the main control unit 501, performs positioning calculation processing based on the received GPS signals, and calculates the latitude of the smartphone 500 Detect the position consisting of longitude and altitude. When the GPS reception unit 570 can acquire position information from the wireless communication unit 510 or the external input / output unit 560 (for example, a wireless LAN), the GPS reception unit 570 can also detect the position using the position information.

  The motion sensor unit 580 includes, for example, a three-axis acceleration sensor and detects the physical movement of the smartphone 500 in accordance with an instruction from the main control unit 501. By detecting the physical movement of the smartphone 500, the moving direction and acceleration of the smartphone 500 are detected. This detection result is output to the main control unit 501.

  The power supply unit 590 supplies power stored in a battery (not shown) to each unit of the smartphone 500 in accordance with an instruction from the main control unit 501.

  The main control unit 501 includes a microprocessor, operates according to a control program and control data stored in the storage unit 550, and controls each unit of the smartphone 500 in an integrated manner. Further, the main control unit 501 includes a mobile communication control function for controlling each unit of the communication system and an application processing function in order to perform voice communication and data communication through the wireless communication unit 510.

  The application processing function is realized by the main control unit 501 operating according to application software stored in the storage unit 550. Application processing functions include, for example, an infrared communication function that controls the external input / output unit 560 to perform data communication with the opposite device, an e-mail function that transmits and receives e-mails, and a web browsing function that browses web pages. .

  Further, the main control unit 501 has an image processing function such as displaying video on the display input unit 520 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function is a function in which the main control unit 501 decodes the image data, performs image processing on the decoding result, and displays an image on the display input unit 520.

  Further, the main control unit 501 executes display control for the display panel 521 and operation detection control for detecting a user operation through the operation unit 540 and the operation panel 522.

  By executing the display control, the main control unit 501 displays an icon for starting application software, a software key such as a scroll bar, or a window for creating an e-mail. Note that the scroll bar refers to a software key for accepting an instruction to move a display portion of an image, such as a large image that does not fit in the display area of the display panel 521.

  Further, by executing the operation detection control, the main control unit 501 detects a user operation through the operation unit 540, or accepts an operation on the icon or an input of a character string in the input field of the window through the operation panel 522. Or In addition, by executing the operation detection control, the main control unit 501 accepts a display image scroll request through a scroll bar.

  Furthermore, by executing the operation detection control, the main control unit 501 causes the operation position with respect to the operation panel 522 to overlap with the display panel 521 (display area) or other outer edge part (non-display area) that does not overlap with the display panel 21. ) In response to the determination result, a touch panel control function for controlling the sensitive area of the operation panel 522 and the display position of the software key is provided.

  The main control unit 501 can also detect a gesture operation on the operation panel 522 and execute a preset function in accordance with the detected gesture operation. Gesture operation is not a conventional simple touch operation, but an operation that draws a trajectory with a finger or the like, designates a plurality of positions at the same time, or combines these to draw a trajectory for at least one of a plurality of positions. means.

  The camera unit 541 is a digital camera that captures an image using an image sensor such as a CMOS or a CCD, and has the same function as the image capturing apparatus 100 illustrated in FIG.

  The camera unit 541 can switch between a manual focus mode and an autofocus mode. When the manual focus mode is selected, the photographing lens of the camera unit 541 can be focused by operating a focus icon button or the like displayed on the operation unit 540 or the display input unit 520. In the manual focus mode, the live view image obtained by combining the split images is displayed on the display panel 521 so that the in-focus state during the manual focus can be confirmed. In addition, you may make it provide the smart phone 500 with the hybrid finder 220 shown in FIG.

  Further, the camera unit 541 converts image data obtained by imaging into compressed image data such as JPEG (Joint Photographic coding Experts Group) under the control of the main control unit 501. The converted image data can be recorded in the storage unit 550 or output through the input / output unit 560 or the wireless communication unit 510. In the smartphone 500 shown in FIG. 24, the camera unit 541 is mounted on the same surface as the display input unit 520, but the mounting position of the camera unit 541 is not limited to this, and the camera unit 541 may be mounted on the back surface of the display input unit 520. Alternatively, a plurality of camera units 541 may be mounted. Note that when a plurality of camera units 541 are mounted, the camera unit 541 used for imaging may be switched and imaged alone, or a plurality of camera units 541 may be used simultaneously for imaging. it can.

  The camera unit 541 can be used for various functions of the smartphone 500. For example, an image acquired by the camera unit 541 can be displayed on the display panel 521, or the image of the camera unit 541 can be used as one of operation inputs of the operation panel 522. Further, when the GPS receiving unit 570 detects the position, the position can also be detected with reference to an image from the camera unit 541. Furthermore, referring to an image from the camera unit 541, the optical axis direction of the camera unit 541 of the smartphone 500 is determined without using the triaxial acceleration sensor or in combination with the triaxial acceleration sensor. It is also possible to determine the current usage environment. Of course, the image from the camera unit 541 can be used in the application software.

  In addition, various information can be added to still image or moving image data and recorded in the storage unit 550, or can be output through the input / output unit 560 or the wireless communication unit 510. Examples of the “various information” herein include, for example, position information acquired by the GPS receiving unit 570 and image information of the still image or moving image, audio information acquired by the microphone 532 (sound text conversion by the main control unit or the like). May be text information). In addition, posture information acquired by the motion sensor unit 580 may be used.

  In each of the above embodiments, the image pickup device 20 having the first to third pixel groups is illustrated, but the present invention is not limited to this, and only the first pixel group and the second pixel group. An image sensor made of A digital camera having this type of image sensor generates a three-dimensional image (3D image) based on the first image output from the first pixel group and the second image output from the second pixel group. 2D images (2D images) can also be generated. In this case, the generation of the two-dimensional image is realized, for example, by performing an interpolation process between pixels of the same color in the first image and the second image. Moreover, you may employ | adopt a 1st image or a 2nd image as a two-dimensional image, without performing an interpolation process.

  In each of the above-described embodiments, the example in which the split image is displayed in the display area of the normal image has been described. However, the present invention is not limited to this, and the normal image (first image) is displayed on the display device. A split image (second display image) may be displayed without displaying an example of the display image. Alternatively, the split image may be displayed using the full screen. As used herein, the term “split image” refers to a case where an image sensor including only a phase difference pixel group (for example, a first pixel group and a second pixel group) is used, and a phase difference pixel at a predetermined ratio with respect to a normal pixel. When using an image sensor in which (for example, the first pixel group and the second pixel group) are arranged, an image output from the phase difference image group (for example, the first image output from the first pixel group) And a split image based on the second image output from the second pixel group. As described above, the present invention is not limited to a mode in which both the normal image and the split image are simultaneously displayed on the same screen of the display device, and the normal image is displayed in a state where the split image display is instructed. When the instruction is released, the display control unit 36 may perform control to display the split image without displaying the normal image on the display device.

12 CPU
20 Image sensor 26 Memory 28 Image processing unit 30 Normal processing unit 32 Split image processing unit 36 Display control unit 100 Imaging device 213 Display unit 247 LCD

Claims (22)

The first display is based on the image output from the image pickup device including the first and second pixel groups in which the subject image that has passed through the first and second regions in the photographing lens is divided into pupils, respectively. A first display image generator for generating an image;
A first image and a second display image generating unit that generates a second display image to be used to verify the focus on the basis of the second image output from the first and second pixel groups,
An acquisition unit that acquires color information of the first display image generated by the first display image generation unit;
A determination unit that determines a color having a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition unit as a display color of the second display image;
A display for displaying an image;
The display unit displays the first display image generated by the first display image generation unit, and the second display is displayed in a display area of the first display image. A display control unit for displaying the second display image generated by the image generation unit;
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the color having the different color characteristic is a color that can be visually recognized as having a color characteristic different from the color characteristic of the first display image.   The image processing apparatus according to claim 1, wherein the acquisition unit acquires the color information from a part or all of the first display image.   The said acquisition part acquires the said color information based on the said 1st image for a display and the said 1st image, a 2nd image, or the said 2nd image for a display. Image processing apparatus. The imaging element includes a normal imaging area used for imaging the first display image and a focus confirmation imaging area used for imaging the second display image and adjacent to the normal imaging area. Have
The said acquisition part acquires the said color information from the image output from the edge part by the side of the boundary with the said focus confirmation imaging area in the said normal imaging area among said 1st images for a display. 5. The image processing apparatus according to any one of 4.   The image processing apparatus according to claim 1, wherein the color information is color information indicating a color characteristic of an object color of the first display image.   The image processing apparatus according to claim 6, wherein the color characteristic is a hue.   The image processing apparatus according to claim 6, wherein the color having the different color characteristic is a color separated from the object color of the first display image by a predetermined angle in a hue circle.   9. The image processing apparatus according to claim 6, wherein the color characteristic of the object color of the first display image is a composition ratio of primary color components in the first display image. 10.   The image processing apparatus according to claim 9, wherein the converted composition ratio obtained by converting the composition ratio using an input / output function including a decrease function is the different color characteristic.   The image processing apparatus according to claim 9, wherein the converted composition ratio obtained by converting the composition ratio using color conversion data is the different color characteristic.   The image processing apparatus according to claim 11, wherein the color conversion data is color conversion data created based on an input / output function including a reduction function.   The image processing apparatus according to claim 1, wherein the color information is color information indicating a color characteristic of a light source color of the first display image. Each of the first and second pixel groups is a single color pixel group;
The second display image generating section, wherein the output from the first and second pixel group based on the first image and the second image, achromatic as the second display image The image processing apparatus according to claim 1, which generates an image.   When the second display image is an achromatic image, the determination unit has a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition unit. The image processing apparatus according to claim 14, wherein a chromatic color is determined as a display color of the second display image.   When the first display image and the second display image are achromatic images, the determination unit determines the color characteristics of the first display image based on the color information acquired by the acquisition unit. The image processing apparatus according to claim 1, wherein chromatic colors having different color characteristics are determined as display colors of the second display image.   The determination unit is a chromatic color having a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition unit, and the second display image has the chromatic color. The image according to any one of claims 1 to 16, wherein a chromatic color different between a region based on one image and a region based on the second image is determined as a display color of the second display image. Processing equipment.   The determination unit is based on the color information acquired by the acquisition unit when satisfying a predetermined condition for bringing the color characteristic of the first display image and the color characteristic of the second display image close to each other. The color having the color characteristic brought close to the color characteristic of the first display image by a predetermined degree is further determined as the display color of the second display image. The image processing apparatus according to claim 1. The imaging device includes a third pixel group for outputting the third image are imaged object image without being pupil split,
The first display image generation unit generates the first display image based on the third image output from the third pixel group. The image processing apparatus according to item. An image processing apparatus according to any one of claims 1 to 19,
A storage unit for storing an image output from the image sensor;
An imaging apparatus including: The first display is based on the image output from the image pickup device including the first and second pixel groups in which the subject image that has passed through the first and second regions in the photographing lens is divided into pupils, respectively. A first display image generation step for generating an image;
A second display image generating step of generating a second display image to be used to verify the focus based on the first image and the second image output from the first and second pixel groups,
An acquisition step of acquiring color information of the first display image generated by the first display image generation step ;
A determination step of determining a color having a color characteristic different from the color characteristic of the first display image based on the color information acquired by the acquisition step as a display color of the second display image;
The display unit for displaying an image displays the first display image generated by the first display image generation step, and the first display image is displayed in the display area of the first display image. A display control step for displaying the second display image generated by the two display image generation steps;
An image processing method including: Computer
The first display image generation unit, the second display image generation unit, the acquisition unit, the determination unit, and the display control in the image processing apparatus according to any one of claims 1 to 19. An image processing program for functioning as a unit.