WO2020218180A1 - Image processing device and method, and electronic apparatus - Google Patents

Abstract

This image processing device (100) is provided with: an image generation unit (103) for separating and generating a surface reflection light image (GS) and an internal reflection light image (GI) from a captured image (GRAW) on the basis of an imaging signal corresponding to the captured image of an imaging subject (OBJ) irradiated with linearly polarized light; a detection unit (104) for detecting contrast evaluation values (EVC) regarding the surface reflection light image (GS) and the internal reflection light image (GI); and a determination unit (105) for determining whether to reach a focus position regarding the reflection light image to be focused on the basis of the contrast evaluation values (EVC).

Description

Image processing equipment, methods and electronic devices

This disclosure relates to image processing devices, methods and electronic devices.

The contrast autofocus method is known as one of the conventional autofocus methods.
The contrast autofocus method is a method of focusing by measuring the contrast of an captured image in a sensor field through a lens. More specifically, in the contrast autofocus method, the focus lens is moved to move in a direction in which the contrast is higher, and the focusing control is terminated when the contrast fluctuation amount becomes equal to or less than a predetermined threshold value.

Japanese Patent No. 5952111 Japanese Unexamined Patent Publication No. 2015-141394

By the way, in the case of imaging an imaging object having a layered structure such as human skin and the irradiation light used for photographing can be transmitted inside the imaging object, the reflected light of the irradiation light. Is a mixture of surface-reflected light, which is reflected light from the surface, and internally reflected light, which is reflected light from the inside.

In this case, since the optical distance is different between the surface and the inside of the object to be imaged, the appropriate focus position is also different.
For example, when the object to be imaged is the skin, the surface of the skin transmits visible light, so that the image of the surface of the skin and the image of the inside of the skin (subcutaneous tissue) are overlapped with each other. It was. As a result, due to the influence of each other's images, it was not possible to correctly perform focusing control on either the skin surface or the inside of the skin.

Especially for the inside of the skin, the contrast on the surface of the skin is very high compared to the contrast inside the skin, so that the contrast inside the skin cannot be measured normally, and it is difficult to control the contrast autofocus.

The present disclosure has been made in view of such a situation, even when the surface of the object to be imaged has transparency and the inside can be imaged at the same time, such as the skin surface and the inside of the skin. It is an object of the present invention to provide an image processing device, a method, and an electronic device capable of correctly focusing on either the surface or the inside of an image-imaging object.

In order to achieve the above object, the image processing apparatus of the present disclosure includes a surface reflected light image and an internally reflected light image from the captured image based on the imaging signal corresponding to the captured image of the imaged object irradiated with linear polarization. The image generation unit that separates and generates the image, the detection unit that detects the contrast evaluation value for each of the surface reflected light image and the internally reflected light image, and the focus position for the reflected light image to be focused based on the contrast evaluation value. It is provided with a determination unit for determining whether or not the image has been used.

It is a principle explanatory drawing of the image processing apparatus of embodiment. It is explanatory drawing of the detection deviation of a focal point position. It is a schematic block diagram of the image processing apparatus of 1st Embodiment. It is explanatory drawing of the configuration example of one pixel which constitutes 4 polarization sensor unit. It is explanatory drawing of the reflection component calculation part of 1st Embodiment. It is explanatory drawing of an example of the user interface screen of the focus target selection part. It is an operation processing flowchart of 1st Embodiment. It is a schematic block diagram of the image processing apparatus of 2nd Embodiment. It is explanatory drawing of the reflection component calculation part of 2nd Embodiment. It is operation explanatory drawing of the contrast calculation pixel determination part. It is explanatory drawing of 3rd Embodiment. It is explanatory drawing of the absorption wavelength. It is explanatory drawing of an example of the user interface screen of the focus target selection part of 5th Embodiment. It is a schematic block diagram of the image processing apparatus of 6th Embodiment. It is explanatory drawing of an example of the display image of the preview image output part.

Next, the embodiments of the present disclosure will be described in detail with reference to the drawings.
[1] Explanation of the Principle of the Embodiment First, prior to the detailed explanation of the embodiment, the principle of the embodiment will be described.
FIG. 1 is an explanatory diagram of the principle of the image processing apparatus of the embodiment.
The image processing device 100 has a focusing lens 101 that can be driven along the optical axis direction, images an imaged object OBJ that has a predetermined vibrating surface and is irradiated with linearly polarized light, and outputs an imaged RAW image GRAW. Generated from the surface reflected light image GS generated from the component of the surface reflected light LRS of the imaged object included in the image pickup unit 102 and the polarized RAW image output by the image pickup unit 102 and the component of the internal reflected light LRI of the image pickup object OBJ. It includes an image generation unit 103 that generates and outputs the internally reflected light image GI.

Further, the image processing device 100 includes a detection unit 104 that detects the contrast evaluation value EVC for each of the surface reflected light image GS and the internally reflected light image GI, and the surface reflected light image GS and the internally reflected light image based on the contrast evaluation value EVC. A drive control signal CDR is input from the contrast AF control unit 105, which functions as a determination unit for determining whether or not the focus position has been reached for each of the GIs and controls the drive of the focusing lens 101, and the contrast AF control unit 105. It is provided with a drive unit 106 that drives the focusing lens 101 under the control of the contrast AF control unit 105.

In this case, the contrast evaluation value EVC is a value corresponding to the contrast difference (brightness difference) of the captured image, and the higher the contrast difference, that is, the larger the contrast evaluation value EVC, the more in focus (focusing). It will be).

Further, in the above description, it is determined whether or not the focus position has been reached for each of the surface reflected light image GS and the internally reflected light image GI based on the contrast evaluation value EVC. However, when focusing on only one of the surface reflected light image GS and the internally reflected light image GI, the contrast evaluation value EVC corresponding to either one of the reflected light images (for example, the surface reflected light image GS) is set. Based on this, it may be determined whether or not the focus position has been reached for the one reflected light image.

Here, the principle of the embodiment will be described.
FIG. 2 is an explanatory diagram of the detection deviation of the in-focus position.
When the polarized RAW image shown in FIG. 2 (a) is used as it is as in the conventional case, that is, when the surface reflected light image GS and the internally reflected light image GI are used without separation, as shown in FIG. 2 (b). In addition, the focus position FCSp of the surface image is affected by the internally reflected light image GI shown in FIGS. 2 (c) and 2 (d) with respect to the optimum focus position FCS, and is on the focus position side of the internally reflected light image GI. It is detected by shifting to.

Similarly, as shown in FIG. 2B, the focus position FCIp of the internal image is affected by the surface reflected light image GS shown in FIGS. 2 (e) and 2 (f) with respect to the optimum focus position FCI. Therefore, the surface reflected light image GS is detected by shifting to the focus position side, and the correct focus position cannot be obtained in either the surface image or the internal image.

In order to solve this, in the present embodiment, the surface reflected light image GS corresponding to the surface reflected light LRS and the internally reflected light image GI corresponding to the internally reflected light LRI are separated from each other.
Thereby, according to the present embodiment, the focus position can be calculated for each image, and a more suitable focus position can be calculated.

The principle of this embodiment will be specifically described below.
For example, when a human skin is used as an image pickup object OBJ and linearly polarized light (for example, linearly polarized light having a vibration surface of 90 °) is irradiated, the incident light irradiated to the skin is applied to the surface of the skin. At the same time as being reflected, it is incident on the inside of the skin, scattered inside the skin, reflected, and emitted from the surface of the skin again.

In this case, the surface reflected light LRS reflected on the surface of the skin becomes specularly reflected light and surface scattered light. These surface scattered light and specularly reflected light are polarized light.
On the other hand, the internally reflected light LRI that is incident and reflected inside the skin becomes internally scattered light by being scattered inside. This internally scattered light is unpolarized.

Therefore, for example, the polarization of four types of polarized light (for example, 0 °, 45 °, 90 °, 135 °) is detected by using a four-polarized light sensor capable of detecting each, and the irradiated linearly polarized light vibrates at 90 °. In some cases, the surface reflected light component has the largest amount of a 90 ° polarized light component (mainly a specular reflected light component). On the other hand, as for the components of the unpolarized internally scattered light, the polarized components of 0 °, 45 °, 90 °, and 135 ° appear almost evenly.

Further, in the case of the above example, the 0 ° polarization component having a vibration plane orthogonal to the 90 ° polarization component is subtracted from the 90 ° polarization component, so that the 90 ° polarization component is the surface reflected light. The ingredients will remain mainly.

Therefore, in the image generation unit 103, the surface reflected light image (specular reflected light image + surface scattered image) GS is generated by generating an image from the pixel data corresponding to the polarization component of 90 °, and the polarization of 0 ° is performed. An internally reflected light image (internally scattered image) GI is generated by generating an image from pixel data corresponding to the components.
Then, the detection unit 104 outputs the contrast evaluation value EVC for each of the surface reflected light image GS and the internally reflected light image GI.

As a result, the contrast AF control unit 105 determines whether or not the focus position has been reached based on the contrast evaluation value EVC, and has a higher contrast image for each of the surface reflected light image GS and the internally reflected light image GI. Is the in-focus image.

That is, the contrast AF control unit 105 has a plurality of different focal positions obtained by driving the focusing lens 101 via the drive unit 106 for each of the surface reflected light image GS and the internally reflected light image GI. Of the images, the image with higher contrast is assumed to be the in-focus image. Therefore, the optimum focus position can be detected for each of the surface reflected light image GS and the internally reflected light image GI, and the optimum focused image can be obtained for each.

In other words, according to the principle of the present embodiment, it is possible to prevent the surface reflected light image GS and the internally reflected light image GI from interfering with each other and shifting the focus position from the optimum focus position.

On the other hand, when the surface reflected light image GS and the internally reflected light image GI cannot be separated as in the conventional case, the contrast is relatively higher than that of the surface reflected light image GS and the internally reflected light image GI, so that the inner surface image The contrast of the image cannot be measured correctly, and the accuracy of the contrast autofocus is naturally lowered.
However, according to the principle of the present embodiment, as described above, the contrast can be correctly detected even in the internally reflected light image GI, and the optimum focused image can be obtained.

[2] First Embodiment First, the image processing apparatus of the first embodiment will be described.
FIG. 3 is a schematic block diagram of the image processing apparatus of the first embodiment.
The image processing device 10 is roughly classified into an image imaging unit 11, a reflection component calculation unit 12, a focus target selection unit 13, a contrast detection unit 14, a contrast storage unit 15, and a contrast autofocus (AF) control unit. A focus lens driving unit 17 and a focus lens driving unit 17 are provided.

The image capturing unit 11 has a focusing lens 21 and a 4-polarized sensor unit 22, images an image-imaging object OBJ (human skin in this embodiment), and reflects a polarized RAW image (data) GRAW as a reflection component. Output to the calculation unit 12.

The reflection component calculation unit 12 separates and generates a surface reflected light image (data) GS and an internally reflected light image (data) GI based on the input polarized RAW image (data) GRAW, and causes the focus target selection unit 13 to perform the process. Output.

In this case, the surface reflected light image GS includes a specular reflected light image due to specular reflection on the surface of the imaged object OBJ of the irradiated linearly polarized light (for example, linearly polarized light having a vibration surface of 90 °) and the imaged object. It is a composite image of the surface scattering image due to surface scattering.

Further, the internally reflected light image GI is an internally scattered image in which the linearly polarized light incident on the inside of the imaging object OBJ is almost unpolarized due to internal scattering.

The focus target selection unit 13 determines the surface reflected light image based on whether the user of the image processing apparatus selects the surface reflected light image or the internally reflected light image, that is, which is the reflected light image to be focused. Either GS or the internally reflected light image GI is output to the contrast detection unit 14.

The contrast detection unit 14 is based on either the surface reflected light image GS or the internally reflected light image GI output by the focus target selection unit 13 and the user's area designation information, and the current frame image (surface reflected light image GS). Alternatively, the contrast evaluation value EVC of any one of the internally reflected light image GI) is output to the contrast AF control unit 16.

The contrast storage unit 15 stores a predetermined number of contrast evaluation values EVC output by the contrast detection unit 14 in chronological order.

The contrast autofocus (AF) control unit 16 compares the difference between the contrast evaluation value EVC of the current frame image and the contrast evaluation value EVC of the previous frame image with a predetermined threshold value (contrast difference threshold value), and presents the difference. When the difference between the contrast evaluation value EVC of the frame image and the contrast evaluation value EVC of the previous frame is equal to or less than a predetermined threshold value, the current frame image is focused on the assumption that a more suitable focus position has been reached. The image is a polarized RAW image.

The focus lens driving unit 17 focuses the image imaging unit 11 when the difference between the contrast evaluation value EVC of the current frame image and the contrast evaluation value EVC of the previous frame still exceeds a predetermined threshold value. The lens 21 is driven to change the focus position and to perform reimaging.

Here, the configuration of the image capturing unit will be described.
The image capturing unit 11 includes a 4-polarization sensor unit 22 as described above.
The four polarization sensor unit 22 receives a first sensor that receives polarized light having a 0 ° vibrating surface, a second sensor that receives polarized light having a 45 ° vibrating surface, and a polarized light having a 90 ° vibrating surface. A third sensor and a fourth sensor that receives polarized light having a vibration surface of 135 ° are arranged in two dimensions in association with each color component of R, G, and B.

In the above configuration, four types of sensors having different vibration planes by 45 ° are provided, but at least three types of polarized images having different vibration planes (including unpolarized light) can be obtained, and at least one pair of vibration planes are orthogonal to each other. If it is configured, it can be applied in the same way.

More specifically, the 4-polarization sensor unit 22 adopts a configuration in which a plurality of (four in the figure) polarizing filters having a pixel configuration in the polarization direction are arranged on an image sensor provided with a color mosaic filter on the imaging surface. ing.

Hereinafter, a specific description will be given.
FIG. 4 is an explanatory diagram of a configuration example of one pixel constituting the 4-polarization sensor unit.
In the example of FIG. 4, a color mosaic filter having an area ratio of R: G: B of 1: 2: 1 is used for one pixel of the image.

The R filter FR includes a light receiving element DR1 in which a polarizing filter having a vibrating surface of 0 ° is arranged on the incident surface, a light receiving element DR2 in which a polarizing filter having a vibrating surface of 45 ° is arranged on the incident surface, and a vibrating surface of 90. The light receiving element DR3 in which the polarizing filter of ° is arranged on the incident surface and the light receiving element DR4 in which the polarizing filter having the vibration surface of 135 ° are arranged on the incident surface correspond.

Similarly, the G filter FG1 includes a light receiving element DG11 in which a polarizing filter having a vibrating surface of 0 ° is arranged on the incident surface, a light receiving element DG12 in which a polarizing filter having a vibrating surface of 45 ° is arranged on the incident surface, and a vibrating surface of 90. The light receiving element DG13 in which the polarizing filter of ° is arranged on the incident surface and the light receiving element DG14 in which the polarizing filter having the vibration surface of 135 ° are arranged on the incident surface correspond.

Further, the G filter FG2 includes a light receiving element DG21 in which a polarizing filter having a vibrating surface of 0 ° is arranged on the incident surface, a light receiving element DG22 in which a polarizing filter having a vibrating surface of 45 ° is arranged on the incident surface, and a vibrating surface of 90. The light receiving element DG23 in which the polarizing filter of ° is arranged on the incident surface and the light receiving element DG24 in which the polarizing filter having the vibration surface of 135 ° are arranged on the incident surface correspond.

Further, the B filter FB includes a light receiving element DB1 in which a polarizing filter having a vibrating surface of 0 ° is arranged on the incident surface, a light receiving element DB2 in which a polarizing filter having a vibrating surface of 45 ° is arranged on the incident surface, and a vibrating surface of 90. The light receiving element DB3 in which the polarizing filter of ° is arranged on the incident surface and the light receiving element DB4 in which the polarizing filter having the vibration surface of 135 ° are arranged on the incident surface correspond.

Then, in the 4-polarization sensor unit 22, light-receiving elements having the number of pixels of the image sensor x 4 are two-dimensionally arranged.

The configuration of the above polarized light sensor unit (4-polarized light sensor unit) is an example, and the surface reflected light LRS (= specular reflected light + surface scattered light [polarized light]) and the internally reflected light LRI (= internally scattered light [unpolarized light]). ]) And any configuration that can be separated can be adopted.

For example, it may be provided with two types of light receiving elements corresponding to at least two types of polarizing filters whose vibration planes are orthogonal to each other, and further provided with a light receiving element without a polarizing filter.

Further, in the example of FIG. 4, a plurality of light receiving elements (four light receiving elements in the example of FIG. 4) correspond to one filter, but it is also possible to configure so that a filter is assigned to each light receiving element. Is.

Next, the reflection component calculation unit will be described.
FIG. 5 is an explanatory diagram of the reflection component calculation unit of the first embodiment.
In FIG. 5, for the sake of easy understanding, only the configuration of one pixel of the imaged RAW image RAW, and by extension, one pixel of the surface reflected light image GS or the internally reflected light image GI is shown.

The reflection component calculation unit 12 interpolates the polarization component based on the imaged RAW image GRAW and separates the reflection component from the polarization component interpolation unit 31 that generates the interpolation 4-polarized image GFC described later and the interpolation 4-polarized image GFC. , A reflection component separating unit 32 that generates a surface reflected light image GS and an internally reflected light image GI.

First, the operation of the polarization component interpolation unit 31 will be described.
In the case of the example of the 4-polarization sensor unit 22 of FIG. 4, in the imaged RAW image GRAW, the first R polarization component R1 having a vibration surface of 0 °, the second R polarization component R2 having a vibration surface of 45 °, and the vibration surface have an R component. A 90 ° thirty-one R polarization component R3 and a fourth R polarization component R4 having a vibration plane of 135 ° are included.

Similarly, in the imaged RAW image GRAW, regarding the G component, the first G polarization component G11 and the first G polarization component G21 having a vibration surface of 0 °, the second G polarization component G12 and the second G polarization component G22 having a vibration surface of 45 °, and vibration. A third G polarizing component G13 and a third G polarizing component G23 having a surface of 90 °, and a fourth G polarizing component G14 and a fourth G polarizing component G24 having a vibration surface of 135 ° are included.

Further, in the imaged RAW image GRAW, regarding the B component, the first B polarizing component B1 having a vibration surface of 0 °, the second B polarization component B2 having a vibration surface of 45 °, the third B polarization component G3 having a vibration surface of 90 °, and the vibration surface. Includes a fourth B polarization component B4 at 135 °.

Therefore, in the imaged RAW image GRAW, the polarization component interpolation unit 31 of the reflection component calculation unit 12 has a first polarized image RG1 in which all one pixel in the image is the first R polarization component R1 and one pixel in the image regarding the R component. The second polarized image RG2, which is all the second R polarization component R2, the third polarized image RG3, where one pixel in the image is all the third R polarization component R3, and the third polarized image RG3, where all one pixel in the image is the fourth R polarization component R4. A 4-polarized image RG4 is generated.

Further, since the polarization component interpolation unit 31 is arranged with the polarization filters corresponding to the two G filters for the G component in the imaged RAW image GRAW, for example, all one pixel in the image is the first G polarization component G11 and the first G polarization component G11. The first polarized image GG1 in which the average value G1 {= (G11 + G21) / 2} of the 1G polarized component G21, and one pixel in the image is the average value G2 {= (G12 + G22) of the second G polarized component G12 and the second G polarized component G22. ) / 2}, the second polarized image GG2, and the third polarized image GG3 in which one pixel in the image is the average value G3 {= (G13 + G23) / 2} of the third G polarized component G13 and the third G polarized component G23. And the 4th G polarized light image GG4 in which one pixel in the image is the average value G4 {= (G14 + G24) / 2} of the 4th G polarized light component G14 and the 4th G polarized light component G24 is generated.

Further, in the imaged RAW image GRAW, the polarization component interpolation unit 31 has a first polarized image BG1 in which all one pixel in the image is the first B polarization component B1 and one pixel in the image is the second B polarization component B2. Generates the second polarized image BG2, the third polarized image BG3 in which all one pixel in the image is the third B polarized component B3, and the fourth polarized image BG4 in which all one pixel in the image is the fourth B polarized component B4. To do.

Further, in the reflection component separation unit 32 of the reflection component calculation unit 12, when the irradiated linearly polarized light has a vibrating surface of 90 °, one pixel in the image corresponding to the vibrating surface orthogonal to the vibrating surface is 0 °. The first polarized image RG1 which is all the first R polarization component R1, and the first polarization where one pixel in the image is the average value G1 {= (G11 + G21) / 2} of the first G polarization component G11 and the first G polarization component G21. The internally reflected light image GI is generated based on the first polarized image BG1 in which the image GG1 and one pixel in the image are all the first B polarized component B1.

Further, in the reflection component separation unit 32, one pixel in the image corresponding to 0 ° from the third polarized image RG3 in which all one pixel in the image corresponding to the vibrating surface 90 ° is the third R polarizing component R3 is From the R image obtained by subtracting the value of the first polarized image RG1 all set to the first R polarized component R1, and the third polarized image GG3 in which one pixel in the image corresponding to the vibration surface of 90 ° is all set to the third G polarized component G3. All 1 pixel in the image corresponding to the vibrating surface 0 ° is the first G image obtained by subtracting the value of the first polarized image GG1 in which the first G polarizing component G1 is set, and all 1 pixel in the image corresponding to the vibrating surface 90 ° is the first. Based on the B image obtained by subtracting the value of the first polarized image BG1 in which all 1 pixel in the image corresponding to the vibration plane corresponding to 0 ° is subtracted from the third polarized image BG3 having the 3B polarized component B3. A surface reflected light image GS is generated.

Next, the focus target selection unit 13 will be described.
FIG. 6 is an explanatory diagram of an example of a user interface screen of the focus target selection unit.
In the example of FIG. 6, the image display area AR1 for displaying the image of the portion currently focused on is displayed on the display screen of the touch panel display, and the focus position is the surface of the imaging object (skin in this example). The surface selection button BT1 and the internal selection button BT2 for setting the focus position inside the image pickup object are arranged.

In this case, the image display area AR1 displays information (“surface” in the example of FIG. 6) indicating that the current focus target is either the surface or the inside.

That is, when the user touches the surface selection button BT1, the image of the surface of the skin, which is the object to be imaged, is displayed in the image display area AR1. When the user touches the internal selection button BT2 while the image of the surface of the skin is displayed, the image of the inside of the skin, which is the object to be imaged, is displayed in the image display area AR1.

Further, when the user touches the surface selection button BT1 again while the image of the inside of the skin is displayed, the image of the surface of the skin, which is the object to be imaged, is displayed again in the image display area AR1.
Then, the display of the image on the last selected side is continued until the user switches the display target.

Next, the contrast detection unit 14 will be described.
In this example, it is assumed that the user can specify the contrast detection area in the image display area AR1 of the touch panel display.

If the user does not specify the contrast detection area, the preset (default) area is specified. The preset area is not particularly limited, and may be the entire area corresponding to the image display area AR1, a predetermined rectangular area in the center of the image display area AR1, or the like.

Even in this case, once the contrast detection area is designated by the user, the designated contrast detection area remains valid until the user again specifies the contrast detection area.
Then, the contrast detection unit 14 calculates the brightness L for each pixel of the image corresponding to the designated contrast detection region, and extracts the maximum brightness Lmax which is the maximum value of the brightness L and the Lmin which is the minimum value of the brightness L. ..

Here, the brightness L is calculated by, for example, the following equation.
When the value of the R component of the pixel for which the brightness L is calculated is R, the value of the G component is G, and the value of the B component is B, the case of the example of FIG. 4 is as follows.
L = (R + 2G + B) / 4

Then, the contrast detection unit 14 calculates the contrast evaluation value EVC from the extracted maximum luminance value Lmax and minimum luminance value Lmin by the following equation.
EVC = (Lmax-Lmin) / (Lmax + Lmin)

Then, the contrast detection unit 14 outputs the calculated contrast evaluation value EVC to the contrast storage unit 15 and the contrast AF control unit 16.
As a result, the contrast storage unit 15 stores the contrast evaluation value EVC in chronological order (input order) (step S16).

Next, the contrast AF control unit 16 will be described.
The contrast AF control unit 16 determines whether or not the focus position has been reached based on the difference between the contrast evaluation value EVC of the image corresponding to the current focus position and the contrast evaluation value EVC of the image corresponding to the previous focus position. If the determination is made and the focus position has not been reached, the focus lens driving unit 17 is controlled to change the focus position and perform imaging again.

Next, the outline operation of the first embodiment will be described.
FIG. 7 is an operation processing flowchart of the first embodiment.
First, the contrast AF control unit 16 sets the initial setting of the driving direction of the focusing lens 21 and the initial contrast evaluation value EVC used for determining the end of contrast AF (step S11).

In this case, the driving direction of the focusing lens 21 may be stored in advance in a memory (not shown).
Further, the initial contrast evaluation value EVC may be stored in advance in the contrast storage unit 15.

Next, the image capturing unit 11 images the imaged object OBJ and outputs the captured RAW image GRAW to the reflection component calculation unit 12 (step S12).
The reflection component calculation unit 12 calculates the reflection component, generates a surface reflected light image GS and an internally reflected light image GI, and outputs the reflected light image GS to the focus target selection unit 13 (step S13).

Subsequently, the user selects the focus target (surface or inside) and specifies the contrast detection target area (step S14).

As a result, the focus target selection unit 13 outputs the reflected light image (either the surface reflected light image GS or the internal reflected light image GI) of the focus target selected by the user to the contrast detection unit 14. As a result, the contrast detection unit 14 performs contrast detection on the input reflected light image in the contrast detection region designated by the user.

Then, the contrast detection unit 14 outputs the detected contrast evaluation value EVC to the contrast storage unit 15 and the contrast AF control unit 16 (step S15).
As a result, the contrast storage unit 15 stores the contrast evaluation value EVC (step S16).

Subsequently, the contrast AF control unit 16 obtains the contrast evaluation value EVC of the image corresponding to the input current focus position and the contrast evaluation value EVCp of the image corresponding to the previous focus position read from the contrast storage unit 15. Comparison (calculation of difference) is performed (step S17).

The contrast AF control unit 16 determines whether or not the difference between the calculated contrast evaluation values EVC is equal to or less than a predetermined threshold value (step S18).

In the determination in step S18, if the difference in the calculated contrast evaluation value EVC is equal to or less than a predetermined threshold value (step S18; Yes), the contrast AF control unit 16 terminates the process assuming that it is in focus.

In the determination in step S18, when the difference between the calculated contrast evaluation values EVC exceeds a predetermined threshold value, the image corresponding to the current focus position is more than the contrast evaluation value EVCp of the image corresponding to the previous focus position. It is determined whether or not the contrast evaluation value EVC is higher (step S19).

In the determination in step S19, if the contrast evaluation value EVC of the image corresponding to the current focus position is higher than the contrast evaluation value EVCp of the image corresponding to the previous focus position (step S19; Yes), the contrast AF The control unit 16 controls the focus lens drive unit 17 to drive the focusing lens 21 in the same direction as the previous time (step S20), and shifts the process to step S12 again to perform imaging (step S12). ..

In the determination in step S19, when the contrast evaluation value EVC of the image corresponding to the current focus position is lower than the contrast evaluation value EVCp of the image corresponding to the previous focus position (step S19; No), the contrast AF The control unit 16 controls the focus lens drive unit 17 to drive the focusing lens 21 in the direction opposite to the previous time (step S21), and shifts the process to step S12 again to perform imaging (step S12). ..

As described above, according to the first embodiment, the surface reflected light image GS and the internally reflected light image GI are separated and generated from the imaged RAW image GRAW obtained by irradiating the imaged object OBJ with linear polarization. Since the contrast AF control is performed separately for each of the above, high-precision autofocus control can be performed without one being affected by the other and the autofocus accuracy being lowered.

[3] Second Embodiment Next, the second embodiment will be described.
FIG. 8 is a schematic block diagram of the image processing apparatus of the second embodiment.
In FIG. 8, the same parts as those in the first embodiment of FIG. 3 are designated by the same reference numerals, and the detailed description thereof shall be incorporated.

The difference between the image processing device 10A of the second embodiment and the image processing device 10 of the first embodiment is that the reflection component calculation unit 12 calculates the degree of polarization (degree of polarization image Gρ) and the area is specified by the user. A point including a contrast calculation pixel determination unit 18 for determining a contrast calculation pixel based on a degree of polarization (degree of polarization image Gρ) and a reflection component image to be focused.

First, the polarization degree calculation operation of the reflection component calculation unit of the second embodiment will be described.
FIG. 9 is an explanatory diagram of the reflection component calculation unit of the second embodiment.
The reflection component calculation unit 12 of the second embodiment interpolates the polarization component based on the imaged RAW image GRAW and generates the interpolation 4-polarized image GFC described later. Based on the polarization component interpolation unit 31 and the interpolation 4-polarized image GFC. It is provided with a reflection component separation unit 32 that separates reflection components and generates a polarization degree image Gρ, a surface reflected light image GS, and an internally reflected light image GI.
Here, since the operation of the polarization component interpolation unit 31 is the same as that of the first embodiment, the operation of the reflection component separation unit 32 will be described.

In the reflection component separation unit 32 of the reflection component calculation unit 12, when the irradiated linearly polarized light has a vibrating surface of 90 °, all one pixel in the image corresponding to the vibrating surface orthogonal to the vibrating surface is 0 °. The first polarized image RG1 as the first R polarized component R1, and the first polarized image in which one pixel in the image is the average value G1 {= (G11 + G21) / 2} of the first G polarized component G11 and the first G polarized component G21. An internally reflected light image GI is generated based on the first polarized image BG1 in which the GG1 and one pixel in the image are all the first B polarized component B1.

Further, in the reflection component separation unit 32, one pixel in the image corresponding to 0 ° from the third polarized image RG3 in which all one pixel in the image corresponding to the vibrating surface 90 ° is the third R polarizing component R3 is From the R image obtained by subtracting the value of the first polarized image RG1 all set to the first R polarized component R1, and the third polarized image GG3 in which one pixel in the image corresponding to the vibration surface of 90 ° is all set to the third G polarized component G3. All 1 pixel in the image corresponding to the vibrating surface 0 ° is the first G image obtained by subtracting the value of the first polarized image GG1 in which the first G polarizing component G1 is set, and all 1 pixel in the image corresponding to the vibrating surface 90 ° is the first. Based on the B image obtained by subtracting the value of the first polarized image BG1 in which all 1 pixel in the image corresponding to the vibration plane corresponding to 0 ° is subtracted from the third polarized image BG3 having the 3B polarized component B3. A surface reflected light image GS is generated.

Further, the reflection component separation unit 32 calculates the degree of polarization ρ of all pixels and generates the degree of polarization image Gρ.
Here, the degree of polarization ρ can be calculated by the equation (1).

Here, the parameters a, b, and c are expressed as follows from the brightness values I ₀ , I ₄₅ , I ₉₀ , and I ₁₃₅ of the four polarized light having vibration planes of 0 °, 45 °, 90 °, and 135 °. To.

Then, a polarization degree image Gρ is generated from all the obtained polarization degree ρ and output to the contrast calculation pixel determination unit 18.

In the above processing, the luminance values I ₀ , I ₄₅ , I ₉₀ , and I ₁₃₅ are determined by the equation (5) from the R component value R, G component value G, and B component value B of the corresponding pixels.
I = (R + 2G + B) / 4 ... (5)

FIG. 10 is an operation explanatory view of the contrast calculation pixel determination unit.
When focusing on the surface of the skin to be imaged, that is, when the main component of the light component input to the image capturing unit is a specular reflection component, a portion having an extremely high luminance level or polarization degree level, for example, FIG. As shown in (a), the portion having a large amount of fat and water is a portion having a so-called “shine” and a high degree of brightness or polarization.

FIG. 10B is an explanatory diagram of the brightness value or the degree of polarization of the pixels used for contrast detection.
As shown in FIG. 10B, pixels having a brightness value or a degree of polarization exceeding a predetermined threshold value should not be used for contrast detection and are therefore excluded. Then, contrast is performed using only the remaining components excluding the pixels having the brightness value or the degree of polarization exceeding the predetermined threshold value (the component corresponding to the pixel having the brightness value or the degree of polarization located on the left side of the threshold value in the figure). It is desirable to perform detection.

Therefore, the contrast calculation pixel determination unit 18 is a target pixel mask image GM for masking pixels to be excluded from contrast detection based on the degree of polarization, the region, and a predetermined threshold value input from the surface reflected light image GS. To generate.

FIG. 10C is an explanatory diagram of an example of the target pixel mask image.
In FIG. 10 (c), the pixels excluded from the contrast detection are displayed in white (1), and the pixels used for the contrast detection are displayed in black (0) to generate a target mask image GM which is a binary image.

As a result, the light that is irradiated with fat or moisture and reflected as it is, that is, the pixel having high brightness or the degree of polarization (the pixel corresponding to the white portion in the case of FIG. 10C) has the contrast value in autofocus. Contrast AF can be performed more reliably without affecting the calculation.
Therefore, according to the second embodiment, more accurate contrast AF can be realized as compared with the first embodiment.

[4] Third Embodiment The difference between this third embodiment and the second embodiment described above is that the detection accuracy is obtained by performing contrast detection using a region that is likely to be in focus at the time of contrast detection. It is a point that is further improved.

FIG. 11 is an explanatory diagram of the third embodiment.
The human body (arms, face) has an uneven shape, and the imaging region includes a region that is in focus as shown by an ellipse in FIG. 11 (a). As shown in FIG. 11A as a region outside the ellipse, it is natural that unfocused regions are mixed. When the object to be imaged is like an arm, the cross section has a substantially elliptical shape, so if it is focused on any position, it will shift from the focus position as the distance from the focused position increases along the elliptical direction. Because it goes.

Therefore, in the third embodiment, when focusing on the surface of the skin, as shown in FIG. 11B, the region where the texture of the skin is detected by image processing such as edge detection of the captured image ( The area where the linear edge is detected) is set as the contrast detection target area. As a result, the contrast AF processing is performed faster and more reliably than in the case where the entire captured image shown in FIG. 11A is processed including the unfocused region as shown in FIG. 11C. Can be done.

In this case, as a skin texture detection method, for example, an image processing method as shown in JP2013-188341A can be mentioned, and the skin texture can be detected quickly and easily.

[5] Fourth Embodiment The contrast calculation pixel determination unit of the second embodiment and the third embodiment described above aims to improve the accuracy of the contrast AF on the surface of the object to be imaged (in the case of the above example, the skin surface). However, the fourth embodiment aims to improve the accuracy of the contrast AF inside the image pickup object.

When applying the fourth embodiment, it is premised that a substance that does not exist on the surface inside the image pickup object and has a large light absorption rate in a specific wavelength band exists.

That is, when human skin is the object to be imaged, the stain (melanin) inside the skin has a large light absorption rate in a specific wavelength band, and by utilizing this, in a specific wavelength band. By increasing the weight of the detection result, it is possible to improve the accuracy of the contrast AF.

FIG. 12 is an explanatory diagram of the absorption wavelength.
For example, in the case of an RGB sensor, as shown in FIG. 12, by weighting the output signal of a B sensor having a short wavelength (for example, a wavelength of 380 nm), the inside of the skin such as hemoglobin and melanin (measurement target). It is possible to perform autofocus sensitively to the area where a specific substance exists (inside the object).
As a result, it is possible to improve the accuracy of the contrast AF inside the image pickup object.

[6] Fifth Embodiment Each of the above embodiments is for focusing on either the surface of the skin to be imaged or a predetermined inside, but the fifth embodiment is a user operation. This is an embodiment in which the focus can be arbitrarily set within a predetermined internal range from the surface.

First, the focus target selection unit of the fifth embodiment will be described.
FIG. 13 is an explanatory diagram of an example of the user interface screen of the focus target selection unit of the fifth embodiment.
In the example of FIG. 13, the image display area AR1 for displaying the image of the portion currently focused on is displayed on the display screen of the touch panel display, and the focus position is the image target from the surface of the image target object (skin in this example). A slide bar SL for making it arbitrary within a predetermined position (predetermined depth) range inside the object is arranged.

In this case, the slide bar SL is provided with a slide button SLB whose display position can be changed to the left and right.
Therefore, the user moves the slide button SLB in the left-right direction so that the image displayed in the image display area AR1 corresponds to the desired focus position.

In this case, in terms of internal processing, when the value of the pixel constituting the surface reflected light image GS is Is and the value of the pixel constituting the internally reflected light image GI is Id, the pixel of the image used for contrast detection is used. The value I of is synthesized based on the equation (6).
I = α ・ Id + (1-α) ・ Is… (6)

Here, when 0 ≦ α ≦ 1 and α = 0, that is, when the slide button SLB is at the right end of the slide bar SL in FIG. 12, the image display area AR1 has the values of all the pixels on the surface. The surface reflected light image GS composed of the values Is of the pixels constituting the reflected light image GS is displayed.

Further, when α = 1, that is, in FIG. 13, when the slide button SLB is at the left end of the slide bar SL, the values of all the pixels in the image display area AR1 are the pixels constituting the internally reflected light image GI. The internally reflected light image GI is displayed with the value Id.
Then, until the user switches the display target, the display of the image corresponding to the value of α corresponding to the position of the last selected slide button SLB is continued.

As described above, according to the fifth embodiment, it is possible to set an arbitrary position as the focus position within a range from the surface of the imaging object to a predetermined internal position.

In the above description, the case where the focus position is changed according to the position of the slide button SLB on the slide bar SL has been described, but the user interface of the focus target selection unit is not limited to this, and various users are not limited to this. It is possible to use an interface.

For example, when the touch panel display is pressure-sensitive, the focus position can be changed according to the pressing pressure, or the focus position can be changed by directly inputting a number (for example, 0 to 10, 0 to 100) using the numeric keypad. It is possible to change the focus position by selecting one of a plurality of buttons (for example, 10 buttons).

[7] Sixth Embodiment The difference between this sixth embodiment and each of the above embodiments is that the preview image for displaying both the surface reflected light image GS and the internally reflected light image GI output by the reflection component calculation unit is displayed. The point is that it has an output unit.
FIG. 14 is a schematic block diagram of the image processing apparatus of the sixth embodiment.
In FIG. 14, the same parts as those in the first embodiment of FIG. 3 are designated by the same reference numerals.
In the preview image output unit 19, when the reflection component calculation unit generates the surface reflected light image GS and the internally reflected light image GI and outputs them, those images are displayed on one screen.

FIG. 15 is an explanatory diagram of an example of a display image of the preview image output unit.
As shown in FIG. 15A, the preview image output unit displays both the surface reflected light image GS and the internally reflected light image GI on the display screen.
Therefore, the user can easily select the image to be focused by touching any of the images.

Further, as shown in FIG. 15B, in addition to the surface reflected light image GS and the internally reflected light image GI, the mask image GM described in the fourth embodiment, the polarization degree image Gρ described in the fifth embodiment, and the like are also included. It is also possible to display it as a preview image as needed.

By adopting such a configuration, it is possible to use it as a reference image for the user to specify the area, or to give information about the cause of the autofocus not being as desired by the user.

Further, in the above description of the sixth embodiment, both the surface reflected light image GS and the internally reflected light image GI are displayed as the preview image, but only one of them is displayed at the time of the main shooting. It is also possible to have a configuration in which both the surface reflected light image GS and the internally reflected light image GI are automatically acquired.

Furthermore, as the preview image, it is possible to use a normal image with normal lighting instead of imaging with polarized lighting.

[8] Modification Example of the Embodiment In the above description, the case where the object to be imaged is the skin has been described as an example, but the illumination light which is polarized light is not only reflected on the surface but also transmitted and scattered inside. The same can be applied to an imaged object such as a food (fruit, etc.).

Further, it is possible to apply not only to a two-layer structure but also to a multi-layer structure on the surface and any layer (including a layer on the more surface side).

In the above description, an image processing device that uses only contrast AF has been described, but it can also be applied to a combination with image plane phase difference AF, which is the mainstream autofocus method for many image pickup devices.

Further, in the above description, the surface reflected light image GS and the internally reflected light image GI are treated as separate images, but by synthesizing both images, all the images are in focus on both the surface and the inside. It is also possible to generate a store image.

Note that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

In addition, this technology can also adopt the following configurations.
(1)
An image generation unit that separates and generates a surface reflected light image and an internally reflected light image from the captured image based on an imaging signal corresponding to the captured image of the imaged object irradiated with linear polarization.
A detection unit that detects a contrast evaluation value for each of the surface reflected light image and the internally reflected light image,
A determination unit that determines whether or not the reflected light image to be focused has reached the focus position based on the contrast evaluation value.
Image processing device equipped with.
(2)
An imaging unit having a focusing lens, imaging the object to be imaged, and outputting the imaging signal.
A drive unit for driving the focusing lens to change the focus position based on the determination result of the determination unit is provided.
(1) The image processing apparatus according to the above.
(3)
The image generation unit generates the surface reflected light image based on the polarization component obtained by removing the polarization component orthogonal to the linear polarization from the linearly polarized light component, and the inside is based on the polarization component orthogonal to the linear polarization. Generate a reflected light image,
The image processing apparatus according to (1) or (2).
(4)
An image region in which the detection unit detects the contrast evaluation value corresponding to the surface reflected light image in a region where the brightness level exceeds a predetermined first threshold value or a region where the polarization degree level exceeds a predetermined second threshold value. With a decision part to decide as,
The image processing apparatus according to any one of (1) to (3).
(5)
The imaging object is human skin,
The detection unit includes a determination unit that determines the texture region in which the texture of the skin is detected as an image region in which the contrast evaluation value corresponding to the surface reflected light image is detected in the detection unit.
The image processing apparatus according to any one of (1) to (3).
(6)
When the detection unit contains a substance having a relatively high absorption rate in a specific wavelength band inside the image pickup object, the weight of the wavelength band is increased and the contrast evaluation value is obtained for the internally reflected light image. To detect,
The image processing apparatus according to any one of (1) to (5).
(7)
The surface reflected light image and the internally reflected light image are input, and a selection unit is provided which outputs either the surface reflected light image or the internally reflected light image to the detection unit based on the designated input.
The image processing apparatus according to any one of (1) to (6).
(8)
The surface reflected light image and the internally reflected light image are input, and the surface reflected light image and the internally reflected light image are combined by changing the ratio based on the designated input, and the combined reflected light image is a contrast evaluation value in the detection unit. Equipped with a selection unit that outputs
The image processing apparatus according to any one of (1) to (6).
(9)
An image output unit for displaying the surface reflected light image and the internally reflected light image is provided.
The image processing apparatus according to any one of (1) to (8).
(10)
A method performed by an image processor
A process of separating and generating a surface reflected light image and an internally reflected light image from the captured image based on an imaging signal corresponding to the captured image of the imaged object irradiated with linear polarization.
The process of detecting the contrast evaluation value for each of the surface reflected light image and the internally reflected light image, and
The process of determining whether or not the reflected light image to be focused has reached the focus position based on the contrast evaluation value, and
A method equipped with.
(11)
An imaging unit that has a focusing lens, images an imaged object irradiated with linear polarization, and outputs an imaging signal.
An image generation unit that separates and generates a surface reflected light image and an internally reflected light image from the captured image based on the image pickup signal corresponding to the captured image of the object to be imaged.
A detection unit that detects a contrast evaluation value for each of the surface reflected light image and the internally reflected light image,
A determination unit that determines whether or not the reflected light image to be focused has reached the focus position based on the contrast evaluation value.
A drive unit that drives the focusing lens to change the focus position based on the determination result of the determination unit.
Electronic equipment equipped with.

10, 10A Image processing device 11 Image imaging unit 12 Reflection component calculation unit 13 Focus target selection unit 14 Contrast detection unit 15 Contrast storage unit 16 Contrast AF control unit 17 Focus lens drive unit 18 Contrast calculation pixel determination unit 19 Preview image output unit 21 Focusing lens 22 4 Polarization sensor unit 31 Polarization component interpolation unit 32 Reflection component separation unit 100 Image processing device 101 Focusing lens 102 Imaging unit 103 Image generation unit 104 Detection unit 105 Contrast AF control unit 106 Drive unit AR1 Image display area CDR drive control signal EVC, EVCp Contrast evaluation value FCI Optimal focus position FCIp Focus position FCS Optimal focus position FCSp Focus position GFC Interpolation 4 Polarized image GI Internal reflected light image GM Target pixel mask image GRAW Imaging RAW image GS Surface reflected light image Gρ Degree Image LRI Internally reflected light LRS Surface reflected light Lmax Maximum brightness Lmin Minimum brightness OBJ Object to be imaged SL Slide bar SLB Slide button

Claims (11)

1. An image generation unit that separates and generates a surface reflected light image and an internally reflected light image from the captured image based on an imaging signal corresponding to the captured image of the imaged object irradiated with linear polarization.
   A detection unit that detects a contrast evaluation value for each of the surface reflected light image and the internally reflected light image,
   A determination unit that determines whether or not the reflected light image to be focused has reached the focus position based on the contrast evaluation value.
   Image processing device equipped with.
2. An imaging unit having a focusing lens, imaging the object to be imaged, and outputting the imaging signal.
   A drive unit that drives the focusing lens to change the focus position based on the determination result of the determination unit is provided.
   The image processing apparatus according to claim 1.
3. The image generation unit generates the surface reflected light image based on the polarization component obtained by removing the polarization component orthogonal to the linear polarization from the linearly polarized light component, and the inside is based on the polarization component orthogonal to the linear polarization. Generate a reflected light image,
   The image processing apparatus according to claim 1.
4. An image region in which the detection unit detects the contrast evaluation value corresponding to the surface reflected light image in a region where the brightness level exceeds a predetermined first threshold value or a region where the polarization degree level exceeds a predetermined second threshold value. With a decision part to decide as,
   The image processing apparatus according to claim 1.
5. The object to be imaged is human skin,
   The detection unit includes a determination unit that determines the texture region in which the texture of the skin is detected as an image region in which the contrast evaluation value corresponding to the surface reflected light image is detected in the detection unit.
   The image processing apparatus according to claim 1.
6. When the detection unit contains a substance having a relatively high absorption rate in a specific wavelength band inside the image pickup object, the weight of the wavelength band is increased and the contrast evaluation value is obtained for the internally reflected light image. To detect,
   The image processing apparatus according to claim 1.
7. The surface reflected light image and the internally reflected light image are input, and a selection unit is provided which outputs either the surface reflected light image or the internally reflected light image to the detection unit based on the designated input.
   The image processing apparatus according to claim 1.
8. The surface reflected light image and the internally reflected light image are input, and the surface reflected light image and the internally reflected light image are combined by changing the ratio based on the designated input, and the combined reflected light image is a contrast evaluation value in the detection unit. Equipped with a selection unit that outputs
   The image processing apparatus according to claim 1.
9. An image output unit for displaying the surface reflected light image and the internally reflected light image is provided.
   The image processing apparatus according to claim 1.
10. A method performed by an image processor
    A process of separating and generating a surface reflected light image and an internally reflected light image from the captured image based on an imaging signal corresponding to the captured image of the imaged object irradiated with linear polarization.
    The process of detecting the contrast evaluation value for each of the surface reflected light image and the internally reflected light image, and
    The process of determining whether or not the reflected light image to be focused has reached the focus position based on the contrast evaluation value, and
    A method equipped with.
11. An imaging unit that has a focusing lens, images an imaged object irradiated with linear polarization, and outputs an imaging signal.
    An image generation unit that separates and generates a surface reflected light image and an internally reflected light image from the captured image based on the image pickup signal corresponding to the captured image of the object to be imaged.
    A detection unit that detects a contrast evaluation value for each of the surface reflected light image and the internally reflected light image,
    A determination unit that determines whether or not the focus position has been reached based on the contrast evaluation value,
    A drive unit that drives the focusing lens to change the focus position of the reflected light image to be focused based on the determination result of the determination unit.
    Electronic equipment equipped with.

Images