JP2023160153A - Imaging apparatus, method for controlling imaging apparatus, and program - Google Patents

Abstract

To provide an imaging apparatus that can improve the accuracy of a score map output by using a hierarchical neural network.SOLUTION: An imaging apparatus 101 comprises: a visible light image pick-up device 104 that receives visible light; and a non-visible light image pick-up device 105 that receives non-visible light. With a visible light image obtained by the visible light image pick-up device 104 as an input image, the imaging apparatus 101 creates a likelihood score map representing the attribute of an area of the input image by using a neural network 201. The imaging apparatus 101 corrects the likelihood score map based on a non-visible light image obtained by the non-visible light image pick-up device 105.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device, a method of controlling the imaging device, and a program.

2. Description of the Related Art There is a known technique for extracting feature amounts from image data and using a discriminator to determine a subject in the image data. One such technology is a Convolutional Neural Network (hereinafter referred to as "CNN"), which is a type of neural network. CNN has the property of sequentially performing local convolution processing in multiple stages. As related techniques, the techniques of Non-Patent Document 1 and Patent Document 1 have been proposed.

In the technique of Non-Patent Document 1, an image is subjected to calculation processing using a CNN, and the feature amounts of the final layer of the CNN are aggregated for each region of interest, and it is determined whether the image is an object or not. This process is performed for all regions of interest.

Furthermore, in the technology of Patent Document 1, connected hierarchical features are generated by connecting the outputs of multiple layers of a hierarchical neural network, and the connected hierarchical features are used to determine attributes such as the sky, buildings, grass, turf, skin, etc. A score map representing the This score map is used, for example, for white balance control and exposure control during photographing.

JP 2019-32773 Publication

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, Faster R-CNN:Towards Real-Time Object Detection with Region Propos al Networks, NIPS 2015

However, with the technique of Patent Document 1 mentioned above, there is a concern that an accurate score map cannot be calculated in white balance and exposure conditions where machine learning is not performed.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging device, a control method for the imaging device, and a program that can improve the accuracy of a score map output using a hierarchical neural network.

In order to achieve the above object, an imaging device of the present invention is an imaging device comprising a visible light imaging means for receiving visible light and a non-visible light imaging means for receiving non-visible light. generating means for generating a score map representing the attributes of a region of the input image using a hierarchical neural network, using the visible light image obtained by the means as an input image; The present invention is characterized by comprising a correction means for correcting the score map based on the optical image.

According to the present invention, it is possible to improve the accuracy of a score map output using a hierarchical neural network.

FIG. 1 is a block diagram schematically showing the configuration of an imaging device according to the present embodiment. FIG. 2 is a block diagram schematically showing the configuration of a machine learning processing section in FIG. 1. FIG. 2 is a flowchart showing the procedure of score map calculation processing performed by the machine learning processing section of FIG. 1. FIG. FIG. 2 is a diagram showing an example of a visible light image as an input image of the machine learning processing unit in FIG. 1. FIG. FIG. 3 is a diagram for explaining generation of a likelihood score map by the attribute determination unit of FIG. 2; 2 is a diagram illustrating an example of an image used by the imaging device of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the null likelihood score map performed by the score map calculation unit of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the grass and turf likelihood score map performed by the score map calculation unit of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the skin likelihood score map performed by the score map calculation unit of FIG. 1. FIG. 2 is a flowchart showing a procedure of control processing executed by the imaging device of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the null likelihood score map performed by the score map calculation unit of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the grass and turf likelihood score map performed by the score map calculation unit of FIG. 1. FIG. FIG. 2 is a diagram for explaining correction of the skin likelihood score map performed by the score map calculation unit of FIG. 1. FIG.

Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. The same components are given the same reference numerals throughout the drawings, and their explanations may be simplified or omitted. Note that the embodiment described below is merely an example, and the present invention is not limited to the configuration described in the embodiment.

FIG. 1 is a block diagram schematically showing the configuration of an imaging apparatus 101 according to this embodiment. In this embodiment, the imaging device 101 is, for example, a digital still camera or a digital video camera that captures an image of a subject. Note that the imaging device 101 is not limited to a digital still camera or a digital video camera, and may be a mobile terminal such as a smartphone or a tablet terminal that has a shooting function.

The imaging device 101 includes one optical system including an imaging optical system 102 and a light separation unit 103, a visible light imaging device 104, and a non-visible light imaging device 105. The imaging device 101 further includes a control section 106, an image processing section 107, a machine learning processing section 108, a score map calculation section 109, a memory 110, and a display section 111.

The visible light image sensor 104 and the non-visible light image sensor 105 are image sensors that use the above optical system as a common optical system. The visible light image sensor 104 receives visible light that has passed through the optical system and generates an image signal. A visible light image is generated based on this image signal. The invisible light image sensor 105 receives infrared light, which is invisible light, that has passed through the optical system and generates an image signal. A non-visible light image is generated based on this image signal. The visible light image sensor 104 and the non-visible light image sensor 105 each include a CMOS sensor, a CCD sensor, or the like. The visible light imaging device 104 and the non-visible light imaging device 105 each convert the subject image formed on the imaging surface into an electrical signal, and output the electrical signal to the image processing unit 107 as an image signal.

The imaging optical system 102 consists of a single lens or a plurality of lens groups. Note that the imaging optical system 102 may include at least one of various control mechanisms such as zoom, focus, aperture, and camera shake correction. The light separation unit 103 is composed of a wavelength selection prism, in which light with a wavelength shorter than a specific wavelength (visible light) passes through the wavelength selection prism, and light with a wavelength longer than the specific wavelength (infrared light) passes through the wavelength selection prism. It is designed to be reflected by Note that transmitting/reflecting means that 80% or more of light is transmitted/reflected. The visible light component that has passed through the wavelength selection prism is photoelectrically converted by a visible light imaging device 104 placed behind the light separation unit 103 and converted into an image. On the other hand, the infrared light component reflected by the wavelength selection prism is photoelectrically converted by the non-visible light imaging device 105 disposed through the optical axis and converted into an image. Here, the specific wavelength is, for example, 600 nm or more and 750 nm or less. In this case, the boundary between visible light and infrared light is defined as 600 nm or more and 750 nm or less. Further, infrared light corresponds to light having a wavelength from a specific wavelength to 2500 nm, for example.

The pixels constituting the visible light image sensor 104 include on-chip color filters in an RGB Bayer array. The RGB format visible light image output from the visible light image sensor 104 includes color information in addition to brightness information. On the other hand, the invisible light image output from the invisible light image sensor 105 includes only brightness information. Note that the visible light image sensor 104 only needs to have a sensitivity distribution mainly using visible light, and may have a sensitivity distribution other than visible light. Furthermore, the non-visible light imaging device 105 only needs to have a sensitivity distribution mainly for infrared light, and may also have a sensitivity distribution for light other than infrared light. Driving of the visible light image sensor 104 and the non-visible light image sensor 105 and reading of image signals are controlled by a control unit 106.

Note that although this embodiment describes a configuration in which light with different spectral characteristics is guided to the visible light image sensor 104 and the non-visible light image sensor 105 using the light separation unit 103, the present invention is not limited to this structure. . For example, the visible light image sensor 104 and the non-visible light image sensor 105 may have separate and independent optical systems, which is a so-called two-lens type. Even in such a configuration, mutually synchronized imaging is performed.

The control unit 106 includes, for example, a CPU, an MPU, and other dedicated arithmetic circuits, and controls the entire imaging device 101 . The image processing unit 107 performs image processing on image signals obtained from the visible light image sensor 104 and the non-visible light image sensor 105, respectively, and generates captured image data for each image sensor. This image processing includes, for example, pixel interpolation processing, color conversion processing, various correction processing such as pixel defect correction and lens correction, detection processing for adjusting black level, focus, exposure, etc., white balance processing, and gamma correction processing. , edge enhancement processing, noise suppression processing, etc. Further, this image processing includes demosaic processing. For example, when a visible light image read out in RGB format is subjected to demosaic processing, the visible light image is converted to an image in YUV format. Furthermore, when the invisible light image is subjected to demosaic processing, the invisible light image is converted into a YUV format image. Note that the YUV format image converted from the invisible light image does not have color information, and the values of U and V are zero.

The machine learning processing unit 108 uses the visible light image as an input image and outputs a score map for each attribute such as "sky", "grass", "skin", etc. using the neural network 201 of FIG. 2, which will be described later. The score map calculation unit 109 uses the invisible light image to perform correction processing on the score map for each attribute obtained by the machine learning processing unit 108.

The imaging device 101 uses the score map that has been corrected by the score map calculation unit 109 to perform white balance control in the image processing unit 107 and to perform exposure control on the visible light image sensor 104 via the control unit 106. control.

Specifically, white balance control uses a score map representing the attribute of "sky" to prevent the ground from becoming too reddish. Furthermore, since a snowy scene that is not a sky tends to have a bluish tint, a score map representing the attribute of "sky" is used to distinguish between shade and sky. In addition, appropriate white balance control is performed that distinguishes between a scene under a mercury lamp light source and a scene with grass and grass using a score map representing the attribute of "grass and grass."

In exposure control, the face area is used to control the face so that it is properly exposed. However, if the face area includes hair, a mask, etc., there is a problem in that appropriate exposure cannot be achieved. Therefore, a score map representing the attribute of "skin" is used to identify only the skin area in the face area, and control is performed so that the skin area is properly exposed.

The memory 110 is composed of nonvolatile memory, RAM, and the like. The nonvolatile memory stores processing procedures (control programs) of the control unit 106 and various parameters. The RAM is used as a work area for the control unit 106, and is also used as a storage area for image processing.

Note that in this embodiment, the control unit 106 may include an image processing unit 107 and a compression/expansion unit (not shown). The processing functions of these blocks can be realized, for example, by the CPU executing a program stored in the memory 110. Alternatively, it may be realized by a dedicated arithmetic circuit that constitutes the control unit 106.

The control unit 106 may further generate a compressed image using a compression/expansion unit (not shown). The compression/expansion section executes still image compression and moving image compression. The image compression method is, for example, H. 264, H. This compression method is based on standards such as H.265, MPEG, and JPEG. Note that the compression/decompression unit may generate an image in any data format such as mp4 or avi format. The compressed image generated by the compression/expansion unit is recorded in the memory 110 or a recording medium (not shown) attached to the imaging device 101. Further, the compressed image is subjected to decompression processing by a compression/expansion section, and the image obtained by the decompression processing is displayed on the display section 111. In addition to displaying images, the display unit 111 displays a user interface (UI) to the user (operator).

FIG. 2 is a block diagram schematically showing the configuration of the machine learning processing unit 108 in FIG. 1. In FIG. 2, the machine learning processing unit 108 includes a neural network 201, a connected feature generation unit 202, and an attribute determination unit 203.

The neural network 201 is a hierarchical neural network having first to nth layers (n is a natural number of 2 or more), and processes a visible light image that is an input image. The connected feature generation unit 202 performs feature generation to generate connected hierarchical features by connecting the outputs (feature maps) of prescribed layers in the neural network 201. The attribute determination unit 203 includes three likelihood determination units 203a to 203c, and generates a score map representing the attributes of the region of the input image. The likelihood determining units 203a to 203c each use the connected hierarchical features generated by the connected feature generating unit 202 to generate a score map of the corresponding attribute. The likelihood determination unit 203a generates an empty likelihood score map that maps the likelihood score of the attribute of “empty” (hereinafter referred to as “empty likelihood score”). The likelihood determination unit 203b generates a grass-grass likelihood score map that is a map of the likelihood score of the attribute "grass-grass" (hereinafter referred to as "grass-grass likelihood score"). The likelihood determination unit 203c generates a skin likelihood score map that maps the likelihood score of the attribute "skin" (hereinafter referred to as "skin likelihood score").

FIG. 3 is a flowchart showing the procedure of score map calculation processing performed by the machine learning processing unit 108 of FIG. 1.

In FIG. 3, first, in step S301, the machine learning processing unit 108 inputs a visible light image to the neural network 201. In this embodiment, as an example, a case will be described in which a visible light image shown in FIG. 4 is input. This visible light image includes a tree area and a person area. Furthermore, the visible light image 401 includes a mountain area with grass and turf, and a sky area as a background area. The neural network 201 processes the visible light image input in step S301.

Next, in step S302, the connected feature generation unit 202 extracts the processing result of the neural network 201 as a feature. Specifically, the connected feature generation unit 202 extracts the output (feature map) of a prescribed layer in the neural network 201. Next, in step S303, the connected feature generation unit 202 generates a connected hierarchical feature by connecting the outputs of the prescribed layers extracted in step S302. Next, in step S304, the attribute determination unit 203 generates a score map for each attribute using the connected hierarchical features. Specifically, the likelihood determination unit 203a in the attribute determination unit 203 generates a null likelihood score map in which the null likelihood scores are mapped, as shown in FIG. The likelihood determination unit 203b generates a grass and grass likelihood score map that maps the grass and grass likelihood scores, as shown in FIG. As shown in FIG. 5, the likelihood determination unit 203c generates a skin likelihood score map in which the skin likelihood scores are mapped. In Figure 5, the area block with the highest likelihood score for each attribute is shown in white, the area block with the smallest likelihood score is shown in black, and each area block is represented in gray gradations according to the likelihood score. . After that, this process ends.

Here, there is a concern that the machine learning processing unit 108 may not be able to calculate an accurate score map in white balance and exposure conditions where machine learning is not performed.

In contrast, in the present embodiment, the likelihood score map is corrected based on the invisible light image obtained by the invisible light image sensor 105.

FIG. 6 is a diagram illustrating an example of an image used by the imaging device 101 of FIG. 1. FIG. 6(a) shows an example of a visible light image. FIG. 6(b) shows an example of a non-visible light image. In this embodiment, the non-visible light image is an image of near-infrared light having a wavelength of 750 nm to 2500 nm. Under near-infrared light, the sky emits less infrared rays, and grass and grass reflect more infrared rays. Therefore, in a non-visible light image, the sky area is dark (black) and the grass area is bright (white). Furthermore, in clothing worn by humans, materials such as cotton, nylon, and wool reflect a large amount of infrared rays, so in non-visible light images, regions of clothing made of these materials are slightly brighter. On the other hand, human skin does not reflect as much infrared rays as grass, grass, cotton, nylon, wool, etc., but it does reflect some infrared rays. Therefore, in a non-visible light image, the skin area has a characteristic that the color is intermediate between black and white (gray color).

Next, the correction of the likelihood score map using the non-visible light image performed by the score map calculation unit 109 will be described.

FIG. 7 is a diagram for explaining the correction of the null likelihood score map performed by the score map calculation unit 109 of FIG. 1. In this embodiment, the invisible light image and the empty likelihood score map generated by the likelihood determination unit 203a are input to the score map calculation unit 109. Note that the sky likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). The sky likelihood score map generated by the likelihood determination unit 203a is not an accurate likelihood score map, but as shown in FIG. ) is getting larger. For example, the empty likelihood score is extremely large in the clothing area (pixel value (192)), and the empty likelihood score is slightly large in the grass area (pixel value (64)). .

First, upon acquiring the sky likelihood score map and the non-visible light image, the score map calculation unit 109 performs an inversion process on the sky likelihood score map. Specifically, the score map calculation unit 109 performs inversion processing such as converting a pixel value (0) to a pixel value (255) and converting a pixel value (255) to a pixel value (0).

Next, the score map calculation unit 109 compares the pixel values of corresponding pixels in the inverted likelihood score map and the non-visible light image, and performs large value selection processing to output the larger pixel value. For example, regarding a region of clothing in which the void likelihood score (pixel value) is extremely large in the void likelihood score map generated by the likelihood determination unit 203a, in the void likelihood score map that has been inverted, the void likelihood score map generated by the likelihood determination unit 203a is The pixel value in this area becomes very small. Therefore, in the area of clothes, the pixel value of the non-visible light image is larger than that of the empty likelihood score map that has undergone inversion processing, and the pixel value of the non-visible light image is output. Similarly, in the grass area, the pixel values of the non-visible light image are larger than those of the inverted sky likelihood score map, so the pixel values of the non-visible light image are output.

Next, the score map calculation unit 109 inverts and outputs the pixel values output by the large value selection process. The score map calculation unit 109 performs such processing on all pixels forming the empty likelihood score map. As a result, in the empty likelihood score map generated by the likelihood determination unit 203a, the empty likelihood scores of grass areas and clothing areas, which had large empty likelihood scores other than sky areas, are corrected to be smaller. The accuracy of the empty likelihood score map can be improved.

FIG. 8 is a diagram for explaining the correction of the grass and turf likelihood score map performed by the score map calculation unit 109 of FIG. 1. In the present embodiment, the invisible light image and the grass likelihood score map generated by the likelihood determination unit 203b are input to the score map calculation unit 109. It is assumed that the grass likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). The grass and grass likelihood score map generated by the likelihood determination unit 203b is not an accurate likelihood score map, but as shown in FIG. is getting bigger. For example, the grass-grass likelihood score is extremely large in the human face region (pixel value (255)), and the grass-grass likelihood score is slightly large in the sky region (pixel value (255)). 32)).

First, upon acquiring the grass and turf likelihood score map and the non-visible light image, the score map calculation unit 109 performs the above-described inversion process on the grass and turf likelihood score map and the non-visible light image, respectively. Next, the score map calculation unit 109 compares the pixel values of corresponding pixels in the inverted grass likelihood score map and the inverted invisible light image, and outputs the larger pixel value. Perform selection processing.

For example, for a region of a person's face that has a very large Kusashiba likelihood score (pixel value) in the Kusashiba likelihood score map generated by the likelihood determination unit 203b, in the Kusashiba likelihood score map that has undergone inversion processing, The above-mentioned inversion process makes the pixel value of this area extremely small (pixel value (0)). Therefore, in the area of a person's face, the pixel value of the inverted invisible light image is larger than that of the inverted grass likelihood score map, and the pixel value of the inverted invisible light image is is output. Similarly, in the sky region, the pixel values of the inverted invisible light image are larger than those of the inverted grass and turf likelihood score map, so the pixel values of the invisible light image are output. Next, the score map calculation unit 109 inverts and outputs the pixel values output by the large value selection process. The score map calculation unit 109 performs such processing on all pixels forming the grass and turf likelihood score map. As a result, in the grass and grass likelihood score map generated by the likelihood determination unit 203b, the grass and grass likelihood score of the person's face area and the sky area where the grass and grass likelihood score is large in areas other than the grass and grass area is determined. The score can be corrected to a smaller value, and the accuracy of the grass and grass likelihood score map can be improved.

FIG. 9 is a diagram for explaining the correction of the skin likelihood score map performed by the score map calculation unit 109 of FIG. 1. In the present embodiment, the invisible light image and the skin likelihood score map generated by likelihood determination section 203c are input to score map calculation section 109. It is assumed that the skin likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). The skin likelihood score map generated by the likelihood determination unit 203c is not an accurate likelihood score map, but as shown in FIG. 9, the skin likelihood score is large in some areas other than the skin area. ing. For example, the skin likelihood score is very large in the clothing area (pixel value (128)), and the skin likelihood score is slightly large in the sky area and grass area (pixel value (128)). (32)).

First, upon acquiring the skin likelihood score map and the non-visible light image, the score map calculation unit 109 performs the above-described inversion process on the skin likelihood score map. Further, the score map calculation unit 109 performs a subtraction process of subtracting a pixel value from a predetermined value, for example, 128, for each pixel of the invisible light image. This predetermined value is, for example, a value determined based on the characteristic that the skin area in a non-visible light image has a color somewhere between black and white (gray color), and is a value corresponding to approximately the middle value in pixel values. be. Note that if the result calculated in the subtraction process is a negative value, a clip process is performed to replace the calculated result with 0. That is, in this embodiment, 0 is obtained as a result of calculation by these processes for an area where the pixel value is equal to or greater than the predetermined value in the non-visible light image, for example, an area of grass or an area of clothes. Next, the score map calculation unit 109 performs the above-described inversion process on the results calculated by these processes to generate a subtracted invisible light image. Next, the score map calculation unit 109 compares the pixel values of corresponding pixels in the inversion-processed skin likelihood score map and the subtraction-processed invisible light image, and selects a large value to output the larger pixel value. Perform processing.

For example, regarding a region of clothing in which the skin likelihood score (pixel value) is very large in the skin likelihood score map generated by the likelihood determination unit 203c, in the skin likelihood score map that has been inverted, the skin likelihood score map generated by the likelihood determination unit 203c is The pixel value of this area is 127. On the other hand, in the subtracted invisible light image, inversion processing is performed on the 0 obtained by the above-described subtraction processing and clipping processing, and the pixel value of the above region becomes 255. In this manner, in the clothing region, the subtraction-processed non-visible light image has a larger pixel value than the inversion-processed skin likelihood score map, so the pixel values of the non-visible light image are output. Similarly, in the grass area, the pixel values of the subtracted invisible light image are larger than the inverted skin likelihood score map, so the pixel values of the invisible light image are output. . Next, the score map calculation unit 109 inverts and outputs the pixel values output by the large value selection process. The score map calculation unit 109 performs such processing on all pixels forming the skin likelihood score map. As a result, in the skin likelihood score map generated by the likelihood determination unit 203c, the skin likelihood score of areas other than skin areas where the skin likelihood score was large, such as areas of grass and clothes, is corrected to be smaller. The accuracy of the skin likelihood score map can be improved.

FIG. 10 is a flowchart showing the procedure of control processing executed by the imaging apparatus 101 of FIG. 1. The control process in FIG. 10 can be realized by the control unit 106 executing a program stored in the memory 110.

In FIG. 10, first, in step S1001, the control unit 106 drives the visible light image sensor 104. Thereby, the visible light image sensor 104 receives visible light and generates an image signal. A visible light image is generated based on this image signal. Next, in step S1002, the control unit 106 instructs the image processing unit 107 to execute image processing of the visible light image. The image processing unit 107 that receives this instruction performs various image processing on the visible light image.

Next, in step S1003, the control unit 106 determines whether the visible light image is an image of a sampling frame for white balance. For example, when the EVF operates at a frame rate of 120 [fps], if the white balance is changed every frame, the screen will flicker, resulting in poor visibility. Therefore, in this embodiment, sampling for white balance is performed at a low frame rate such as 30 [fps].

If it is determined in step S1003 that the visible light image is an image of a sampling frame for white balance, the process advances to step S1004. In step S1004, the control unit 106 drives the invisible light image sensor 105. Thereby, the invisible light image sensor 105 receives infrared light, which is invisible light, and generates an image signal. A non-visible light image is generated based on this image signal.

Next, in step S1005, the control unit 106 instructs the image processing unit 107 to execute image processing of the invisible light image. Upon receiving this instruction, the image processing unit 107 performs various image processing on the non-visible light image. Next, in step S1006, the control unit 106 instructs the machine learning processing unit 108 to generate the above-described likelihood score map. The machine learning processing unit 108 that receives this instruction generates the above-mentioned likelihood score map using the visible light image as an input image. Next, in step S1007, the control unit 106 instructs the score map calculation unit 109 to correct the likelihood score map generated by the machine learning processing unit 108. Upon receiving this instruction, the score map calculation unit 109 uses the non-visible light image to correct the likelihood score map generated by the machine learning processing unit 108, as described above. Next, in step S1008, the control unit 106 determines whether to end imaging. In step S1008, for example, when receiving a predetermined operation from the user that instructs to end imaging, the control unit 106 determines to end imaging. On the other hand, if the above-described predetermined operation is not accepted, the control unit 106 determines that the photographing is not to be completed.

If it is determined in step S1008 that photographing is not to be completed, the process returns to step S1001. If it is determined in step S1008 that the photographing is to be completed, this processing ends.

If it is determined in step S1003 that the visible light image is not an image of a sampling frame for white balance, control to drive the invisible light image sensor 105 is not performed, and the process advances to step S1009. In this manner, in this embodiment, the power consumption of the imaging device 101 is suppressed by controlling the invisible light imaging device 105 to be driven only for frames for which the precision of the likelihood score map is desired to be improved. be able to.

In step S1009, the control unit 106 instructs the machine learning processing unit 108 to generate a score map (not shown) for understanding the face position. Upon receiving this instruction, the machine learning processing unit 108 generates a score map for understanding the face position based on the acquired visible light image. A score map for determining the face position is used to track the subject when photographing. Next, the process advances to step S1008, which will be described later.

According to the embodiment described above, the likelihood score map is corrected based on the invisible light image obtained by the invisible light image sensor 105. Thereby, the accuracy of the likelihood score map output using the neural network 201 can be improved.

Note that in the embodiment described above, a configuration was described in which the invisible light image sensor 105 is driven when the visible light image serving as the input image is an image of a sampling frame for white balance. Not limited to. For example, when the visible light image serving as the input image is an image of a frame used for a predetermined control using a likelihood score map such as exposure control or scene recognition control (landscape, night scene, etc.), the invisible light image sensor 105 It may also be controlled to drive. By controlling in this way, it is possible to minimize the power consumption of the imaging device 101 and improve the accuracy of the likelihood score map used for exposure control and scene recognition control.

Furthermore, in the embodiment described above, the score map calculation unit 109 may correct the likelihood score map using a method different from the method shown in FIGS. 7 to 9 described above.

FIG. 11 is a diagram for explaining the correction of the null likelihood score map performed by the score map calculation unit 109 of FIG. 1. Note that, as described above, the sky likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). In addition, in FIG. 11, similarly to FIG. 7, the empty likelihood score map generated by the likelihood determination unit 203a is not an accurate likelihood score map, but is a partial likelihood score map that is a partial likelihood score map other than the sky area. degree score is increasing. For example, the empty likelihood score is extremely large in the clothing area (pixel value (192)), and the empty likelihood score is slightly large in the grass area (pixel value (64)). .

First, the score map calculation unit 109 compares the invisible light image with a predetermined first threshold, for example, 64. The first threshold is a value larger than a value expected as a pixel value of a sky region in a non-visible light image, and smaller than a value expected as a pixel value of a region other than the sky region in a non-visible light image. It is a value. In a non-visible light image, the sky region is dark (black), that is, the pixel value is close to the minimum value (0), so the reliability of the region whose pixel value is smaller than the first threshold value as a sky region is high. can be said to be high. Therefore, in the invisible light image, a region whose pixel value is smaller than the first threshold is determined to be a region with high reliability as a sky region, and the pixel value of the pixel corresponding to this region is determined in the sky likelihood score map. is used as is. On the other hand, an area in which the pixel value is equal to or greater than the first threshold in the invisible light image is determined to be an area with low reliability as a sky area, and the pixels corresponding to this area in the sky likelihood score map The value is converted to 0. By performing such correction, in the empty likelihood score map generated by the likelihood determination unit 203a, the empty likelihood scores of the grass areas and clothing areas where the empty likelihood scores were large are reduced to 0. can be corrected, and the accuracy of the empty likelihood score map can be improved.

FIG. 12 is a diagram for explaining the correction of the grass and turf likelihood score map performed by the score map calculation unit 109 of FIG. 1. Note that, as described above, it is assumed that the grass likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). In FIG. 12 as well, similarly to FIG. 8, the grass and grass likelihood score map generated by the likelihood determination unit 203b is not an accurate likelihood score map, but is based on the grass and grass in some areas other than the grass and grass area. The likelihood score is large. For example, the grass-grass likelihood score is extremely large in the human face region (pixel value (255)), and the grass-grass likelihood score is slightly large in the sky region (pixel value (255)). 32)).

First, the score map calculation unit 109 compares the invisible light image with a predetermined second threshold, for example, 96. The second threshold value is a value smaller than a value assumed as a pixel value of a grass area in a non-visible light image, and a value assumed as a pixel value of an area other than the grass area in a non-visible light image. It is a larger value. In the non-visible light image, the grass area is bright (white), that is, the pixel value is close to the maximum value (255), so the area whose pixel value is larger than the second threshold is considered as a grass area. It can be said that the reliability is high. Therefore, in the non-visible light image, an area whose pixel value is larger than the second threshold is determined to be an area with high reliability as a grass area, and the pixels corresponding to this area in the grass/grass likelihood score map The pixel value of is used as is. On the other hand, an area where the pixel value is less than or equal to the second threshold in the invisible light image is determined to be an area with low reliability as a grass area, and the pixels corresponding to this area in the grass/grass likelihood score map The pixel value of is converted to 0. By performing such correction, in the grass and grass likelihood score map generated by the likelihood determination unit 203b, the grass and grass likelihood of the person's face area and the sky area where the grass and grass likelihood score was large is reduced. The score can be corrected to 0, and the accuracy of the grass-grass likelihood score map can be improved.

FIG. 13 is a diagram for explaining the correction of the skin likelihood score map performed by the score map calculation unit 109 of FIG. 1. Note that, as described above, the skin likelihood score map and the invisible light image have the same number of pixels, and each pixel has a gradation (level) of 8 bits (0 to 255). In FIG. 13, similarly to FIG. 9, the skin likelihood score map generated by the likelihood determination unit 203c is not an accurate likelihood score map, but is based on the skin likelihood score map in some areas other than the skin area. is getting bigger. For example, the skin likelihood score is very large in the clothing area (pixel value (128)), and the skin likelihood score is slightly large in the sky area and grass area (pixel value (128)). (32)).

First, the score map calculation unit 109 compares the invisible light image with a predetermined third threshold, for example, 128. The third threshold is a value that is larger than the expected pixel value of the skin area in the invisible light image, and is a value that is larger than the expected pixel value of the skin area in the invisible light image. This value is smaller than the expected pixel value for the grass area. In invisible light images, the skin area is somewhat dark (gray), that is, the pixel value is around the middle, so areas with pixel values smaller than the third value are considered to be highly reliable as skin areas. I can say it. Therefore, in the non-visible light image, a region whose pixel value is smaller than the third threshold is determined to be highly reliable as a skin region, and the pixel value of the pixel corresponding to this region is used as is in the skin likelihood score map. be done. On the other hand, an area in which the pixel value is equal to or higher than the third threshold in the invisible light image is determined to be an area with low reliability as a skin area, and the pixels corresponding to this area in the skin likelihood score map The value is converted to 0. By performing such correction, in the skin likelihood score map generated by the likelihood determination unit 203c, the skin likelihood scores of areas such as grass areas and clothing areas where the skin likelihood scores were large are reduced to 0. can be corrected to improve the accuracy of the skin likelihood score map.

As described above, in the embodiment described above, based on the result of comparing the pixel values of pixels constituting the invisible light image with predetermined thresholds (first threshold, second threshold, third threshold), Then, it is determined whether or not to correct the pixel value of the pixel corresponding to the pixel in the likelihood score map. Thereby, the accuracy of the likelihood score map can be improved.

Furthermore, in the embodiments described above, the threshold value differs for each attribute represented by the likelihood score map, so it is possible to appropriately correct the likelihood score map depending on the attribute.

(Other examples)
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a recording medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Further, each function may be divided into those executed by a processor reading a program and those executed by a circuit, and these may be combined.

Further, although preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

The disclosure of this embodiment includes the following configuration and method.

(Structure 1) An imaging device comprising a visible light imaging means for receiving visible light and an invisible light imaging means for receiving non-visible light, wherein the visible light image obtained by the visible light imaging means is used as an input image. generating means for generating a score map representing attributes of a region of the input image using a hierarchical neural network; and correcting the score map based on the invisible light image obtained by the invisible light imaging means. An imaging device comprising: a correction means.
(Structure 2) The correction means generates an inverted score map in which the pixel values of all pixels constituting the score map are inverted, and combines the inverted score map and the invisible light image. A large value selection process is performed to compare the pixel values of corresponding pixels and output the larger pixel value, and the score map is created using a pixel value obtained by inverting the pixel value output by the large value selection process. The imaging device according to configuration 1, wherein the imaging device performs correction.
(Structure 3) The imaging device according to Structure 2, wherein the score map is a sky likelihood score map that maps likelihood scores representing sky attributes of the region of the input image.
(Structure 4) The correction means generates an inverted score map in which the pixel values of all pixels constituting the score map are inverted, and inverts the pixel values of all pixels constituting the invisible light image. Generate an inverted invisible light image that has been inverted, compare the pixel values of corresponding pixels in the inverted score map and the inverted invisible light image, and select the larger pixel value. The imaging device according to configuration 1, wherein a large value selection process to be output is performed, and the score map is corrected using a pixel value obtained by inverting the pixel value outputted by the large value selection process.
(Structure 5) The imaging device according to Structure 4, wherein the score map is a grass likelihood score map that is a map of likelihood scores representing attributes of grass in the area of the input image.
(Structure 6) The correction means generates an inverted score map in which the pixel values of all pixels constituting the score map are inverted, and the correction means generates an inverted score map in which the pixel values of all pixels constituting the invisible light image are inverted. The values obtained by performing predetermined arithmetic processing are inverted to generate an arithmetic-processed invisible light image, and the corresponding pixels are Performing a large value selection process that compares pixel values and outputs the larger pixel value, and correcting the score map using a pixel value obtained by inverting the pixel value output by the large value selection process. The imaging device according to feature 1.
(Structure 7) The predetermined calculation process subtracts the pixel value of the pixel from a predetermined value in the pixels constituting the invisible light image, and the value obtained by subtraction is a positive value. In some cases, the inverted value is used, and if the value obtained by subtraction is a negative value, the inverted value of 0 is used to generate a computationally processed invisible light image. The imaging device according to configuration 6, characterized in that the processing is performed.
(Structure 8) The imaging device according to Structure 6 or 7, wherein the score map is a skin likelihood score map that is a map of likelihood scores representing skin attributes of the region of the input image.
(Structure 9) The correction means calculates the pixel value of the pixel corresponding to the pixel in the score map based on the result of comparing the pixel value of the pixel constituting the invisible light image with a predetermined threshold value. The imaging device according to configuration 1, wherein the imaging device determines whether or not to perform correction.
(Configuration 10) The imaging device according to Configuration 9, wherein the threshold value differs for each attribute represented by the score map.
(Structure 11) Further comprising a control means for controlling driving of the invisible light imaging means, wherein the control means controls the invisible light imaging when the input image is an image of a frame used for predetermined control regarding photography. According to any one of configurations 1 to 10, the control is performed so that the invisible light imaging means is not driven when the input image is not an image of a frame used for the predetermined control. The imaging device described.
(Configuration 12) The imaging device according to Configuration 11, wherein the predetermined control is white balance control, exposure control, or scene recognition control.
(Structure 13) A method for controlling an imaging device comprising a visible light imaging means for receiving visible light and an invisible light imaging means for receiving non-visible light, the visible light image being obtained by the visible light imaging means. is an input image, a generation step of generating a score map representing attributes of a region of the input image using a hierarchical neural network, and a generation step of generating the score map based on the invisible light image obtained by the invisible light imaging means. A method for controlling an imaging device, comprising: a correction step for correcting.

101 Imaging device 104 Visible light image sensor 105 Invisible light image sensor 106 Control unit 108 Machine learning processing unit 109 Score map calculation unit 201 Neural network

Claims (14)

An imaging device comprising visible light imaging means for receiving visible light and invisible light imaging means for receiving non-visible light,
A generation unit that uses a visible light image obtained by the visible light imaging unit as an input image and generates a score map representing attributes of a region of the input image using a hierarchical neural network;
An imaging device comprising: a correction means for correcting the score map based on the invisible light image obtained by the invisible light imaging means. The correction means generates an inverted score map in which the pixel values of all pixels constituting the score map are inverted, and calculates the value of corresponding pixels between the inverted score map and the invisible light image. Performing a large value selection process that compares pixel values and outputs the larger pixel value, and correcting the score map using a pixel value obtained by inverting the pixel value output by the large value selection process. The imaging device according to claim 1, characterized in that: The imaging device according to claim 2, wherein the score map is a sky likelihood score map that is a map of likelihood scores representing sky attributes of the region of the input image. The correction means generates an inverted score map in which pixel values of all pixels constituting the score map are inverted, and inverts the pixel values of all pixels constituting the invisible light image. A large value that generates a processed invisible light image, compares the pixel values of corresponding pixels in the inverted score map and the inverted invisible light image, and outputs the larger pixel value. The imaging device according to claim 1, wherein the score map is corrected by performing selection processing and using pixel values obtained by inverting the pixel values output by the large value selection processing. 5. The imaging device according to claim 4, wherein the score map is a grass likelihood score map that is a map of likelihood scores representing attributes of grass in the area of the input image. The correction means generates an inverted score map in which the pixel values of all pixels constituting the score map are inverted, and performs predetermined arithmetic processing on the pixel values of all pixels constituting the invisible light image. The values obtained are inverted to generate an arithmetic-processed invisible light image, and the pixel values of corresponding pixels are compared between the inverted score map and the arithmetic-processed invisible light image. and performing a large value selection process to output a larger pixel value, and correcting the score map using a pixel value obtained by inverting the pixel value output by the large value selection process. The imaging device according to item 1. The predetermined arithmetic processing includes subtracting the pixel value of the pixel from a predetermined value in the pixels constituting the invisible light image, and if the value obtained by subtraction is a positive value, This is a process that uses the inverted value of the value, and if the value obtained by subtraction is a negative value, uses the inverted value of 0 to generate an arithmetic-processed invisible light image. The imaging device according to claim 6, characterized in that: The imaging device according to claim 6 or 7, wherein the score map is a skin likelihood score map that is a map of likelihood scores representing skin attributes of the region of the input image. The correction means corrects the pixel value of the pixel corresponding to the pixel in the score map based on the result of comparing the pixel value of the pixel constituting the invisible light image with a predetermined threshold value. The imaging device according to claim 1, wherein the imaging device determines whether or not an image is captured. The imaging device according to claim 9, wherein the threshold value differs for each attribute represented by the score map. further comprising a control means for controlling driving of the invisible light imaging means,
The control means drives the invisible light imaging means when the input image is a frame image used for predetermined control regarding photography, and when the input image is not a frame image used for the predetermined control. 2. The imaging apparatus according to claim 1, wherein control is performed so that the non-visible light imaging means is not driven. The imaging device according to claim 11, wherein the predetermined control is white balance control, exposure control, or scene recognition control. A method for controlling an imaging device comprising a visible light imaging means for receiving visible light and an invisible light imaging means for receiving non-visible light, the method comprising:
a generation step of generating a score map representing attributes of a region of the input image using a hierarchical neural network, using the visible light image obtained by the visible light imaging means as an input image;
A method for controlling an imaging device, comprising: a correction step of correcting the score map based on the invisible light image obtained by the invisible light imaging means. A program that causes a computer to execute a method for controlling an imaging device including a visible light imaging device that receives visible light and an invisible light imaging device that receives non-visible light, the program comprising:
The method for controlling the imaging device includes:
a generation step of generating a score map representing attributes of a region of the input image using a hierarchical neural network, using the visible light image obtained by the visible light imaging means as an input image;
A program comprising: a correction step of correcting the score map based on the invisible light image obtained by the invisible light imaging means.

Images