JP2023104667A - Image processing method, image processing apparatus, image processing system, and program - Google Patents

Abstract

To provide an image processing method capable of maintaining a blur correction effect while suppressing adverse effects caused by an aperture value of an optical system and a luminance value of a captured image.SOLUTION: The image processing method includes the steps of: generating information regarding the correction of a captured image based on the captured image; and generating an intensity-adjusted image based on the captured image, the information regarding correction, and a weight map, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image.SELECTED DRAWING: Figure 16

Description

The present invention relates to an image processing method, an image processing apparatus, an image processing system, and a program.

In image recognition or regression tasks, methods using machine learning models can achieve higher accuracy than theory-based methods using assumptions and approximations. Theory-based methods suffer from inaccuracies due to factors neglected by assumptions and approximations. However, in the method using a machine learning model, by training the machine learning model using training data that includes these elements, it is possible to realize estimation that is based on the training data without assumptions or approximations, so task accuracy is improved. improves.

Patent Literature 1 discloses a method of sharpening blur in a captured image using a convolutional neural network (CNN), which is one of machine learning models. Further, Japanese Patent Application Laid-Open No. 2002-200000 discloses a method of weighted averaging a captured image and an estimated image (blurred sharpened image) based on a luminance saturation region to adjust the sharpening strength.

Japanese Patent Application Laid-Open No. 2020-166628

The method disclosed in Japanese Patent Laid-Open No. 2002-200012 does not mention the adverse effects caused by the light diffraction phenomenon. When the aperture value (F-number) of the optical system is increased and a high-brightness subject is imaged, an airy disk or a streak of light occurs due to the diffraction phenomenon of light. When the machine learning model is used to correct the Airy disk and the streak of light, the lack of precision in the training image and the lack of parameters in the machine learning model can cause adverse effects such as unnatural enhancement. Moreover, the degree of conspicuity of this adverse effect varies depending on the peripheral luminance value to be corrected in the captured image. For example, unnatural enhancement is conspicuous in a dark image such as a night scene, but is not conspicuous in a bright image taken outdoors in the daytime.

Moreover, since the generation of the Airy disk and the streak of light depends on the aperture value of the optical system, the effect of correcting an image captured with an aperture value that does not cause adverse effects is also reduced. In addition, averaging is performed according to the determined weights without considering the conspicuousness of adverse effects that differ depending on the luminance value of the captured image. In other words, in a dark image, adverse effects are reduced around the saturated region, but in a bright image, the correction effect around the saturated region is reduced more than necessary.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing method capable of maintaining a blur correction effect while suppressing adverse effects caused by the aperture value of an optical system and the luminance value of a captured image.

An image processing method as one aspect of the present invention includes the steps of generating information regarding correction of the captured image based on the captured image, and generating an intensity adjusted image based on the captured image, the information regarding the correction, and a weight map. wherein the weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image.

Other objects and features of the invention are illustrated in the following examples.

According to the present invention, it is possible to provide an image processing method capable of maintaining the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image.

FIG. 4 is an explanatory diagram of a machine learning model in Example 1; 1 is a block diagram of an image processing system in Example 1. FIG. 1 is an external view of an image processing system in Example 1. FIG. FIG. 5 is an explanatory diagram of adverse effects caused by sharpening in Examples 1 to 3; FIG. 2 is an explanatory diagram of an Airy disk in Examples 1 to 3; FIG. 10 is an explanatory diagram of a beam of light in Examples 1 to 3; 4 is a flow chart of learning of a machine learning model in Examples 1 to 3. FIG. 4 is a flow chart of model output generation in Examples 1 and 2. FIG. 5 is a flowchart of sharpening intensity adjustment in the first embodiment. 4A and 4B are explanatory diagrams of a captured image and a saturation effect map in Example 1. FIG. 4A and 4B are explanatory diagrams of a captured image and a saturation effect map in Example 1. FIG. FIG. 4 is an explanatory diagram of a weight map in Example 1; FIG. 4 is an explanatory diagram of a weight map in Example 1; FIG. 11 is a block diagram of an image processing system in Example 2; FIG. 11 is an external view of an image processing system in Example 2; 10 is a flowchart of sharpening intensity adjustment in Example 2. FIG. FIG. 10 is an explanatory diagram of a weight map in Example 2; FIG. 11 is a block diagram of an image processing system in Example 3; FIG. 11 is an external view of an image processing system in Example 3; 10 is a flowchart of model output and sharpening intensity adjustment in Example 3. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

The gist of the present invention will be described before the specific description of the embodiments. According to the present invention, an estimated image is generated by using a machine learning model to sharpen the blur caused by the optical system from the captured image captured using the optical system. Then, a weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and the captured image and the estimated image are weighted averaged. A weight map is used to determine the proportion of each image in weighted averaging the captured image and the estimated image, and has continuous signal values. For example, when the numerical value of the weight map determines the ratio of the captured image, if the numerical value is 0.5, the weighted average of the ratio of the captured image and the sharpened blurred image by 50% is obtained as an intensity adjusted image. Also, if the value of the weight map is 1, the intensity adjusted image is the captured image.

Blurring caused by the optical system includes blurring due to aberration, diffraction, defocus, action of an optical low-pass filter, deterioration of the pixel aperture of the imaging element, and the like. Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, and the like. Neural networks include Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs), and the like.

Blur sharpening refers to processing for restoring frequency components of an object that have been reduced or lost due to blur. When sharpening the blur, if the aperture value of the optical system is increased and a high-brightness subject is imaged, an airy disk or a streak of light will occur due to the diffraction phenomenon of light. When the machine learning model is used to correct the Airy disk and the streak of light, adverse effects such as unnatural enhancement occur due to insufficient accuracy of the learning image and insufficient parameters of the machine learning model. Details of the Airy disk and the beam of light, insufficient accuracy of the learning image, and insufficient parameters of the machine learning model will be described later.

Moreover, the conspicuity of this adverse effect varies depending on the brightness of the image. For example, unnatural enhancement is conspicuous in a dark image such as a night scene, but is not conspicuous in a bright image taken outdoors in the daytime. Therefore, according to the present invention, a weight map is generated based on information regarding the aperture value of the optical system used to capture the captured image and information regarding the brightness value of the captured image, and weighted averaging of the captured image and the estimated image is performed. This makes it possible to maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. In the following, the stage of learning the weights of the machine learning model is referred to as the learning phase, and the stage of sharpening the blur by the machine learning model using the learned weights is referred to as the estimation phase.

First, an image processing system according to Embodiment 1 of the present invention will be described. In this embodiment, the task of the machine learning model is to sharpen the blur of a captured image including brightness saturation (increase the resolution of the captured image). Further, the sharpening blur refers to blur caused by aberration and diffraction generated in an optical system and an optical low-pass filter. However, the effect of the invention can be similarly obtained in the case of sharpening pixel aperture, defocus, and blur due to blurring. Moreover, it is possible to apply the present embodiment to tasks other than blur sharpening, and obtain effects. Specifically, tasks such as upsampling to increase the number of pixels in a captured image, and conversion (shape conversion) of defocus blur in the captured image. Defocus blur conversion includes, for example, conversion from bilinear blur to Gaussian blur or ball blur. Bilinear blur has a point spread function (PSF) with separated peaks. As a result, the subject, which is originally a single line, appears to be double-blurred when defocused. Ball blur has a PSF that is flat in intensity. Gaussian blur has a Gaussian-distributed PSF. Other defocus blur to be converted includes, for example, defocus blur caused by vignetting, ring-shaped defocus blur caused by pupil shielding such as a catadioptric lens, and the like.

FIG. 2 is a block diagram of the image processing system 100 in this embodiment. FIG. 3 is an external view of the image processing system 100. As shown in FIG. The image processing system 100 has a learning device 101 and an image processing device 103 connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 by wire or wirelessly. A captured image obtained by capturing an object space using the imaging device 102 is input to the image processing device 103 . The captured image is blurred due to aberration and diffraction by the optical system 102a in the image capturing apparatus 102 and the optical low-pass filter of the image sensor 102b, and the subject information is attenuated.

The image processing apparatus 103 uses a machine learning model to perform blur sharpening on the captured image, and generates a saturation effect map and a blur sharpened image (model output). The details of the saturation influence map will be described later. The machine learning model is learned by the learning device 101, and the image processing device 103 acquires information on the machine learning model in advance from the learning device 101 and stores it in the storage unit 103a. The image processing apparatus 103 also has a function of adjusting the intensity of blur sharpening by performing weighted addition of the captured image and the sharpened blur image. The details of the learning and estimation of the machine learning model and the adjustment of the blur sharpening intensity will be described later. The user can adjust the blur sharpening intensity while checking the image displayed on the display device 104 . The blurred and sharpened image that has undergone intensity adjustment is stored in the storage unit 103a or the recording medium 105, and is output to an output device 106 such as a printer as necessary. Note that the captured image may be grayscale or may have a plurality of color components. Further, it may be an undeveloped RAW image or an image after development.

Next, with reference to FIGS. 4A to 4C and FIG. 5, a description will be given of the adverse effect caused by the light diffraction phenomenon that occurs when blur sharpening is performed using a machine learning model. 4A and 4B are cross-sectional views of the PSF (the vertical axis is the signal value and the horizontal axis is the spatial coordinate) when the aperture value of the optical system is increased, and the scale of the vertical axis is different. FIG. 5 is a plan view of the PSF. The diffraction phenomenon of light produces a diffraction pattern with a bright central area surrounded by dark concentric ring zones. This is called an Airy disk. This diffraction pattern has a weak signal value, but when a point light source or strong light is captured, it can be visually confirmed even on the captured image. The dashed-dotted line 141 in FIG. 4C shows the ideal signal value obtained by sharpening the PSF and reducing the diffraction pattern, and the dotted line 142 shows the signal value when the diffraction pattern is emphasized by sharpening. As for the sharpening of the blur, it is preferable to use the one-dot chain line 141 . However, due to insufficient accuracy of the learning image or insufficient parameters of the machine learning model, the diffraction pattern may be emphasized as indicated by the dotted line 142 . This is because the edge of the diffraction pattern is erroneously recognized as an object and sharpened. Insufficient accuracy of training images and insufficient parameters of machine learning models will be described later.

Next, with reference to FIG. 6, the light beam caused by the diffraction phenomenon of light will be described. A streak of light means a long, narrow streak of light that occurs when a point light source or strong light is captured. FIG. 6 is an example of a beam of light. The number of streaks depends on the number of aperture blades 143 of the optical system 102a. If the number of diaphragm blades 143 is an odd number, streaks twice as many as the number are generated. On the other hand, when the number of aperture blades 143 is even, streaks are generated. Since the number of aperture blades 143 shown in FIG. 6 is five, ten streaks of light 144 are generated. The sharpness of the beam depends on the aperture value of the optical system. As the aperture value increases, sharpness increases. Also, even with the same aperture value, the appearance of the streak differs depending on the type of the optical system 102a. As with the Airy disk, unnatural enhancement of the light beam may occur due to insufficient accuracy of the training image or insufficient parameters of the machine learning model.

Next, the lack of accuracy of learning images will be described in detail. When blur sharpening is performed using a machine learning model, the sharpening accuracy depends on the accuracy of the learning image. In other words, a learning image that reproduces the blurring of the optical system 102a to be sharpened with high accuracy is required. However, if an attempt is made to exhaustively learn blurring at zoom positions, aperture values, and object distances that can be taken by the optical system 102a during imaging, the number of learning images becomes enormous. As a result, problems such as deterioration in individual blur sharpening accuracy and learning failure to converge occur. Therefore, it is necessary to discretely learn the zoom position, the aperture value, and the blur at the subject distance that the optical system 102a can take, and have the machine learning model predict the intermediate region. However, in that case, sharpening the blur in the intermediate region not included in the learning image may cause the edge of the diffraction pattern to be erroneously recognized as an object and sharpened.

Next, the lack of parameters in the machine learning model will be explained in detail. A machine learning model has multiple layers, and at each layer a linear sum of layer inputs and weights is taken. When a CNN that uses the convolution of the input and the filter as a linear sum (the value of each element of the filter corresponds to the weight, and may also include the sum with the bias) as a machine learning model, the number of layers of the filter is a parameter corresponds to the number of In other words, insufficient parameters means an insufficient number of filter layers. Since there is a trade-off relationship between the number of filter layers, learning time, and processing speed, parameter shortages may occur. It is possible to reduce adverse effects by using a machine learning model trained by incorporating a method of using a captured image and a brightness saturation map corresponding to the captured image as input data for the machine learning model, and a method of generating a saturation influence map. However, it is difficult to eliminate the harmful effects completely. A technique using a luminance saturation map and a technique for generating a saturation influence map will be described in detail.

Next, the brightness saturation map will be explained. A brightness saturation map is a map that represents a brightness saturation region in a captured image. In an area where brightness saturation occurs (brightness saturated area), information about the structure of the subject space is lost, and false edges may appear at the boundaries of each area, making it impossible to extract the correct feature amount of the subject. Therefore, by inputting the brightness saturation map, the neural network can identify the problem area as described above, so that the deterioration of the estimation accuracy can be suppressed.

Next, the saturation influence map will be described. Even with a luminance saturation map, machine learning models may not make correct decisions. For example, when a region of interest is in the vicinity of a region with saturated brightness, the machine learning model can determine that the region of interest is a region affected by saturation of brightness because there is a region with saturated brightness in the vicinity of the region of interest. However, if the region of interest is located at a position away from the luminance saturated region, it is not easy to determine whether or not this region is affected by luminance saturation, resulting in high ambiguity. As a result, the machine learning model may make an erroneous decision at a position away from the luminance saturated region. As a result, when the task is blur sharpening, the sharpening process corresponding to the saturated blurred image is performed on the non-saturated blurred image. At this time, artifacts are generated in the sharpened blur image, and the accuracy of the task is lowered. Therefore, it is preferable to generate a saturation effect map from a blurred captured image using a machine learning model.

A saturation influence map is a map (a signal sequence arranged spatially) representing the magnitude and range of signal values spread by blurring of an object in a luminance-saturated area of a captured image. By having the machine learning model generate the saturation influence map, the machine learning model can highly accurately estimate the presence or absence of the influence of luminance saturation in the captured image and the magnitude thereof. By generating the saturation influence map, the machine learning model can execute the processing to be executed on the regions affected by the luminance saturation and the processing to be executed on the other regions in each appropriate region. can. Therefore, by letting the machine learning model generate the saturation influence map, the accuracy of the task is improved compared to the case where the saturation influence map is not generated (only the recognition labels and the blur sharpened image are generated directly from the captured image). improves.

Although the above two methods are effective, it is difficult to completely eliminate the harmful effects. Therefore, in this embodiment, a weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and weighted averaging of the captured image and the estimated image is performed. This makes it possible to maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image.

Next, learning of a machine learning model executed by the learning device 101 will be described with reference to FIG. FIG. 7 is a flowchart of learning of the machine learning model. The learning device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and each step in FIG. 7 is mainly executed by each unit of the learning device 101.

First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is the image having signal values higher than the second signal value. The second signal value is a signal value corresponding to the luminance saturation value of the captured image. However, since the signal value may be normalized when inputting to the machine learning model, the second signal value and the luminance saturation value of the captured image do not necessarily match. Since the machine learning model is trained based on the original image, the original image should preferably be an image with various frequency components (edges with different orientations and strengths, gradations, flat areas, etc.). The original image may be a photographed image or CG (Computer Graphics).

Subsequently, in step S102, the calculation unit 101c blurs the original image to generate a blurred image. A blurred image is an image input to a machine learning model during learning, and corresponds to a captured image during estimation. The blur to be given is the blur to be sharpened. In this embodiment, the blur generated by the aberration and diffraction of the optical system 102a and the optical low-pass filter of the imaging element 102b is applied. The shape of the blur due to aberration and diffraction of the optical system 102a changes depending on the image plane coordinates (image height and azimuth). It also changes depending on the state of zooming, aperture, and focus of the optical system 102a. When learning a machine learning model that sharpens all of these blurs at once, it is preferable to generate a plurality of blurred images using a plurality of blurs generated by the optical system 102a. Also, in the blurred image, signal values exceeding the second signal value are clipped. This is done in order to reproduce the brightness saturation that occurs during the imaging process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

Subsequently, in step S103, the calculation unit 101c sets the first region based on the image based on the original image and the threshold value of the signal value. In this embodiment, a blurred image is used as an image based on the original image, but the original image itself may be used. A first region is set by comparing the signal value of the blurred image and the threshold value of the signal value. More specifically, a region in which the signal value of the blurred image is equal to or greater than the signal value threshold is defined as the first region. In this embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness-saturated region (saturated region) of the blurred image. However, the signal value threshold and the second signal value do not have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

Subsequently, in step S104, the calculation unit 101c generates a first area image having signal values of the original image in the first area. The first area image has signal values different from those of the original image in areas other than the first area. More desirably, the first area image has the first signal value in areas other than the first area. In this embodiment, the first signal value is 0, but the invention is not so limited. In this embodiment, the first area image has the signal value of the original image only in the area where the brightness of the blurred image is saturated, and the signal value is 0 in other areas.

Subsequently, in step S105, the calculation unit 101c blurs the first area image and generates a saturation influence correct map. The added blur is the same as the blur added to the blurred image. As a result, a saturated effect correct map, which is a map (a spatially arranged signal sequence) representing the magnitude and range of signal values spread by blurring (degradation) at the time of imaging, from an object in a luminance-saturated area of a blurred image. is generated. In the present embodiment, the saturation effect correct map is clipped by the second signal value in the same manner as the blurred image, but clipping is not necessarily required.

Subsequently, in step S106, the acquisition unit 101b acquires the correct model output. Since the task of this embodiment is blur sharpening, the correct model output is an image with less blur than the blurred image. In this embodiment, the correct model output is generated by clipping the original image with a second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is performed in the same way when generating a blurred image in step S102. Further, step S106 may be executed at any time after step S101 and before step S107.

Subsequently, in step S107, the calculation unit 101c uses a machine learning model to generate a saturation influence map and a model output based on the blurred image. In this example, the machine learning model shown in FIG. 1 is used, but the invention is not limited to this. A blurred image 201 and a luminance saturation map 202 are input to a machine learning model. A brightness saturation map 202 is a map that indicates a brightness-saturated region (where the signal value is equal to or greater than the second signal value) of the blurred image 201 . For example, it can be generated by binarizing the blurred image 201 with the second signal value. However, the brightness saturation map 202 is not necessarily essential. The blurred image 201 and the brightness saturation map 202 are connected in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may be converted into feature maps, and these feature maps may be linked in the channel direction. Also, information other than the luminance saturation map 202 may be added to the input.

A machine learning model has multiple layers, and at each layer a linear sum of layer inputs and weights is taken. The initial value of the weight should be determined by random numbers or the like. In this embodiment, the machine learning model is a CNN that uses the convolution of the input and the filter as the linear sum (the value of each element of the filter corresponds to the weight, and may include the sum with the bias). is not limited to this. Also, in each layer, nonlinear transformation is performed by an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have residual blocks and Skip Connections (also called Shortcut Connections) as needed. As a result of going through multiple layers (16 convolutional layers in this example), a saturation influence map 203 is generated.

In this embodiment, the saturation effect map 203 is obtained by summing the output of the layer 211 and the brightness saturation map 202 for each element, but the configuration is not limited to this. A saturation influence map may be generated directly as an output of layer 211 . Alternatively, the saturation effect map 203 may be the result of performing arbitrary processing on the output of the layer 211 . Next, the saturation effect map 203 and the blurred image 201 are concatenated in the channel direction and input to subsequent layers to generate the model output 204 as a result of passing through multiple layers (16 convolution layers in this embodiment). The model output 204 is also generated by taking the element-by-element sum of the output of the layer 212 and the blurred image 201, but the configuration is not limited to this. In this embodiment, convolution with 64 types of 3×3 filters is executed in each layer (the number of filter types in the layers 211 and 212 is the same as the number of channels of the blurred image 201). is not limited to

Subsequently, in step S108, the updating unit 101d updates the weight of the machine learning model based on the error function. In this example, the error function is the weighted sum of the error between the saturation influence map 203 and the correct saturation influence map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. For updating the weight, it is preferable to use an error back propagation method (Backpropagation) or the like. An error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the saturation effect map 203 and the brightness saturation map 202 and the difference component between the saturation effect correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 are used.

Subsequently, in step S109, the updating unit 101d determines whether learning of the machine learning model is completed. Completion of learning can be determined based on whether the number of weight update iterations has reached a predetermined number, whether the amount of weight change during updating is smaller than a predetermined value, and the like. If it is determined in step S109 that learning has not been completed, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, when it is determined that the learning has been completed, the updating unit 101d ends the learning, and stores the configuration and weight information of the machine learning model in the storage unit 101a.

With the above learning method, the machine learning model can estimate a saturation effect map representing the magnitude and range of signal values spread by the blur of the subject in the luminance-saturated area of the blurred image (captured image when estimating). Explicitly estimating the saturation influence map enables the machine learning model to perform blur sharpening for each saturated and non-saturated blurred image in appropriate regions, thus reducing artifacts. be.

Next, blur sharpening of a captured image using a learned machine learning model, which is executed by the image processing apparatus 103, will be described with reference to FIG. FIG. 8 is a flowchart of model output generation. The image processing apparatus 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, and each step in FIG. 8 is mainly executed by each unit of the image processing apparatus 103.

First, in step S201, the acquisition unit 103b acquires a captured image and a machine learning model. The configuration and weight information of the machine learning model are acquired from the storage unit 103a.

Subsequently, in step S202, the sharpening unit (first generating means) 103c generates information regarding correction using a machine learning model. In this embodiment, the information about correction is a sharpened blur image (model output) obtained by sharpening the blur of the captured image. It should be noted that correction components for blur sharpening may be used instead of blur sharpening images (images obtained by correcting captured images). The machine learning model has the configuration shown in FIG. 1, as during learning. As in the case of learning, a luminance saturation map representing a luminance-saturated region of a captured image is generated and input, and a saturation influence map and model output are generated. As an example, FIGS. 10A and 11A show blurred sharpened images (captured images), and FIGS. 10B and 11B show saturation effect maps. FIGS. 10A and 10B are scenes in which the average luminance value of the captured image is bright. FIGS. 11A and 11B are scenes in which the average luminance value of the captured image is dark. A method for calculating the average luminance value will be described later.

Next, with reference to FIG. 9, synthesis of a captured image and a model output, which is executed by the image processing apparatus 103, will be described. FIG. 9 is a flow chart of sharpening intensity adjustment. Each step in FIG. 9 is mainly executed by each unit of the image processing apparatus 103 .

First, in step S211, the acquisition unit 103b acquires the imaging state from the captured image. The imaging state is, for example, the aperture value (F number) of the optical system 102a and the pixel pitch of the imaging device 102b. How the streak of light and the Airy disk appear in the captured image depends on the aperture value of the optical system 102a and the pixel pitch of the image sensor 102b. Specifically, the larger the aperture value of the optical system 102a and the smaller the pixel pitch, the more conspicuous the streak of light and the Airy disk become. Therefore, a weight map is generated according to the aperture value and pixel pitch obtained in step S211.

Subsequently, in step S212, the acquisition unit 103b acquires information about the luminance value of the captured image. Here, the information about the brightness value of the captured image is a statistic about the brightness value of the captured image, and is at least one of the average value, the median value, the variance, or the histogram of the brightness value of the captured image. In addition, the statistic regarding the brightness value of the captured image, which is information regarding the brightness value of the captured image, may be related to the entire captured image, or may be a statistic for each region obtained by dividing the captured image.

In this embodiment, the average luminance value of the captured image is acquired as the information on the luminance value of the captured image. However, when obtaining the average luminance value of the captured image, if there are many saturated areas in the captured image, a large average luminance value is obtained even for a dark image such as a night scene, and the image is determined to be bright. may be Therefore, it is preferable to obtain the average luminance value of the second image from which the saturation influence map is removed from the captured image. In this case, the information about the brightness value of the captured image is generated based on the statistic about the brightness value of the second image obtained by removing the saturation influence map from the captured image. By acquiring the average luminance value of the area not saturated in the captured image in the second image, it is possible to appropriately determine whether the captured image is a bright scene or a dark scene. It should be noted that the use of the saturation influence map is not essential, and any information relating to the saturation region may be used. For example, the second image may be generated by removing the luminance saturation map from the captured image regarding the luminance saturation of the captured image.

Subsequently, in step S213, the sharpening unit 103c generates a weight map based on the information regarding the aperture value of the optical system 102a and the information regarding the luminance value of the captured image. In this embodiment, when the aperture value of the optical system 102a is F22 (predetermined aperture value) or more and the pixel pitch of the image sensor 102b is less than 6 μm (predetermined pixel pitch), the sharpening unit 103c performs intensity adjustment. . In other cases, the sharpening unit 103c generates a weight map in which the weights of the captured images are all zero. Alternatively, the processing of steps S212 to S215 may be omitted without generating the weight map. Note that the aperture value (predetermined aperture value) and the pixel pitch threshold (predetermined pixel pitch) for performing intensity adjustment can be determined to be arbitrary numerical values. For example, intensity adjustment may be performed when the aperture value of the optical system 102a is F16 or more and the pixel pitch of the image sensor 102b is less than 4 μm. Note that in this embodiment, the weight map may be generated based on the aperture value without considering the pixel pitch.

Further, in this embodiment, the intensity adjustment is not limited to whether or not it is performed with reference to a predetermined aperture value (for example, F22). For example, a weight map may be used in which the weight of the captured image changes continuously (stepwise) based on the aperture value. That is, if the aperture value can be set to the first aperture value or to a second aperture value that is larger than the first aperture value, the weight map will be set to the second aperture value rather than the first aperture value. The weight of the captured image is higher in the data.

Next, a weight map based on luminance values will be described. In this embodiment, the weight map is data in which the smaller the luminance value, the larger the weight of the captured image. For example, when the average luminance value of the captured image is used as the information about the luminance value of the captured image, the smaller the average luminance value, the greater the weight of the captured image. Specifically, the relationship between the average brightness value and the weight of the weight map is held as a linear function, the weight corresponding to the average brightness value of the captured image is obtained, and the weight map is generated.

For example, the weight of the captured image corresponding to the average luminance value obtained from the relational expression shown in FIG. 12 is used. FIG. 12 is an explanatory diagram of the weight map, in which the horizontal axis indicates the average luminance value and the vertical axis indicates the weight of the captured image. However, the relationship between the weight map adjustment value and the average luminance value is not limited to this. Although FIG. 12 shows average luminance values in an undeveloped RAW image, average luminance values after development may be used. When calculating the average luminance value in the RAW image, it is preferable to calculate the average luminance value after subtracting the signal value in the optical black area in the image sensor 102b. This makes it possible to calculate an average luminance value that does not depend on the ISO sensitivity or the image sensor 102b. Further, when the average luminance value is obtained for each region obtained by dividing the captured image, a weight may be obtained from the average luminance value of each region to generate a weight map. Details of this will be described later in the second embodiment.

Note that even when the average value, median value, variance, or histogram of the luminance values of the captured image is obtained as the statistic regarding the signal value of the captured image, the smaller the statistic, the greater the weight of the captured image. Generate a weight map. For example, when using a histogram of luminance values of a captured image, a weight map is generated such that the smaller the center of gravity or peak of the histogram, the greater the weight of the captured image.

Subsequently, in step S214 of FIG. 9, the sharpening unit 103c adjusts the weight map based on the information regarding the saturated region of the captured image. In this example, the information about the saturated region is a saturation influence map, and not all of the RGB may be saturated. Also, the saturation influence map is normalized from 0 to 1 by the set signal value, and this is used for adjustment of the weight map. Specifically, the normalized weight map is applied (multiplied) to the weight map generated in step S213. As a result, the weight of the weight map generated in step S213 acts on the saturation area influence, and intensity adjustment is possible only for the saturation influence area where the light streak and Airy disk are conspicuous. Note that application of the saturation influence map is not essential, and the entire captured image may be subject to intensity adjustment. Using a saturation influence map allows adjustment of the correction strength up to the saturation influence region. A luminance saturation map may be used instead of the saturation influence map, or a luminance saturation map obtained by blurring each image height may be used.

As an example, FIGS. 13A and 13B show final weight maps. In this embodiment, in the weight map, the larger the pixel value, the larger the weight of the captured image, and the smaller the pixel value, the smaller the weight of the captured image. FIG. 13A shows a scene with a bright average luminance value, and FIG. 13B shows a scene with a dark average luminance value. In the saturation effect region of FIG. 13B, which is a region in which adverse effects are more noticeable, the weight of the captured image is large.

Preferably, the weight map is further generated based on information regarding the optical performance of the optical system used to capture the captured image. Specifically, when the optical performance is low, the weight of the captured image is increased, and when the optical performance is high, the weight of the captured image is decreased. The information about the optical performance can be calculated based on at least one of the zoom position, the aperture diameter, or the object distance when the captured image is captured, and the magnitude and range of the signal value of the PSF for each image height of the optical system. can. It should be noted that the use of the PSF is not essential, and any information relating to optical performance may be used. For example, an optical transfer function may be used.

Subsequently, in step S215 in FIG. 9, the sharpening unit (second generating means) 103c uses the weight map generated in step S214 to perform a weighted average (synthesis) of the captured image and the blurred sharpened image, An intensity adjusted image 205 is generated.

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of maintaining the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. can.

Next, an image processing system according to Embodiment 2 of the present invention will be described. In this embodiment, an average brightness value is obtained for each region obtained by dividing the captured image, and a weight corresponding to the average brightness value is obtained for each region to generate a weight map. FIG. 14 is a block diagram of the image processing system 300 in this embodiment. FIG. 15 is an external view of the image processing system 300. As shown in FIG. The image processing system 300 has a learning device 301 , an imaging device 302 and an image processing device 303 . The learning device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are connected by a wired or wireless network.

The imaging device 302 has an optical system 321 , an imaging element 322 , a storage section 323 , a communication section 324 and a display section 325 . The captured image is transmitted to the image processing device 303 via the communication unit 324 . The image processing apparatus 303 receives the captured image via the communication unit 332 and sharpens the blur using the machine learning model configuration and weight information stored in the storage unit 331 . The configuration of the machine learning model and the weight information are learned by the learning device 301 , are obtained from the learning device 301 in advance, and are stored in the storage unit 331 . Furthermore, the image processing device 303 has a function of adjusting the intensity of blur sharpening. The sharpened blur image (model output) in which the blur of the captured image is sharpened and the intensity-adjusted image in which the intensity is adjusted are transmitted to the imaging device 302 , stored in the storage unit 323 , and displayed on the display unit 325 .

Learning data generation and weight learning (learning phase) performed by the learning device 301, and blur sharpening (estimation phase) of the captured image using the learned machine learning model performed by the image processing device 303 are performed. Since it is the same as Example 1, it is omitted.

Next, with reference to FIG. 16, synthesis of a captured image and a model output, which is executed by the image processing device 303, will be described. FIG. 16 is a flowchart of sharpening intensity adjustment. Each step in FIG. 16 is mainly executed by each unit of the image processing apparatus 303 .

First, in step S311, the acquisition unit 333 acquires the imaging state from the captured image. The imaging state includes, but is not limited to, the aperture value of the optical system 321 in the imaging device 302 and the state of the pixel pitch of the imaging device 322 . In this embodiment, the intensity adjustment is performed when the aperture value of the optical system 321 is F16 or more and the pixel pitch of the imaging element 322 is 4 μm or more. In other cases, in generating a weight map to be described later, a weight map is generated in which the weights of the captured images are all 0, and intensity adjustment is not performed. Alternatively, the processing of steps S312 to S314 may be omitted without generating the weight map.

Subsequently, in step S312, the acquisition unit 333 acquires information about the luminance value of the captured image. In this embodiment, an average brightness value is obtained for each region obtained by dividing the captured image, and a weight corresponding to the average brightness value is obtained for each region to generate a weight map. 17A and 17B are explanatory diagrams of weight maps, FIG. 17A showing a captured image, and FIG. 17B showing an average luminance value for each area. In FIG. 17B, 2001 is a low average brightness value area, 2002 is a high average brightness value area, and 2003 is an intermediate average brightness value area.

Subsequently, in step S313 of FIG. 16, the sharpening unit 334 generates a weight map based on the aperture value of the optical system 321 and the information on the brightness value of the captured image. Note that the method of generating the weight map is the same as that of the first embodiment, so the description thereof will be omitted. Subsequently, in step S314, the sharpening unit 334 uses the weight map generated in step S312 to perform a weighted average (synthesis) of the captured image and the blurred sharpened image (model output) to generate the intensity adjusted image 205. .

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of maintaining the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. can.

Next, an image processing system in Example 3 of the present invention will be described. FIG. 18 is a block diagram of an image processing system 400 in this embodiment. FIG. 19 is an external view of the image processing system 400. As shown in FIG. The image processing system 400 has a learning device 401 , a lens device 402 , an imaging device 403 , a control device (first device) 404 , an image estimation device (second device) 405 , and networks 406 and 407 .

Each of the learning device 401 and the image estimation device 405 is, for example, a server. A control device 404 is a device operated by a user such as a personal computer or a mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and is a machine learning model for sharpening blur from an image captured using the lens device 402 and the imaging device 403. learn the weights of Note that the learning method, that is, the generation of learning data and the learning of weights (learning phase) performed by the learning device 401 are the same as those in the first embodiment, and therefore will be omitted.

The imaging device 403 has an imaging device 403a, and the imaging device 403a photoelectrically converts an optical image formed by the lens device 402 to obtain a captured image. The lens device 402 and the imaging device 403 are detachable and can be combined with each other. The control device 404 includes a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d. Control according to the operation. Alternatively, the captured image captured by the imaging device 403 may be stored in the storage unit 404c in advance, and the captured image may be read.

The image estimation device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d, and is configured to communicate with the control device 404. FIG. The image estimating device 405 executes blur sharpening processing of the captured image in response to a request from the control device 404 connected via the network 406 . The image estimating device 405 acquires learned weight information from the learning device 401 connected via the network 406 at the time of estimating blur sharpening or in advance, and uses it for estimating blur sharpening of the captured image. The estimated image after estimating blur sharpening is transmitted to the control device 404 again after sharpening intensity adjustment, stored in the storage unit 404c, and displayed on the display unit 404b.

Next, with reference to FIG. 20, blur sharpening of the captured image executed by the control device 404 and the image estimation device 405 will be described. FIG. 20 is a flowchart of model output and sharpening intensity adjustment. Each step in FIG. 20 is mainly executed by each part of the control device 404 or the image estimation device 405 .

First, in step S401, the acquisition unit 404d of the control device 404 acquires the captured image and the sharpening intensity specified by the user. Subsequently, in step S<b>402 , the communication unit (transmitting means) 404 a transmits to the image estimating device 405 a request regarding execution of the captured image and blur sharpening estimation processing.

Subsequently, in step S403, the communication unit (receiving means) 405a of the image estimation device 405 receives and acquires the captured image and the processing request transmitted from the control device 404. FIG. Subsequently, in step S404, the acquisition unit 405b acquires the learned weight information corresponding to the captured image from the storage unit 405c. The weight information is read in advance from the storage unit 401a and stored in the storage unit 405c. Subsequently, in step S405, the sharpening unit 405d uses a machine learning model to generate a blurred sharpened image (model output) in which the blur of the captured image is sharpened from the captured image. The machine learning model has the configuration shown in FIG. 1, as during learning. As in the case of learning, a luminance saturation map representing a luminance-saturated region of a captured image is generated and input, and a saturation influence map and model output are generated.

Subsequently, in step S406, the sharpening unit 405d generates a weight map. The weight map generation method is the same as in the first embodiment. Adjust the default weight map for the user-specified sharpening strength. Note that a pre-adjusted weight map may be held within a range in which the intensity can be adjusted. Subsequently, in step S407, the sharpening unit 405d synthesizes the captured image and the blurred sharpened image (model output) based on the weight map. Subsequently, in step S408, the communication unit 405a transmits the composite image to the control device 404. FIG. Subsequently, in step S<b>409 , the communication unit 404 a of the control device 404 acquires the estimated image transmitted from the image estimation device 405 .

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of maintaining the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. can.
(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions. The image processing apparatus is sufficient as long as it has the image processing function of the present invention, and can be implemented in the form of an imaging apparatus or a PC.

According to each embodiment, an image processing method, an image processing apparatus, an image processing program, and an image processing method capable of maintaining blur correction effects while suppressing adverse effects caused by the aperture value of an optical system and the brightness value of a captured image. , can provide a storage medium.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

103 Image processing device 103c sharpening unit (first generating means, second generating means)

Claims (16)

generating information about correction of the captured image based on the captured image;
generating an intensity-adjusted image based on the captured image, the information about the correction, and a weight map;
An image processing method, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image. 2. The image processing method according to claim 1, wherein said weight map is generated based on information regarding a saturated region in said captured image. 3. The image according to claim 2, wherein the information about the saturated region is a saturation effect map representing the magnitude and range of signal values spread by the subject in the saturated region of the captured image due to blurring during imaging. Processing method. 4. The image processing method of claim 3, wherein the saturation influence map is generated with a machine learning model. 5. The image processing method according to claim 1, wherein the weight map is generated based on information regarding optical performance of an optical system used to capture the captured image. The information about the optical performance includes at least one of the zoom position of the captured image, the aperture value, or the subject distance, and the magnitude and range of the signal value of the point spread function for each image height of the optical system. 6. The image processing method according to claim 5, wherein the calculation is performed based on 7. The image processing method according to claim 1, wherein the information about the correction is an image obtained by correcting the captured image or a correction component. 8. The image processing method according to any one of claims 1 to 7, wherein the correction is to increase the resolution of the captured image or to transform the shape of defocus blur of the captured image. 5. The method according to claim 3, wherein the information about the brightness value of the captured image is generated based on statistics about the brightness value of a second image obtained by removing the saturation effect map from the captured image. Image processing method. 10. The image processing method according to claim 9, wherein said statistic is at least one of an average value, a median value, a variance, or a histogram. 9. The image processing method according to any one of claims 1 to 8, wherein the weight map is data in which the smaller the luminance value, the larger the weight of the captured image. 11. The image processing method according to claim 9 or 10, wherein the weight map is data in which the smaller the average brightness value in the second image, the larger the weight of the captured image. The aperture value can be set to a first aperture value or a second aperture value larger than the first aperture value,
13. The image according to any one of claims 1 to 12, wherein the weight map is data in which the weight of the captured image is greater at the second aperture value than at the first aperture value. Processing method. a first generating means for generating information regarding correction of the captured image based on the captured image;
a second generation means for generating an intensity-adjusted image based on the captured image, the information about the correction, and the weight map;
The image processing apparatus, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image. An image processing system having a first device and a second device communicable with each other,
The first device has transmission means for transmitting a request for executing processing on the captured image to the second device,
The second device is
receiving means for receiving the request;
a first generating means for generating information regarding correction of the captured image based on the captured image;
a second generation means for generating an intensity-adjusted image based on the captured image, the information about the correction, and the weight map;
The image processing system, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image. A program that causes a computer to execute the image processing method according to any one of claims 1 to 13.