JP2023088349A - Image processing method, image processing apparatus, image processing system, and program - Google Patents

Abstract

To provide an image processing method for properly adjusting correction strength in accordance with settings configured by a user, in a regression task using machine learning on a blurred image, an image processing apparatus, an image processing system, and a program.SOLUTION: An image processing method includes: a step of inputting a captured image into a machine learning model to generate information on correction; a step of generating a strength-adjusted image on the basis of, the captured image, the correction information, and a weight map; and a step of adjusting correction strength of the strength-adjusted image by modifying the weight map on the basis of the set strength. The correction strength includes first correction strength in a first region of the strength-adjusted image, based on information on optical performance of an optical system used in imaging of the captured image and information on a saturated region, and second correction strength in a second region of the strength-adjusted image. A ratio of the second correction strength with respect to the first correction strength is varied on the basis of, the set strength.SELECTED DRAWING: Figure 8

Description

The present invention relates to an image processing method, an image processing apparatus, an image processing system, and a program.

In image recognition or regression tasks, methods using machine learning models can achieve higher accuracy than theory-based methods using assumptions and approximations. Theory-based methods suffer from inaccuracies due to factors neglected by assumptions and approximations. However, in the method using a machine learning model, by training the machine learning model using training data that includes these elements, it is possible to realize estimation that is based on the training data without assumptions or approximations, so task accuracy is improved. improves.

Patent Literature 1 discloses a method of sharpening blur in a captured image using a convolutional neural network (CNN), which is one of machine learning models. Further, Japanese Patent Application Laid-Open No. 2002-200000 discloses a method of weighted averaging a captured image and an estimated image (blurred sharpened image) based on a luminance saturation region to adjust the sharpening strength.

Japanese Patent Application Laid-Open No. 2020-166628

By the way, there is a need for the user to set the sharpening strength (correction strength). When the correction strength is increased according to user settings, the method disclosed in Patent Document 1 increases the correction strength in the luminance saturated region and the non-saturated region at a constant ratio. Insufficient or excessive correction strength in luminance saturation areas may occur.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing method capable of appropriately adjusting correction strength according to user settings in a regression task using machine learning for a blurred image.

An image processing method as one aspect of the present invention includes steps of inputting a captured image into a machine learning model to generate information about correction, and generating an intensity-adjusted image based on the captured image, the information about correction, and a weight map. and adjusting the correction strength of the strength-adjusted image by changing the weight map based on the set strength, wherein the correction strength is the optical system used to capture the captured image. a first correction strength in a first region of the intensity-adjusted image and a second correction strength in a second region of the intensity-adjusted image based on information about the optical performance and information about the saturation region of A ratio of the second correction strength to the first correction strength varies based on the set strength.

Other objects and features of the invention are illustrated in the following examples.

According to the present invention, it is possible to provide an image processing method capable of appropriately adjusting correction strength according to user settings in a regression task using machine learning for a blurred image.

FIG. 4 is an explanatory diagram of a machine learning model in Example 1; 1 is a block diagram of an image processing system in Example 1. FIG. 1 is an external view of an image processing system in Example 1. FIG. FIG. 5 is an explanatory diagram of adverse effects caused by sharpening in Examples 1 to 3; FIG. 5 is an explanatory diagram of adverse effects caused by sharpening intensity adjustment in Examples 1 to 3; 4 is a flow chart of learning of a machine learning model in Examples 1 to 3. FIG. 4 is a flow chart of model output generation in Examples 1 and 2. FIG. 5 is a flowchart of sharpening intensity adjustment in the first embodiment. 4A and 4B are explanatory diagrams of a captured image and a saturation effect map in Example 1. FIG. FIG. 4 is a diagram showing the relationship between optical performance indices and weights in Examples 1 to 3; FIG. 4 is an explanatory diagram of a weight map in Example 1; FIG. 4 is an explanatory diagram of a user interface in Example 1; FIG. 4 is an explanatory diagram of correction strength in Example 1; FIG. 11 is a block diagram of an image processing system in Example 2; FIG. 11 is an external view of an image processing system in Example 2; 10 is a flowchart of sharpening intensity adjustment in Example 2. FIG. FIG. 11 is an explanatory diagram of a user interface in Example 2; FIG. 11 is a block diagram of an image processing system in Example 3; FIG. 11 is an external view of an image processing system in Example 3; 10 is a flowchart of model output and sharpening intensity adjustment in Example 3. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

The gist of the present invention will be described before the specific description of the embodiments. The present invention generates an estimated image (blur sharpened image) by sharpening the blur caused by the optical system using a machine learning model from an image captured using the optical system. Then, the upper limit value of the weight map is determined based on the luminance saturation region where the luminance is saturated, and the captured image and the estimated image are weighted and added. The weight map is used to determine the proportion of each image in the weighted addition of the captured image and the estimated image, and has continuous signal values. For example, if the numerical value of the weight map determines the percentage of blurred sharpened images, a numerical value of 0 leaves the intensity adjusted image as the captured image. Also, if the value of the weight map is 1, the intensity-adjusted image becomes the blurred sharpened image.

Blurring caused by the optical system includes blurring due to aberration, diffraction, defocus, action of an optical low-pass filter, deterioration of the pixel aperture of the imaging element, and the like. Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, and the like. Neural networks include Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs), and the like.

Blur sharpening refers to processing for restoring frequency components of an object that have been reduced or lost due to blur. When blur is sharpened, undershoot, ringing, or the like cannot be suppressed depending on the captured image, and these adverse effects may occur in the estimated image. Specifically, when the subject is largely blurred due to the aberration of the optical system, or when the image has a brightness saturation area, a problem occurs. A luminance saturated region may occur in an image depending on the dynamic range of the image sensor and the exposure at the time of imaging. In the luminance saturated region, information about the structure of the subject space cannot be obtained, which tends to cause problems. These adverse effects can be suppressed by performing weighted addition of the captured image and the estimated image based on the luminance-saturated region and the optical performance.

However, it is preferable to have a function of adjusting the correction strength in order to meet the needs of increasing the correction strength even if some adverse effects are allowed. Here, if the upper limit value of the correction strength is the same in the luminance saturated region and the non-saturated region (the correction strength in the luminance saturated region and the non-saturated region is changed at a constant ratio), an appropriate Intensity adjustments cannot be made. In other words, if the correction strength is increased, the correction strength in the non-saturated area is insufficient (when the upper limit is set to the allowable amount of adverse effects in the saturated area), or the correction strength in the luminance saturated area is excessive (the upper limit is set to the non-saturated area). (when it is adjusted to the allowable amount of harmful effects).

Therefore, in each embodiment, the upper limit value of the weight map is determined for each of the luminance saturated region and the non-saturated region (the ratio of the correction strength of the luminance saturated region and the non-saturated region is changed based on the set strength), and the captured image and Weighted addition is performed with the estimated image. This makes it possible to control the sharpness and harmful effects of blurring. In the following, the stage of learning the weights of the machine learning model is referred to as the learning phase, and the stage of sharpening the blur by the machine learning model using the learned weights is referred to as the estimation phase.

First, an image processing system according to Embodiment 1 of the present invention will be described. In this embodiment, the task of the machine learning model is to sharpen the blur of a captured image including brightness saturation (increase the resolution of the captured image). Further, the sharpening blur refers to blur caused by aberration and diffraction generated in an optical system and an optical low-pass filter. However, the effect of the invention can be similarly obtained in the case of sharpening pixel aperture, defocus, and blur due to blurring. Moreover, it is possible to implement the present embodiment in the same manner for tasks other than blur sharpening, and to obtain effects. Specifically, tasks such as upsampling to increase the number of pixels in a captured image, and conversion (shape conversion) of defocus blur in the captured image. Defocus blur conversion includes, for example, conversion from bilinear blur to Gaussian blur or ball blur. A bilinear blur has a PSF with separated peaks. As a result, the subject, which is originally a single line, appears to be double-blurred when defocused. Ball blur has a PSF that is flat in intensity. Gaussian blur has a Gaussian-distributed PSF. Other defocus blur to be converted includes, for example, defocus blur caused by vignetting, ring-shaped defocus blur caused by pupil shielding such as a catadioptric lens, and the like.

FIG. 2 is a block diagram of the image processing system 100 in this embodiment. FIG. 3 is an external view of the image processing system 100. As shown in FIG. The image processing system 100 has a learning device 101 and an image processing device 103 connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 by wire or wirelessly. A captured image obtained by capturing an object space using the imaging device 102 is input to the image processing device 103 . The captured image is blurred due to aberration and diffraction by the optical system 102a in the image capturing apparatus 102 and the optical low-pass filter of the image sensor 102b, and the subject information is attenuated.

The image processing apparatus 103 uses a machine learning model to perform blur sharpening on the captured image, and generates a saturation effect map and a blur sharpened image (model output). The details of the saturation influence map will be described later. The machine learning model is learned by the learning device 101, and the image processing device 103 acquires information on the machine learning model in advance from the learning device 101 and stores it in the storage unit 103a. The image processing apparatus 103 also has a function of adjusting the intensity of blur sharpening by performing weighted addition of the captured image and the sharpened blur image. The details of the learning and estimation of the machine learning model and the adjustment of the blur sharpening intensity will be described later. The user can adjust the blur sharpening intensity while checking the image displayed on the display device 104 . The blurred and sharpened image that has undergone intensity adjustment is stored in the storage unit 103a or the recording medium 105, and is output to an output device 106 such as a printer as necessary. Note that the captured image may be grayscale or may have a plurality of color components. Further, it may be an undeveloped RAW image or an image after development.

Next, with reference to FIGS. 4A to 4C, a decrease in estimation accuracy (evil due to sharpening) that occurs when blur sharpening is performed by a machine learning model will be described. FIGS. 4A to 4C are explanatory diagrams of adverse effects caused by sharpening, and show spatial changes in signal values of an image. Here, the saturation value is 255 because the image is an 8-bit image. In FIGS. 4A to 4C, a solid line indicates a captured image (blurred image), and a dotted line indicates a sharpened blurred image obtained by sharpening the blur of the captured image using a machine learning model. FIG. 4A shows a non-saturated brightness subject with large blur due to optical system aberration, FIG. 4B shows a non-saturated brightness subject with small blur due to optical system aberration, and FIG. 4C shows blur due to optical system aberration This is the result of sharpening a luminance-saturated subject with a small .

When the blur due to the aberration of the optical system is large, undershoot occurs on the dark side of the edge. Also, even if the blur due to optical system aberrations is small, when sharpening a subject with saturated brightness, undershoots that do not occur with non-saturated subjects and pixel values that were originally saturated will decrease. are doing. In an area where brightness saturation occurs (brightness saturated area), information about the structure of the subject space is lost, and false edges may appear at the boundaries of each area, making it impossible to extract the correct feature amount of the subject. Therefore, the estimation accuracy of the machine learning model is lowered. From the above results, it can be seen that the adverse effects associated with sharpening depend on the performance of the optical system and the brightness saturation region.

For the above-mentioned correction, a machine learning model is used that incorporates a method that uses a captured image and a luminance saturation map corresponding to the captured image as input data for the machine learning model, and a method that generates a saturation effect map. . In other words, it is possible to reduce the harmful effects by using these methods, but it is difficult to eliminate the harmful effects completely. A technique using a luminance saturation map and a technique for generating a saturation influence map will be described in detail.

First, the luminance saturation map will be explained. A brightness saturation map is a map that represents a brightness saturation region in a captured image. In an area where brightness saturation occurs (brightness saturated area), information about the structure of the subject space is lost, and false edges may appear at the boundaries of each area, making it impossible to extract the correct feature amount of the subject. Therefore, by inputting the brightness saturation map, the neural network can identify the problem area as described above, so that the deterioration of the estimation accuracy can be suppressed.

Next, the saturation influence map will be described. Even with a luminance saturation map, machine learning models may not make correct decisions. For example, when a region of interest is in the vicinity of a region with saturated brightness, the machine learning model can determine that the region of interest is a region affected by saturation of brightness because there is a region with saturated brightness in the vicinity of the region of interest. However, if the region of interest is located at a position away from the luminance saturated region, it is not easy to determine whether or not this region is affected by luminance saturation, resulting in high ambiguity. As a result, the machine learning model may make an erroneous decision at a position away from the luminance saturated region. As a result, when the task is blur sharpening, the sharpening process corresponding to the saturated blurred image is performed on the non-saturated blurred image. At this time, artifacts are generated in the sharpened blur image, and the accuracy of the task is lowered. Therefore, it is preferable to generate a saturation effect map from a blurred captured image using a machine learning model.

A saturation effect map is a map (a signal sequence arranged spatially) that represents the magnitude and range of signal values that have spread due to blurring (degradation) during imaging of an object in a luminance saturation region of a captured image. By having the machine learning model generate the saturation influence map, the machine learning model can highly accurately estimate the presence or absence of the influence of luminance saturation in the captured image and the magnitude thereof. By generating the saturation influence map, the machine learning model can execute the processing to be executed on the regions affected by the luminance saturation and the processing to be executed on the other regions in each appropriate region. can. Therefore, by letting the machine learning model generate the saturation influence map, the accuracy of the task is improved compared to the case where the saturation influence map is not generated (only the recognition labels and the blur sharpened image are generated directly from the captured image). improves.

Although the above two methods are effective, it is difficult to completely eliminate the harmful effects as described with reference to FIGS. Therefore, the adverse effect can be suppressed by performing weighted addition of the captured image and the sharpened blurred image. The dashed-dotted lines in FIGS. 4A and 4C are signal values obtained by weighting and adding the captured image and the sharpened blurred image. The weighted addition reduces undershoot in dark areas and decreases in pixel values that were originally saturated, while maintaining the sharpening effect of blurring. In this embodiment, a weight map used for weighted addition of the captured image and the sharpened blurred image is generated based on the performance of the optical system and the brightness saturation region. Based on the performance of the optical system and the brightness saturation region, while maintaining the sharpness of FIG. Therefore, it is possible to control the sharpness and adverse effects of blurring.

Next, the intensity adjusting function of the correction intensity will be described. It is preferable to have a function of adjusting the correction strength in order to meet the need to increase the correction strength even if some adverse effects are allowed. In that case, it is necessary to determine the upper limit of the correction strength. is excessive. A luminance saturation region includes a saturation influence map.

FIGS. 5A and 5B are explanatory diagrams of the adverse effect of sharpening intensity adjustment, and the sharpening intensity is adjusted by changing the weight map used when performing weighted addition of the captured image and the blurred sharpened image. shows the change in signal value (spatial change in the signal value of the image) in each case. Since the image is an 8-bit image, the saturation value is 255. FIG. 5A shows signal value changes in a saturated luminance region (region where luminance is saturated), and FIG. 5B shows signal value changes in a non-saturated region (region where luminance is not saturated). A dashed line 1001 is the signal value of the captured image. A dotted line 1002 is the signal value of an intensity-adjusted image obtained by adjusting the sharpness intensity of blur using the weight map. A dashed-dotted line 1003 is the maximum intensity image obtained by doubling the corrected intensity with respect to the intensity adjusted image.

In the case of FIG. 5A, an undershoot occurs on the dark side of the edge in the maximum intensity image. On the other hand, in FIG. 5B, no undershoot occurs even in the maximum intensity image. That is, when the correction strength is increased, adverse effects occur first in the luminance saturation region. A two-dot chain line 1004 is the signal value of the intensity-adjusted image in which the brightness saturation region is clipped with a correction strength of 100% and the non-saturation region is clipped with a correction strength of 200%. The correction strength of 100% is the sharpened blur image output by the machine learning model, and the correction strength of 200% means that the absolute value of the difference between the sharpened blur image and the captured image is doubled and added to the captured image. means to According to the two-dot chain line 1004, it is possible to suppress the adverse effect of the luminance saturated region while maintaining the correction strength of the non-saturated region.

Next, with reference to FIG. 6, learning of the machine learning model executed by the learning device 101 will be described. FIG. 6 is a flowchart of learning of the machine learning model. The learning device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an updating unit 101d, and each step in FIG.

First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is the image having signal values higher than the second signal value. The second signal value is a signal value corresponding to the luminance saturation value of the captured image. However, since the signal value may be normalized when inputting to the machine learning model, the second signal value and the luminance saturation value of the captured image do not necessarily match. Since the machine learning model is trained based on the original image, the original image should preferably be an image with various frequency components (edges with different orientations and strengths, gradations, flat areas, etc.). The original image may be a photographed image or CG (Computer Graphics).

Subsequently, in step S102, the calculation unit 101c blurs the original image to generate a blurred image. A blurred image is an image input to a machine learning model during learning, and corresponds to a captured image during estimation. The blur to be given is the blur to be sharpened. In this embodiment, the blur generated by the aberration and diffraction of the optical system 102a and the optical low-pass filter of the imaging element 102b is given. The shape of the blur due to aberration and diffraction of the optical system 102a changes depending on the image plane coordinates (image height and azimuth). It also changes depending on the state of zooming, aperture, and focus of the optical system 102a. When learning a machine learning model that sharpens all of these blurs at once, it is preferable to generate a plurality of blurred images using a plurality of blurs generated by the optical system 102a. Also, in the blurred image, signal values exceeding the second signal value are clipped. This is done in order to reproduce the brightness saturation that occurs during the imaging process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

Subsequently, in step S103, the calculation unit 101c sets the first region based on the image based on the original image and the threshold value of the signal value. In this embodiment, a blurred image is used as an image based on the original image, but the original image itself may be used. A first region is set by comparing the signal value of the blurred image and the threshold value of the signal value. More specifically, a region in which the signal value of the blurred image is equal to or greater than the signal value threshold is defined as the first region. In this embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness-saturated region (saturated region) of the blurred image. However, the signal value threshold and the second signal value do not have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

Subsequently, in step S104, the calculation unit 101c generates a first area image having signal values of the original image in the first area. The first area image has signal values different from those of the original image in areas other than the first area. More desirably, the first area image has the first signal value in areas other than the first area. In this embodiment, the first signal value is 0, but the invention is not so limited. In this embodiment, the first area image has the signal value of the original image only in the area where the brightness of the blurred image is saturated, and the signal value is 0 in other areas.

Subsequently, in step S105, the calculation unit 101c blurs the first area image and generates a saturation influence correct map. The added blur is the same as the blur added to the blurred image. As a result, a saturated effect correct map, which is a map (a signal train arranged spatially) representing the magnitude and range of signal values spread due to degradation during imaging, is generated from the subject in the luminance saturated area of the blurred image. be. In the present embodiment, the saturation effect correct map is clipped by the second signal value in the same manner as the blurred image, but clipping is not necessarily required.

Subsequently, in step S106, the acquisition unit 101b acquires the correct model output. Since the task of this embodiment is blur sharpening, the correct model output is an image with less blur than the blurred image. In this embodiment, the correct model output is generated by clipping the original image with a second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is performed in the same way when generating a blurred image in step S102. Further, step S106 may be executed at any time after step S101 and before step S107.

Subsequently, in step S107, the calculation unit 101c uses a machine learning model to generate a saturation influence map and a model output based on the blurred image. In this example, the machine learning model shown in FIG. 1 is used, but the invention is not limited to this. A blurred image 201 and a luminance saturation map 202 are input to a machine learning model. A brightness saturation map 202 is a map that indicates a brightness-saturated region (where the signal value is equal to or greater than the second signal value) of the blurred image 201 . For example, it can be generated by binarizing the blurred image 201 with the second signal value. However, the brightness saturation map 202 is not necessarily essential. The blurred image 201 and the brightness saturation map 202 are connected in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may be converted into feature maps, and these feature maps may be linked in the channel direction. Also, information other than the luminance saturation map 202 may be added to the input.

A machine learning model has multiple layers, and at each layer a linear sum of layer inputs and weights is taken. The initial value of the weight should be determined by random numbers or the like. In this embodiment, the machine learning model is a CNN that uses the convolution of the input and the filter as the linear sum (the value of each element of the filter corresponds to the weight, and may include the sum with the bias). is not limited to this. Also, in each layer, nonlinear transformation is performed by an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have residual blocks and Skip Connections (also called Shortcut Connections) as needed. As a result of going through multiple layers (16 convolutional layers in this example), a saturation influence map 203 is generated.

In this embodiment, the saturation effect map 203 is obtained by summing the output of the layer 211 and the brightness saturation map 202 for each element, but the configuration is not limited to this. A saturation influence map may be generated directly as an output of layer 211 . Alternatively, the saturation effect map 203 may be the result of performing arbitrary processing on the output of the layer 211 . Next, the saturation effect map 203 and the blurred image 201 are concatenated in the channel direction and input to subsequent layers to generate the model output 204 as a result of passing through multiple layers (16 convolution layers in this embodiment). The model output 204 is also generated by taking the element-by-element sum of the output of the layer 212 and the blurred image 201, but the configuration is not limited to this. In this embodiment, convolution with 64 types of 3×3 filters is executed in each layer (the number of filter types in the layers 211 and 212 is the same as the number of channels of the blurred image 201). is not limited to

Subsequently, in step S108, the updating unit 101d updates the weight of the machine learning model based on the error function. In this example, the error function is the weighted sum of the error between the saturation influence map 203 and the correct saturation influence map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. For updating the weight, it is preferable to use an error back propagation method (Backpropagation) or the like. An error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the saturation effect map 203 and the brightness saturation map 202 and the difference component between the saturation effect correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 are used.

Subsequently, in step S109, the updating unit 101d determines whether learning of the machine learning model is completed. Completion of learning can be determined based on whether the number of weight update iterations has reached a predetermined number, whether the amount of weight change during updating is smaller than a predetermined value, and the like. If it is determined in step S109 that learning has not been completed, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, when it is determined that the learning has been completed, the updating unit 101d ends the learning, and stores the configuration and weight information of the machine learning model in the storage unit 101a.

With the above learning method, the machine learning model can estimate a saturation effect map representing the magnitude and range of signal values spread by the blur of the subject in the luminance-saturated area of the blurred image (captured image when estimating). Explicitly estimating the saturation influence map enables the machine learning model to perform blur sharpening for each saturated and non-saturated blurred image in appropriate regions, thus reducing artifacts. be.

Next, blur sharpening of a captured image using a learned machine learning model, which is executed by the image processing apparatus 103, will be described with reference to FIG. FIG. 7 is a flowchart of model output generation. The image processing apparatus 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, and each step in FIG. 7 is mainly executed by each unit of the image processing apparatus 103.

First, in step S201, the acquisition unit 103b acquires a captured image and a machine learning model. The configuration and weight information of the machine learning model are acquired from the storage unit 103a.

Subsequently, in step S202, the sharpening unit (first generating means) 103c generates information regarding correction using a machine learning model. In this embodiment, the information about correction is a sharpened blur image (model output) obtained by sharpening the blur of the captured image. It should be noted that correction components for blur sharpening may be used instead of blur sharpening images (images obtained by correcting captured images). The machine learning model has the configuration shown in FIG. 1, as in learning. As in the case of learning, a luminance saturation map representing a luminance-saturated region of a captured image is generated and input, and a saturation influence map and model output are generated. As an example, FIG. 9A shows a blurred sharpened image (captured image), and FIG. 9B shows a saturation effect map.

Next, with reference to FIG. 8, synthesis of a captured image and a model output, which is executed by the image processing apparatus 103, will be described. FIG. 8 is a flowchart of sharpening intensity adjustment. Each step in FIG. 8 is mainly executed by each unit of the image processing apparatus 103 .

First, in step S211, the acquisition unit 103b acquires the shooting state from the captured image. The photographing state is (z, f, d) indicating the zoom position, aperture diameter, and object distance of the optical system 102a, and the pixel pitch of the image sensor 102b.

Subsequently, in step S212, the acquisition unit 103b acquires information (optical performance index) on the optical performance of the optical system 102a based on the shooting state acquired in step S211. The optical performance index is stored in the storage unit 103a. The optical performance index is information about the optical performance of the optical system 102a used to capture the captured image that is independent of the subject space, and does not include information that is not independent of the subject space, such as a saturation effect map. In this embodiment, the magnitude (peak value) and range (spreading degree) of the point spread function (PSF) are used as the optical performance index. A peak value is the maximum signal value that the PSF has, and a range means the number of pixels having a value equal to or greater than a certain threshold. When sharpening blur with a machine learning model, even if the peak value is the same, blur with a smaller number of pixels having a value equal to or greater than a certain threshold produces less adverse effects. Use Also, since the peak value and range of the PSF depend on the pixel pitch of the imaging device 102, optical performance indices corresponding to a plurality of pixel pitches are stored and intermediate values are created by interpolation.

Note that the optical performance index may be any index that reflects the optical performance, so another numerical value such as an optical transfer function (OTF) may be used as the optical performance index. For example, the optical performance index is calculated based on at least one of the zoom position, aperture diameter, and subject distance when capturing the captured image, and the magnitude and range of the signal value of the point spread function for each image height of the optical system. be done.

The storage unit 103a stores (stores) only optical performance indices for discretely selected imaging states in order to reduce the number of optical performance indices (the number of data). Therefore, if the optical performance index corresponding to the imaging state acquired in step S211 or the optical performance index corresponding to the imaging state close to the imaging state is not stored in the storage unit 103a, the optical performance index that is as close as possible to the imaging state is stored in the storage unit 103a. to select. Then, by correcting the optical performance index so as to optimize it for the imaging state acquired in step S211, an optical performance index to be actually used is created.

Subsequently, in step S213, the sharpening unit 103c generates a first weight map and a second weight map from the optical performance index. When determining the weight, the weight of the blurred sharpened image is determined for each image height from the relational expression shown in FIG. 10 based on the optical performance index acquired in step S212. FIG. 10 is a diagram showing the relationship between optical performance indices and weights. In FIG. 10, the horizontal axis indicates the optical performance index, and the vertical axis indicates the weight of the sharpened blurred image.

A solid line 121 is for the non-saturated region, and a dotted line 122 is for the luminance saturated region. Even if the optical performance index is the same, the weight of the blurred sharpened image is lowered in the luminance saturated region where adverse effects are likely to occur. Although a linear expression is used in this embodiment, the relational expression is not limited to a linear expression. Also, the relational expression can be freely changed. An image height for which no optical performance index is held is generated by interpolation from the held image height point. A first weight map for the non-saturated region is generated from the solid line 121, and a second weight map for the luminance saturated region is generated from the dotted line 122. FIG.

Subsequently, in step S214 of FIG. 8, the sharpening unit 103c generates weight maps based on the first weight map, the second weight map, and the saturation influence map. That is, the weight map is generated based on information on optical performance (first weight map, second weight map) and information on the saturated region of the captured image. In this example, the information about the saturated region is a saturation influence map, and not all of the RGB may be saturated. The saturation influence map is also normalized by the second signal value and used to combine the first weight map and the second weight map. That is, when the normalized numerical value falls between 0 and 1, both the first weight map and the second weight map contribute. Using a saturation influence map allows adjustment of the correction strength up to the saturation influence region. It should be noted that it is not essential to use a saturation influence map, and a brightness saturation map may be used, or a brightness saturation map obtained by blurring each image height may be used.

FIGS. 11A to 11C are explanatory diagrams of weight maps as an example. In this embodiment, in the weight map, the larger the pixel value, the stronger the correction strength, and the smaller the pixel value, the weaker the correction strength. In a general optical system, the optical performance degrades as it goes off-axis, so a gradation weight map such as that shown in FIG. 11B is often used. In addition, even if the optical performance is the same, the saturation effect area is likely to be detrimental, so the correction strength is reduced even if the image height is the same. In the weight map, the smaller the pixel value, the stronger the correction strength, and the larger the pixel value, the weaker the correction strength.

Subsequently, in step S215 of FIG. 8, the acquiring unit 103b acquires the sharpening strength specified (set) by the user. An example of intensity adjustment by the user will be described with reference to FIG. FIG. 12 is an explanatory diagram of the user interface. The user can adjust the correction strength by moving the slider to change the set strength. Alternatively, a numerical value for the set intensity may be entered. For example, if the set strength is 1.0, the default strength adjusted image is obtained by synthesizing the captured image and the model output based on the weight map generated in step S214. When the set intensity is set to 1.5, an intensity-adjusted image is generated in which the corrected intensity is 1.5 times the default intensity.

Subsequently, in step S216, the sharpening unit 103c changes the weight map according to the acquired sharpening strength. FIG. 11 shows a modified example of the weight map, and FIG. 11B shows the weight map when the set strength is 1.0, which is the default weight map generated in step S211. FIG. 11(C) shows a weight map in the case of a set strength of 2.0, in which the default weight map is uniformly doubled on the screen. FIG. 11A shows a weight map when the set strength is 0, and the weights are all 0. FIG.

Here, changes in correction strength will be described with reference to FIG. FIG. 13 is an explanatory diagram of correction strength. Reference numeral 1101 denotes changes in correction strength in non-saturated regions, and reference numeral 1102 denotes changes in correction strength in luminance saturated regions. Both have the same image height and are areas with high optical performance. In this embodiment, the correction strength in the non-saturated region is defined as the first correction strength, and the correction strength in the luminance saturated region is defined as the second correction strength. The correction strength of the saturation influence region clips at 100%. Assuming that the set strength of 1.0 is the first set strength and the set strength of 2.0 is the second set strength, the first correction strength and the second correction strength are obtained at the first set strength and the second set strength. ratio is different. That is, the ratio of the second correction strength to the first correction strength changes based on the set strength.

In FIG. 13, reference numeral 1103 denotes changes in correction strength in non-saturated regions, and 1104 denotes changes in correction strength in luminance saturated regions. Both have the same image height and are regions with low optical performance. Since the optical performance is low, the default correction strength is low, and even with the set strength of 2.0, the correction strength in the luminance saturation region does not reach 100%. As described above, the upper limit of the correction strength differs depending on the optical performance. Note that the relational expression between the correction strength and the set strength shown in FIG. 13 is not limited to this. For example, instead of clipping, a formula for asymptotically approximating 1102 to a correction strength of 100% at a set strength of 2.0 may be used. Further, there is no upper limit to the set strength, and a set strength of 2.0 or more may be set.

Subsequently, in step S217 in FIG. 8, the sharpening unit 103c performs weighted addition of the captured image and the sharpened blurred image (model output) based on the weight map to generate the intensity adjusted image 205. FIG. Note that step S216 can be omitted, and all intensity adjustment images that can be set with a slider may be generated in advance, and the images may be displayed according to the intensity specified by the user.

In this embodiment, the sharpening section 103c functions as a first generator, a second generator, and an adjuster. The first generating means inputs the captured image to the machine learning model and generates information regarding correction. The second generating means generates an intensity-adjusted image based on the captured image, information on correction, and the weight map. The adjuster adjusts the correction strength of the strength-adjusted image by changing the weight map based on the set strength. Further, in the present embodiment, the correction strength is a first correction strength in the first region of the intensity-adjusted image based on information about the optical performance of the optical system used to capture the captured image and information about the saturation region; and a second correction intensity in a second region of the intensity adjusted image. A ratio of the second correction strength to the first correction strength varies based on the set strength. Preferably, when the set intensity is a first set intensity, the ratio is the first ratio, and when the set intensity is a second set intensity stronger than the first set intensity, the ratio is the first ratio is a second ratio that is lower than

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of controlling the sharpness of blur and adverse effects in a regression task using machine learning for a blurred image. .

Next, an image processing system according to Embodiment 2 of the present invention will be described. In this embodiment, the intensity of the non-saturated area and the brightness saturated area are adjusted with separate sliders. FIG. 14 is a block diagram of the image processing system 300 in this embodiment. FIG. 15 is an external view of the image processing system 300. As shown in FIG. The image processing system 300 has a learning device 301 , an imaging device 302 and an image processing device 303 . The learning device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are connected by a wired or wireless network.

The imaging device 302 has an optical system 321 , an imaging element 322 , a storage section 323 , a communication section 324 and a display section 325 . The captured image is transmitted to the image processing device 303 via the communication unit 324 . The image processing apparatus 303 receives the captured image via the communication unit 332 and sharpens the blur using the machine learning model configuration and weight information stored in the storage unit 331 . The configuration of the machine learning model and the weight information are learned by the learning device 301 , are obtained from the learning device 301 in advance, and are stored in the storage unit 331 . Furthermore, the image processing device 303 has a function of adjusting the intensity of blur sharpening. The sharpened blur image (model output) in which the blur of the captured image is sharpened and the intensity-adjusted image in which the intensity is adjusted are transmitted to the imaging device 302 , stored in the storage unit 323 , and displayed on the display unit 325 .

Learning data generation and weight learning (learning phase) performed by the learning device 301, and blur sharpening (estimation phase) of the captured image using the learned machine learning model performed by the image processing device 303 are performed. Since it is the same as Example 1, it is omitted.

Next, with reference to FIG. 16, synthesis of a captured image and a model output, which is executed by the image processing device 303, will be described. FIG. 16 is a flowchart of sharpening intensity adjustment. Each step in FIG. 16 is mainly executed by each unit of the image processing apparatus 303 .

First, in step S311, the sharpening unit 334 generates a weight map. Note that the method of generating the weight map is the same as that of the first embodiment, so the description thereof is omitted. Subsequently, in step S312, the acquisition unit 333 acquires the sharpening strength specified by the user. An example of intensity adjustment by the user will be described with reference to FIG. FIG. 17 is an explanatory diagram of the user interface. The user can adjust the correction strength by moving the slider. In Example 2, the user can individually adjust the correction strengths of the luminance saturated region and the non-saturated region.

Subsequently, in step S313, the sharpening unit 334 changes the weight map according to the acquired sharpening strength. Subsequently, in step S314, the sharpening unit 334 performs weighted addition of the captured image and the blurred sharpened image (model output) based on the weight map to generate an intensity adjusted image.

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of controlling the sharpness of blur and adverse effects in a regression task using machine learning for a blurred image. .

Next, an image processing system in Example 3 of the present invention will be described. FIG. 18 is a block diagram of an image processing system 400 in this embodiment. FIG. 19 is an external view of the image processing system 400. As shown in FIG. The image processing system 400 has a learning device 401 , a lens device 402 , an imaging device 403 , a control device (first device) 404 , an image estimation device (second device) 405 , and networks 406 and 407 .

Each of the learning device 401 and the image estimation device 405 is, for example, a server. A control device 404 is a device operated by a user such as a personal computer or a mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and is a machine learning model for sharpening blur from an image captured using the lens device 402 and the imaging device 403. learn the weights of Note that the learning method, that is, the generation of learning data and the learning of weights (learning phase) performed by the learning device 401 are the same as those in the first embodiment, and therefore will be omitted.

The imaging device 403 has an imaging device 403a, and the imaging device 403a photoelectrically converts an optical image formed by the lens device 402 to obtain a captured image. The lens device 402 and the imaging device 403 are detachable and can be combined with each other. The control device 404 includes a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d. Control according to the operation. Alternatively, the captured image captured by the imaging device 403 may be stored in the storage unit 404c in advance, and the captured image may be read.

The image estimation device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d, and is configured to communicate with the control device 404. FIG. The image estimating device 405 executes blur sharpening processing of the captured image in response to a request from the control device 404 connected via the network 406 . The image estimating device 405 acquires learned weight information from the learning device 401 connected via the network 406 at the time of estimating blur sharpening or in advance, and uses it for estimating blur sharpening of the captured image. The estimated image after estimating blur sharpening is transmitted to the control device 404 again after sharpening intensity adjustment, stored in the storage unit 404c, and displayed on the display unit 404b.

Next, with reference to FIG. 20, blur sharpening of the captured image executed by the control device 404 and the image estimation device 405 will be described. FIG. 20 is a flowchart of model output and sharpening intensity adjustment. Each step in FIG. 20 is mainly executed by each part of the control device 404 or the image estimation device 405 .

First, in step S401, the acquisition unit 404d of the control device 404 acquires the captured image and the sharpening intensity specified by the user. Subsequently, in step S<b>402 , the communication unit (transmitting means) 404 a transmits to the image estimating device 405 a request regarding execution of the captured image and blur sharpening estimation processing.

Subsequently, in step S403, the communication unit (receiving means) 405a of the image estimation device 405 receives and acquires the captured image and the processing request transmitted from the control device 404. FIG. Subsequently, in step S404, the acquisition unit 405b acquires the learned weight information corresponding to the captured image from the storage unit 405c. The weight information is read in advance from the storage unit 401a and stored in the storage unit 405c. Subsequently, in step S405, the sharpening unit 405d uses a machine learning model to generate a blurred sharpened image (model output) in which the blur of the captured image is sharpened from the captured image. The machine learning model has the configuration shown in FIG. 1, as during learning. As in the case of learning, a luminance saturation map representing a luminance-saturated region of a captured image is generated and input, and a saturation influence map and model output are generated.

Subsequently, in step S406, the sharpening unit 405d generates a weight map. The weight map generation method is the same as in the first embodiment. Adjust the default weight map for the user-specified sharpening strength. Note that a pre-adjusted weight map may be held within a range in which the intensity can be adjusted. Subsequently, in step S407, the sharpening unit 405d performs weighted addition of the captured image and the blurred sharpened image (model output) based on the weight map. Subsequently, in step S408, the communication unit 405a transmits the composite image to the control device 404. FIG. Subsequently, in step S<b>409 , the communication unit 404 a of the control device 404 acquires the estimated image transmitted from the image estimation device 405 .

With the above configuration, according to this embodiment, it is possible to provide an image processing system capable of controlling the sharpness of blur and adverse effects in a regression task using machine learning for a blurred image. .

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, an image processing method, an image processing apparatus, and an image processing capable of appropriately adjusting correction strength according to user settings in a regression task using machine learning for a blurred image. A program and a storage medium can be provided.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

103 Image processing device 103c sharpening unit (first generating means, second generating means, adjusting means)

Claims (14)

inputting the captured image into a machine learning model to generate information about the correction;
generating an intensity-adjusted image based on the captured image, the information about the correction, and a weight map;
adjusting the correction strength of the strength-adjusted image by changing the weight map based on a set strength;
The correction strength is a first correction strength in a first region of the strength-adjusted image and the strength adjustment based on information about the optical performance of an optical system used to capture the captured image and information about the saturation region. a second correction intensity in a second region of the image;
The image processing method, wherein a ratio of the second correction strength to the first correction strength is changed based on the set strength. when the set intensity is a first set intensity, the ratio is a first ratio;
2. The method according to claim 1, wherein said ratio is a second ratio lower than said first ratio when said setting intensity is a second setting intensity stronger than said first setting intensity. Image processing method. 3. The image processing method according to claim 1, wherein the set intensity is an intensity set by a user. 4. The image processing method according to any one of claims 1 to 3, wherein the correction strength is a weight of information regarding the correction that is weighted and added to the captured image based on the weight map. The first region is a non-saturated region,
5. The image processing method according to claim 1, wherein the second area is a saturated area. 6. The information about the saturation region is a saturation effect map representing the magnitude and range of signal values spread by the subject in the saturation region of the captured image due to blurring during imaging. 1. The image processing method according to item 1. 7. The image processing method of claim 6, wherein the saturation influence map is generated with the machine learning model. The information about the optical performance is based on at least one of a zoom position, an aperture diameter, or a subject distance at the time of capturing the captured image, and the magnitude and range of the signal value of the point spread function for each image height of the optical system. 8. The image processing method according to any one of claims 1 to 7, wherein the image processing method is calculated by 9. The image processing method according to any one of claims 1 to 8, wherein the information about the correction is an image obtained by correcting the captured image or a correction component. 10. The image processing method according to any one of claims 1 to 9, wherein the correction is to increase the resolution of the captured image or to transform the shape of defocus blur of the captured image. 11. The image processing method according to any one of claims 1 to 10, wherein the upper limit value of the correction strength is determined based on information regarding the optical performance. a first generating means for inputting a captured image into a machine learning model and generating information about correction;
a second generation means for generating an intensity-adjusted image based on the captured image, the information about the correction, and a weight map;
adjusting means for adjusting the correction strength of the strength-adjusted image by changing the weight map based on the set strength;
The correction strength is a first correction strength in a first region of the strength-adjusted image and the strength adjustment based on information about the optical performance of an optical system used to capture the captured image and information about the saturation region. a second correction intensity in a second region of the image;
The image processing apparatus, wherein a ratio of the second correction strength to the first correction strength is changed based on the set strength. An image processing system having a first device and a second device communicable with each other,
The first device has transmission means for transmitting a request for executing processing on the captured image to the second device,
The second device is
receiving means for receiving the request;
a first generation means for inputting the captured image into a machine learning model and generating information on correction;
a second generation means for generating an intensity-adjusted image based on the captured image, the information about the correction, and a weight map;
adjusting means for adjusting the correction strength of the strength-adjusted image by changing the weight map based on the set strength;
The correction strength is a first correction strength in a first region of the strength-adjusted image and the strength adjustment based on information about the optical performance of an optical system used to capture the captured image and information about the saturation region. a second correction intensity in a second region of the image;
An image processing system, wherein a ratio of the second correction strength to the first correction strength varies based on the set strength. A program for causing a computer to execute the image processing method according to any one of claims 1 to 11.

Images