JP2021089654A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To execute correction processing of bringing the feature of an image to be corrected closer to the feature of a target image in terms of an arbitrary item related to the feature of the image.SOLUTION: For an image to be corrected subjected to correction processing and a target image referred to in the correction processing, an evaluation value acquisition unit acquires evaluation values for a plurality of evaluation items. A data-for-learning generation unit generates data for learning based on the acquired evaluation values for the image to be corrected and the target image. A learning model generation unit generates learning models related to the correction processing for the evaluation items based on the generated data for learning. An image correction unit executes the correction processing of using a learning model selected from the learning models for evaluation items to correct the image to be corrected so as to bring the image to be corrected closer to the impression of the target image in terms of an arbitrary evaluation item.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

In recent years, with the spread of devices capable of image processing such as smartphones, an increasing number of users perform correction processing after capturing an image to process and edit the captured image. When performing correction processing on an image, a filter that can be processed to a preset texture using an image processing application installed on the device, and a slide bar related to brightness, saturation, contrast, etc. are used. Often make adjustments. Patent Document 1 describes a technique for reflecting the characteristics of the color space of another image (target image) with respect to the image to be corrected.

Japanese Unexamined Patent Publication No. 2015-226164

The features of the image are not limited to the hue, but there are various other items such as contrast and noise, and the user does not always want to bring the image to be corrected closer to the color space of the target image. Absent. However, with the technique described in Patent Document 1, it is not possible to perform correction processing for bringing the features of the image to be corrected closer to the features of the target image for various items related to the features of the image. The present invention has been made in view of such circumstances, and an object of the present invention is to enable a correction process for bringing the characteristics of an image to be corrected closer to the characteristics of a target image for any item related to the characteristics of the image. And.

The image processing apparatus according to the present invention includes an evaluation value acquisition means for acquiring evaluation values for each of a plurality of evaluation items for each of the correction target image to be corrected and the target image referred to in the correction processing. A data generation means for generating learning data based on the acquired evaluation values of the correction target image and the target image, and a learning model related to the correction processing based on the generated learning data are generated for each evaluation item. It is characterized by having a model generation means and an image correction means for correcting the correction target image by using the learning model selected from the learning models for each generated evaluation item.

According to the present invention, it is possible to execute a correction process for bringing the feature of the image to be corrected closer to the feature of the target image for any item related to the feature of the image.

It is a figure which shows an example of the hardware composition of the image processing apparatus 100. It is a figure which shows an example of the appearance of the image processing apparatus 100. It is a figure which shows an example of the functional structure of the image processing apparatus 100. It is a flowchart which shows the processing example of the image processing apparatus 100 in 1st Embodiment. It is a figure which shows an example of a graphical user interface. It is a flowchart which shows the example of the evaluation value acquisition processing in 1st Embodiment. It is a flowchart which shows the example of the process of calculating the evaluation value of noise. It is a flowchart which shows the example of the process of calculating the evaluation value of a histogram. It is a flowchart which shows the example of the generation process of learning data in 1st Embodiment. It is a figure explaining the generation process of learning data. It is a flowchart which shows the example of the image correction processing using the learning model in 1st Embodiment. It is a figure explaining the learning process in the learning model generation unit 304. It is a flowchart which shows the example of the generation process of learning data in 2nd Embodiment. It is a flowchart which shows the processing example of the image processing apparatus 100 in 3rd Embodiment. It is a figure which shows an example of a graphical user interface. It is a flowchart which shows the example of the generation process of the learning model in 3rd Embodiment. It is a flowchart which shows the example of the image correction processing using the learning model in 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are examples, and do not necessarily limit the present invention. Moreover, not all of the configurations described in the present embodiment are indispensable for the means for solving the present invention.

[First Embodiment]
In the present embodiment, an example will be described in which a desired image processing is executed on an image to be corrected based on another image captured in a scene different from the correction target.

<Hardware configuration of image processing device 100>
FIG. 1A is a diagram showing an example of the hardware configuration of the image processing device 100 according to the present embodiment. The image processing device 100 is realized by, for example, a smartphone, a tablet terminal, a personal computer (including a tablet PC), or the like. The image processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an input / output interface (I / F) 104, a touch panel display 105, an imaging unit 106, and a communication I / F 107.

The CPU (central processing unit) 101 controls each configuration of the image processing device 100 connected to the system bus 108. The ROM (read only memory) 102 is a rewritable non-volatile memory, and stores a program, image data, a learning model, and the like of the image processing device 100. The RAM (random access memory) 103 functions as a main memory, a work memory, or the like of the CPU 101. The CPU 101 uses the RAM 103 as a work memory to execute an OS (operating system) and various programs stored in the ROM 102, the external storage device 110, and the like to control the image processing device 100 and perform various processes. The process shown in the flowchart described later is realized by the CPU 101 executing a program read from the ROM 102, the external storage device 110, or the like and expanded in the RAM 103.

An external storage device 110 is connected to the input / output I / F 104 via the serial bus 109. The external storage device 110 is, for example, an HDD (hard disk drive) and stores programs, image data, and the like. The touch panel display 105 is an input / output unit in which a display for displaying an image or the like and a touch panel for detecting the position of touch by an indicator such as a finger are integrated. In addition, in order to correspond to the touch panel display 105, the display unit for displaying an image or the like and the input unit for receiving input from the user may be provided separately. The imaging unit 106 is an imaging unit that acquires an image to be imaged. The communication I / F 107 performs two-way communication by wire or wireless with another device or the like. The communication I / F 107 in this embodiment can send and receive data to and from a communication partner via a wireless LAN (local area network). Further, the communication I / F 107 can perform indirect communication with other communication devices in addition to direct communication via a relay device.

FIG. 2 shows an example of the appearance of the image processing device 100 according to the present embodiment. FIG. 2A is a diagram showing a surface (hereinafter, referred to as a display surface) of the image processing device 100 on the side having the touch panel display 105. FIG. 2B is a diagram showing a surface (hereinafter, referred to as a back surface) opposite to the display surface of the image processing device 100. The image processing device 100 in the present embodiment has an in-camera 201 provided on the display surface of the image processing device 100 and an out-camera 202 provided on the back surface of the image processing device 100 as the image pickup unit 106. The in-camera 201 is installed at a position and a direction in which the face of the user who browses the display (display screen) can be imaged.

<Functional configuration of image processing device 100>
FIG. 3 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the present embodiment. The image processing device 100 realizes the functions of the functional units shown in FIG. 3 by executing the program stored in the ROM 102 or the like by the CPU 101 using the RAM 103 as the work memory. The image processing apparatus 100 may be configured so that a part or all of the functional units described below are realized by one or a plurality of processing circuits other than the CPU 101.

The image processing device 100 includes an image acquisition unit 301, an evaluation value acquisition unit 302, a learning data generation unit 303, a learning model generation unit 304, an image correction unit 305, and a display control unit 306. The image acquisition unit 301 acquires the correction target image and the target image. The correction target image is an image to be corrected, and the target image is an image to be referred to as a target when the correction target image is corrected. The correction target image and the target image are, for example, images captured in different scenes.

The evaluation value acquisition unit 302 analyzes each of the correction target image and the target image acquired by the image acquisition unit 301, and acquires the evaluation value of each image for each of a plurality of different evaluation items related to the characteristics of the image. As an example, the evaluation value acquisition unit 302 acquires evaluation values for each item of color, noise, and histogram as evaluation values representing the features of the image. The evaluation value acquired by the evaluation value acquisition unit 302 is not limited to this, and evaluation values of other items may be acquired, for example, evaluation values such as gloss and contrast may be acquired. You may do it.

The learning data generation unit 303 generates learning data to be used for generating a learning model related to the correction process based on the evaluation value acquired by the evaluation value acquisition unit 302. The learning model generation unit 304 performs learning processing using the learning data generated by the learning data generation unit 303, and generates a learning model related to the correction processing. The image correction unit 305 corrects the correction target image acquired by the image acquisition unit 301 by using the learning model generated by the learning model generation unit 304. The display control unit 306 controls the display of a user interface for having the user input information necessary for processing, an image related to correction processing, and the like.

<Processing executed by the image processing device 100>
FIG. 4 is a flowchart showing a processing example of the image processing apparatus 100 according to the present embodiment. When the CPU 101 reads the program from the ROM 102 or the like and executes it, each process of the flowchart shown in FIG. 4 is realized. Hereinafter, each step (step) is represented by adding "S" to the beginning of the reference numeral.

In S401, the display control unit 306 causes the touch panel display 105 to display a graphical user interface (GUI: Graphical User Interface) for having the user input information necessary for processing. Further, the display control unit 306 receives an instruction from the user via the GUI. FIG. 5 shows an example of the GUI displayed on the touch panel display 105 in the image processing device 100 according to the present embodiment.

First, the display control unit 306 causes the touch panel display 105 to display the GUI shown in FIG. 5A for accepting the selection of the image to be corrected (correction target image) in the correction process. In the correction target image selection area 502, a plurality of images that are candidates for the correction target image are displayed. When a user touches any of the images displayed in the selection area 502, the touch panel display 105 detects the image specified by the touch operation and causes the display control unit 306 to perform a touch operation. Sends information indicating the specified image. Further, the folder designation button 503 is a button for accepting the selection and designation of the folder in which the image is saved. When the folder designation button 503 is pressed by the user and it is detected, the display control unit 306 displays the folder candidates in which the images are stored so that they can be selected. Further, when the display control unit 306 receives the designation of the folder by the user's touch operation or the like, the display control unit 306 displays the image stored in the designated folder in the selection area 502. In this way, the display control unit 306 accepts the user's designation of the correction target image while displaying the image that is a candidate for the correction target image in the selection area 502 in response to the instruction from the user. The move button 501 is a button for receiving an instruction from the user to proceed to the next menu (menu for selecting a target image). When the specification of the image to be corrected is completed, the user can press the move button 501.

When the move button 501 is pressed by the user and is detected, the display control unit 306 then shows in FIG. 5 (b) for accepting the selection of the image (target image) to be referred to as the target in the correction process. Display the GUI. In the target image selection area 512, a plurality of images that are candidates for the target image are displayed. Similar to the case of the correction target image, information indicating the image designated by the user is transmitted to the display control unit 306 via the touch panel display 105, and the display control unit 306 accepts the user to specify the target image. The folder designation button 513 is a button for accepting selection and designation of a folder in which an image is saved, similar to the folder designation button 503 shown in FIG. 5 (a). The move button 511 is a button for receiving an instruction from the user to proceed to the next menu (a menu for executing the correction process for the image to be corrected). When the designation of the target image is completed, the user can press the move button 511.

When the move button 511 is pressed and detected, the display control unit 306 displays the GUI shown in FIG. 5C on the touch panel display 105. In the GUI shown in FIG. 5C, the correction target image area 522 is displayed. In the correction target image area 522, the correction target image before or after the correction processing is displayed. The correction application button 521 is a button for instructing the application of the correction process to the image to be corrected. When the correction application button 521 is pressed by the user, the image processing device 100 starts the processing after S402 shown in FIG. The restore button 523 is a button for returning the image after the correction process to the image before the correction process.

In the examples shown in FIGS. 5A and 5B, up to 9 images can be displayed in a list in the selection area 502 of the correction target image and the selection area 512 of the target image, respectively. is there. However, the number of images that can be displayed in the selection areas 502 and 512 is not limited to this, and is arbitrary. If the number of images stored in the specified folder is larger than the number of images that can be displayed in the selection areas 502 and 512, the images are displayed according to the user's slide (swipe) operation on the selection areas 502 and 512. The image to be displayed may be switched as appropriate.

Returning to FIG. 4, in S402, the image acquisition unit 301 acquires the image data of the image to be corrected. When the image acquisition unit 301 receives the information for specifying the image designated as the correction target image from the display control unit 306 according to the user's instruction, the image acquisition unit 301 specifies the location where the image data of the image designated as the correction target image is stored. , Read image data from ROM 102.

In S403, the image acquisition unit 301 acquires the image data of the target image. When the image acquisition unit 301 receives the information for specifying the image designated as the target image from the display control unit 306 according to the user's instruction, the image acquisition unit 301 identifies the place where the image data of the image designated as the target image is stored, and ROM 102. Read the image data from.

Here, it is assumed that the image data of the image to be corrected and the target image are composed of image data of each color of R (red), G (green), and B (blue). It is assumed that the image data of each color is data representing an image consisting of 8-bit pixel values for each pixel. Note that this is an example, and the image data of the correction target image and the target image is not limited to this. Further, the order in which the processing of S402 and the processing of S403 are executed is arbitrary, and it is sufficient that the image data of the correction target image and the target image are acquired before the processing of S404 is started.

In S404, the evaluation value acquisition unit 302 performs an evaluation value acquisition process using the image data of the correction target image read in S402 and the image data of the target image read in S403, and performs the correction target image and the target image. Get the evaluation value for each of. Here, the evaluation value acquisition unit 302 acquires the evaluation values for each of a plurality of different evaluation items by calculating the evaluation values of the correction target image and the target image for a plurality of the same items. In the present embodiment, as described above, the evaluation values of the color, noise, and histogram of the image are acquired as an example. The details of the evaluation value acquisition process executed by the evaluation value acquisition unit 302 in S404 will be described later.

In S405, the learning data generation unit 303 performs a learning data generation process, and generates learning data for each evaluation item based on the evaluation values of the correction target image and the target image acquired in S404. The learning data generation unit 303 generates a plurality of sets of data in which the image data before conversion and the image data after conversion are paired for each evaluation item, and stores the data as a learning data set in the ROM 102 or the like. The details of the learning data generation process executed by the learning data generation unit 303 in S405 will be described later.

In S406, the learning model generation unit 304 and the image correction unit 305 perform image correction processing using the learning model. In the image correction process using the learning model, the learning model generation unit 304 generates a learning model for each evaluation item based on the learning data generated in S405. Further, the image correction unit 305 performs correction processing using the learning model generated by the learning model generation unit 304, and corrects the correction target image acquired in S402. Details of the image correction process using the learning model executed by the learning model generation unit 304 and the image correction unit 305 in S406 will be described later.

In S407, the display control unit 306 causes the touch panel display 105 to display the image (corrected correction target image) obtained by correcting the correction target image in S406. Then, the image processing device 100 ends the processing.

<Evaluation value acquisition process (S404)>
FIG. 6 is a flowchart showing an example of the evaluation value acquisition process executed by the evaluation value acquisition unit 302 in S404 shown in FIG.

In S601, the evaluation value acquisition unit 302 calculates an evaluation value regarding the color of the image for each of the correction target image and the target image. In the present embodiment, the evaluation value acquisition unit 302 calculates the chromaticity value in each of the correction target image and the target image as the evaluation value related to the tint. Specifically, the evaluation value acquisition unit 302 first averages the pixel values of all the pixels in each image for each color. Next, the evaluation value acquisition unit 302 converts the average value of each color into an XYZ value using a predetermined conversion formula such as sRGB or AdobeRGB, and further converts the chromaticity values a (red-green) and b (yellow-) of CIECAM02. Blue) is used to convert the XYZ value to a chromaticity value. As the evaluation value related to the color tint, the evaluation value related to the tint of the light source may be calculated instead of the tint of the entire image. For example, a histogram may be created for the luminance component from the input image, and the evaluation value may be calculated based on the average value of the pixel values of 5% from the upper side of the histogram. Alternatively, as the conversion from the XYZ value to the chromaticity value, the CIE-xy chromaticity value or the chromaticity values a and b of the CIE-Lab may be used.

In S602, the evaluation value acquisition unit 302 performs the processing shown in FIG. 7 for each of the correction target image and the target image to calculate the evaluation value related to noise. FIG. 7 is a flowchart showing an example of processing for calculating the evaluation value of noise.

In S701, the evaluation value acquisition unit 302 converts the pixel value of each pixel into a brightness value L * (x, y) for each of the correction target image and the target image by using the sRGB conversion formula and the CIE-Lab conversion formula. Convert. Further, the evaluation value acquisition unit 302 calculates the spatial frequency characteristic F (u, v) by Fourier transforming the brightness value L * (x, y) by the equation (1). In the formula (1), M is the number of pixels in the horizontal direction in the image for which the evaluation value is calculated, and N is the number of pixels in the vertical direction in the image for which the evaluation value is calculated.

Next, in S702, the evaluation value acquisition unit 302 applies a desired band filter to the spatial frequency characteristic F (u, v) calculated in S701, and obtains the Wiener spectrum WS (u, v) based on the equation (2). calculate. In equation (2), BPF (u, v) is a band filter used to evaluate two-dimensional image noise. In the present embodiment, the evaluation value acquisition unit 302 calculates the noise evaluation value for two bands by using _{the high frequency band filter (BPF High} ) and the low frequency band filter (BPF _{Low), but the evaluation value acquisition unit 302 is limited to this.} Instead, three or more band filters may be used.

Next, in S703, the evaluation value acquisition unit 302 calculates the integrated value of the Wiener spectrum WS (u, v) calculated in S702 for each frequency band, and uses the integrated value as the noise evaluation value. The noise evaluation value indicates that the larger the evaluation value, the more two-dimensional image noise is contained in the image.
Next, in S704, the evaluation value acquisition unit 302 saves the evaluation value of noise calculated in S703 for each frequency band, and ends the process shown in FIG. 7.

Returning to FIG. 6, in S603, the evaluation value acquisition unit 302 performs the processing shown in FIG. 8 for each of the correction target image and the target image to calculate the evaluation value related to the histogram of the image. FIG. 8 is a flowchart showing an example of processing for calculating the evaluation value of the histogram.

In S801, the evaluation value acquisition unit 302 converts the pixel value of each pixel in each of the correction target image and the target image into a brightness value.
Next, in S802, the evaluation value acquisition unit 302 sets the skewness and kurtosis as the feature amount of the histogram of the brightness value for each of the correction target image and the target image based on the brightness value of each pixel calculated in S801. calculate. The evaluation value acquisition unit 302 calculates the skewness using the equation (3) and calculates the kurtosis using the equation (4).

Next, in S803, the evaluation value acquisition unit 302 saves the skewness and kurtosis calculated in S802 as the evaluation values of the histogram, and ends the process shown in FIG.

When the evaluation value acquisition unit 302 calculates the color, noise, and the evaluation value of the histogram as described above, the evaluation value acquisition process ends. In the example shown in FIG. 6, the evaluation values are calculated in the order of image color (S601), noise (S602), and histogram (S603), but the evaluation value is not limited to this. In the evaluation value acquisition process, it suffices if the color, noise, and histogram evaluation values of the image can be acquired, and the order in which the evaluation values are calculated is arbitrary. That is, the order in which the processing of S601, the processing of S602, and the processing of S603 shown in FIG. 6 are executed is in no particular order.

<Learning data generation process (S405)>
FIG. 9 is a flowchart showing an example of the learning data generation process executed by the learning data generation unit 303 in S405 shown in FIG. Further, FIG. 10 is a diagram schematically showing a process executed by the learning data generation unit 303.

As shown in FIG. 10, in the learning data generation process, the learning data generation unit 303 first corrects each of the evaluation items of color, noise, and histogram from the image database (DB) 1001 for learning. Each image that is close to the evaluation value of the image is extracted. The learning data generation unit 303 extracts a plurality of images 1011 having evaluation values similar to each other in terms of color from the image DB 1001 for learning based on the evaluation values of the color of the image to be corrected. Further, the learning data generation unit 303 extracts a plurality of images 1021 having evaluation values approximated with respect to noise from the learning image DB 1001 based on the evaluation value of the noise of the image to be corrected. Further, the learning data generation unit 303 extracts a plurality of images 1031 having evaluation values approximated with respect to the histogram from the learning image DB 1001 based on the evaluation values of the histogram of the image to be corrected.

The learning data generation unit 303 extracts an image from the image database (DB) 1001 and then performs a conversion process on the extracted image so as to approximate the evaluation value of the target image. Then, the learning data generation unit 303 generates a plurality of sets of learning data with a pair of the image before conversion and the image after conversion as one set, and saves it as a learning data set. The learning data generation unit 303 performs conversion processing on each of the plurality of images 1011 based on the evaluation value of the color of the target image, and sets a plurality of sets of the original image 1011 and the converted image 1012 as a pair. Generate data. Further, the learning data generation unit 303 converts each of the plurality of images 1021 based on the evaluation value of the noise of the target image, and sets a plurality of pairs of the original image 1021 and the converted image 1022. Generate data for. Further, the learning data generation unit 303 performs conversion processing on each of the plurality of images 1031 based on the evaluation value of the histogram of the target image, and a plurality of sets of the original image 1031 and the image 1032 after the conversion processing as a pair. Generate data for. By performing such learning data generation processing, it is possible to improve the conversion accuracy when the image to be corrected is brought closer to the impression of the target image. The details of the process will be described below.

In S901 of FIG. 9, the learning data generation unit 303 acquires the evaluation values of the correction target image and the target image for each evaluation item acquired by the evaluation value acquisition unit 302 in S404 shown in FIG.
In S902, the learning data generation unit 303 acquires an image data group prepared in advance as image data for learning. In the present embodiment, as an example, 10,000 images of various categories such as people, landscape images, sports, hobbies, and foods are prepared and held in advance, but images of a specific category are held according to the user's taste. However, the number of images to be prepared is not limited to this. The conversion accuracy can be improved as the number of images matches the category of the image to be corrected. The order in which the processing of S901 and the processing of S902 are executed is in no particular order, and may be executed before the next processing of S903 is started.

In S903, the learning data generation unit 303 extracts image data close to the evaluation value of the image to be corrected from the image data acquired in S902. For the image data acquired in S902, the evaluation value is calculated in advance by the same method as the evaluation value acquisition process described above. In the present embodiment, the learning data generation unit 303 is based on the evaluation value of the image data calculated in advance, from the one closest to the evaluation value of the correction target image (the one having a smaller difference from the evaluation value of the correction target image). (From) 100 images are extracted in order. This image data is extracted for each evaluation item of color, noise, and histogram. In the present embodiment, 100 images are extracted for each evaluation item, but the number of images to be extracted is arbitrary. Further, although a predetermined number of images are extracted from the one closer to the evaluation value of the correction target image, the difference from the evaluation value of the correction target image is within a predetermined range without limiting the number of images. The image may be extracted.

Next, in S904, the learning data generation unit 303 converts the image data extracted in S903 with reference to the evaluation value of the target image. For the conversion, for the color and histogram, a 3 × 10 transformation matrix in which the color conversion coefficient is described is set as a correction parameter, and the difference between the evaluation value of the converted image and the evaluation value of the target image is minimized. The parameters are determined so as to perform image conversion. The correction parameters may be approximated by 3DLUT or a polynomial. Regarding noise, if the noise of the target image is less than that of the image to be corrected, a low-pass filter is set as a correction parameter, and if the noise of the target image is large, the standard deviation σ is centered on the brightness of each pixel. Performs the process of adding a normally distributed random number with. The learning data generation unit 303 determines the filter size and coefficient, or the standard deviation σ representing the magnitude of noise addition as a correction parameter so that the difference between the evaluation value of the corrected image and the evaluation value of the target image is minimized. Then, image conversion is performed using the determined correction parameters.

Next, in S905, the learning data generation unit 303 uses image data of 100 images extracted in S903 for each evaluation item as a pair of images extracted in S903 and images converted in S904 as a learning data set. save.

In S906, the learning data generation unit 303 determines whether or not learning data has been generated for all the evaluation items. When it is determined that the learning data has been generated for all the evaluation items (YES in S906), the learning data generation unit 303 ends the learning data generation process. On the other hand, when the learning data generation unit 303 determines that the generation of the learning data for all the evaluation items has not been completed (NO in S906), the learning data generation unit 303 returns to the processing of S903 and generates the learning data. The processing after S903 is performed for the evaluation items that do not exist.

<Image correction processing using a learning model (S406)>
FIG. 11 is a flowchart showing an example of image correction processing using the learning model executed by the learning model generation unit 304 and the image correction unit 305 in S406 shown in FIG.

In S1101, the learning model generation unit 304 acquires the learning data generated by the learning data generation process in S405.

In S1102, the learning model generation unit 304 performs the learning process using the learning data acquired in S1101. In the present embodiment, the learning process uses a convolution neural network (CNN), and the outline of the process is shown in FIG. 12 (a). In the present embodiment, as an example, the CNN model uses a VGG16 model consisting of 16 layers having 13 convolutional layers and 3 fully connected layers. As illustrated in FIG. 12B, the convolutional layer is filtered (3 × 3) and the non-linear processing using the activation function is performed on the input image, and the pooling layer is the reduction processing of the feature map processed by the convolutional layer. I do. In this embodiment, the ReLU function is used as the activation function, but other non-linear processing such as a sigmoid function may be used. The loss function also uses the sum of squares error and uses gradient descent to optimize the weighting parameters so that the loss function is minimized. The learning model generation unit 304 sets the image before conversion and the image after conversion in the learning data generated in S405 in the input image and the teacher image, respectively, and performs the learning process to generate the learning model. In this embodiment, CNN is used as the learning process, but other learning processes such as fix2pix, which is one method of GAN (generative adversarial network) that generates images from the relationship between images, can also be used. Good.

In S1103, the learning model generation unit 304 determines whether or not the learning model has been generated for all the evaluation items. When the learning model generation unit 304 determines that the learning model has been generated for all the evaluation items (YES in S1103), the process proceeds to S1104. On the other hand, when the learning model generation unit 304 determines that the generation of the learning model for all the evaluation items is not completed (NO in S1103), the process returns to the process of S1102 and the evaluation items for which the learning model is not generated are learned. Perform processing.

In S1104, the image correction unit 305 acquires the evaluation values of the correction target image and the target image for each evaluation item acquired by the evaluation value acquisition unit 302 in S404 shown in FIG.

In S1105, the image correction unit 305 calculates the difference value of the evaluation value for each evaluation item based on the evaluation value of the correction target image and the target image for each evaluation item acquired in S1104. Further, the image correction unit 305 calculates the difference value of the evaluation value from the learning models generated by the learning model generation unit 304 as described above so as to perform the correction processing for the item having the largest difference value of the evaluation value. Select the learning model that matches the largest endpoint. Then, the image correction unit 305 performs correction processing using the selected learning model, and corrects the image to be corrected. In the present embodiment, the correction target image is corrected by selecting one learning model corresponding to the evaluation item having the largest difference value between the evaluation values of the correction target image and the target image. Depending on the situation, a plurality of learning models may be applied to correct the image to be corrected. For example, a threshold value is set in advance for each evaluation item, and a learning model is applied to each evaluation item in which the difference value between the evaluation values of the correction target image and the target image exceeds the set threshold value to correct the correction target image. May be good.

In S1106, the image correction unit 305 saves the image corrected in S1105 and ends the image correction process using the learning model.

As described above, according to the first embodiment, the correction target image and the target image are analyzed to obtain evaluation values for a plurality of different evaluation items, and a learning model according to the evaluation values is obtained for each evaluation item. Generate and correct the image to be corrected using the generated learning model. By generating the learning data used to generate the learning model based on the acquired evaluation values, the conversion accuracy of the learning model is improved, and the impression (texture) of the target image is brought closer to any evaluation item related to the image features. The correction process can be executed.

[Second Embodiment]
In the learning data generation process of the first embodiment, an image database for learning is prepared in advance, and image data for learning is extracted and converted for each evaluation item based on the evaluation value from the database for learning. Explained how to generate data for. In the second embodiment, a method of generating learning data based on a target image specified by the user will be described. In the second embodiment, the conversion accuracy of the learning model is improved by using the target image that is actually the target in the correction process of the image to be corrected as the learning data. The second embodiment is different from the first embodiment described above in the process of generating learning data, and other configurations and processes are the same as those of the first embodiment. Hereinafter, the learning data generation process in the second embodiment will be described.

FIG. 13 is a flowchart showing an example of the learning data generation process executed by the learning data generation unit 303 in the second embodiment.
In S1301, the learning data generation unit 303 acquires the evaluation values of the correction target image and the target image for each evaluation item in the same manner as the processing of S404 shown in FIG.
In S1302, the learning data generation unit 303 acquires the target image in the same manner as the processing of S403 shown in FIG. The order in which the processing of S1301 and the processing of S1302 are executed is in no particular order, and may be executed before the next processing of S1303 is started.

In S1303, the learning data generation unit 303 refers to the evaluation values of the correction target image and the target image, and converts the target image. For the conversion, for the color tint and the histogram, a 3 × 10 transformation matrix in which the color conversion coefficient is described is set as a correction parameter, and the evaluation value of the image after the conversion of the target image and the evaluation value of the correction target image are set. Image conversion is performed by determining the parameters so that the difference is minimized. The correction parameters may be approximated by 3DLUT or a polynomial. Regarding noise, if the noise of the target image is larger than the noise of the image to be corrected, a low-pass filter that reduces the noise is set as a correction parameter, and the filter size and coefficient are set so that the difference between the evaluation values becomes small. Determine and perform image conversion. On the other hand, when the noise of the target image is less than the noise of the image to be corrected, a process of adding a normal distribution random number with the standard deviation σ as the correction parameter centered on the brightness of each pixel is performed so that the difference in the evaluation values becomes small. The standard deviation σ is determined and noise addition processing is performed.

Next, in S1304, the learning data generation unit 303 saves the image data as a set of the target image acquired in S1302 and the image converted in S1303 as learning data to be used for learning.

In S1305, the learning data generation unit 303 determines whether or not learning data has been generated for all the evaluation items. When it is determined that the learning data has been generated for all the evaluation items (YES in S1305), the learning data generation unit 303 ends the learning data generation process. On the other hand, when the learning data generation unit 303 determines that the generation of the learning data for all the evaluation items has not been completed (NO in S1305), the learning data generation unit 303 returns to the processing of S1303 and generates the learning data. The processing after S1303 is performed for the evaluation items that do not exist.

According to the second embodiment, by generating the learning data based on the target image, the conversion accuracy of the learning model is improved, and the impression (texture) of the target image is obtained for any evaluation item related to the characteristics of the image. It is possible to execute the correction process to bring it closer.

[Third Embodiment]
In the first embodiment, evaluation values for a plurality of different evaluation items are acquired, a learning model corresponding to the evaluation values of the target image is generated, and the correction target image is corrected. In the third embodiment, a method of presenting a correction result image candidate for each evaluation item to the user and simply determining the correction process for the correction target image while referring to the correction result will be described. In addition, a method of improving the conversion accuracy of the learning model by performing additional learning using the past converted image with reference to the past conversion history in the image data when presenting the correction result image will also be described. Since the hardware configuration and functional configuration of the image processing device in the third embodiment are the same as those in the image processing device in the first embodiment, the description thereof will be omitted.

<Processing executed by the image processing device 100>
FIG. 14 is a flowchart showing a processing example of the image processing apparatus 100 according to the third embodiment. When the CPU 101 reads the program from the ROM 102 or the like and executes it, each process of the flowchart shown in FIG. 14 is realized.

In S1401, the display control unit 306 causes the GUI to be displayed on the touch panel display 105. FIG. 15 shows an example of the GUI displayed on the touch panel display 105 in the present embodiment. In FIG. 15, the same components as those shown in FIG. 5 are designated by the same reference numerals. The GUI shown in FIGS. 15 (a) and 15 (b) is the same as the GUI shown in FIGS. 5 (a) and 5 (b) in the first embodiment, and thus the description thereof will be omitted.

In the GUI shown in FIG. 15 (b), when the move button 511 is pressed and detected, the display control unit 306 causes the GUI shown in FIG. 15 (c) to be displayed on the touch panel display 105. In the GUI shown in FIG. 15C, the correction target image area 522 is displayed. The correction application button 521 is a button for instructing the application of the correction process using the learning model to the image to be corrected. The restore button 523 is a button for returning the corrected image to the image before the correction process. The history reference button 1501 is a button for referring to the history of the past corrected image and the learning model used for the correction process. The model selection button 1502 is a button for instructing the presentation of a candidate for a corrected image. In the selection image area 1503, candidates for correction images for selecting a learning model to be used for correction processing are displayed.

Returning to FIG. 14, in S1402, the image acquisition unit 301 acquires the image data of the image to be corrected in the same manner as the process of S402 in the first embodiment.
In S1403, the image acquisition unit 301 acquires the image data of the target image to be corrected when the correction target image is corrected, in the same manner as the process of S403 in the first embodiment. The order in which the processing of S1402 and the processing of S1403 are executed is arbitrary, and it is sufficient that the image data of the correction target image and the target image are acquired before the processing of S1404 is started.

In S1404, the evaluation value acquisition unit 302 obtains the image data of the correction target image read out in S1402 and the image data of the target image read out in S1403 in the same manner as the processing of S404 in the first embodiment. Use to perform evaluation value acquisition processing. The evaluation value acquisition unit 302 acquires evaluation values for each of a plurality of different evaluation items for each of the correction target image and the target image.

In S1405, the evaluation value acquisition unit 302 detects the pressing of the history reference button 1501 by the user, identifies the location where the image data indicating the conversion history and the learning model indicating the conversion history specified by the user's instruction are stored, and the image from the ROM 102. Read the data and learning model. In the conversion history, the trained model used by the user to bring the image data closer to the target image in the past and the corrected image data are recorded in the ROM 102 in association with the creation date and time.

In S1406, the learning model generation unit 304 performs a learning model generation process, and generates a learning model for each evaluation item based on the past conversion history acquired in S1405 and the evaluation values of the correction target image and the target image. .. The details of the learning model generation process executed by the learning model generation unit 304 in S1406 will be described later.

In S1407, the image correction unit 305 and the display control unit 306 perform image correction processing using the learning model. In the image correction process using the learning model, the image correction unit 305 acquires the learning model generated in S1406 and performs the correction process, and corrects the image to be corrected acquired in S1402. Further, the display control unit 306 displays a GUI for having the user input information necessary for processing and an image obtained by correction processing by the image correction unit 305 (corrected correction target image) on a touch panel display. It is displayed on 105. Details of the image correction process using the learning model executed by the image correction unit 305 and the display control unit 306 in S1407 will be described later.

<Learning model generation process (S1406)>
FIG. 16 is a flowchart showing an example of a learning model generation process executed by the learning model generation unit 304 in S1406 shown in FIG.

In S1601, the learning model generation unit 304 acquires a learned model suitable for each of a plurality of evaluation items. It is assumed that the trained model is stored in the ROM 102 in advance for each evaluation item. For example, for the evaluation item of color, a trained model that performs correction suitable for color is stored, and for the evaluation item of noise, a trained model that corrects for noise is stored.

In S1602, the learning model generation unit 304 refers to the past conversion history acquired in S1405 and acquires the image data converted in the past in response to an instruction from the user.
In S1603, the learning model generation unit 304 performs additional learning of the trained model for each evaluation item by using the trained model acquired in S1601 and the image data acquired in S1602.

In S1604, the learning model generation unit 304 determines whether or not additional learning of the trained model using the past converted image has been performed for all the evaluation items. When the learning model generation unit 304 determines that the additional learning of the trained model has been performed for all the evaluation items (YES in S1604), the learning model generation unit 304 proceeds to the process of S1605. On the other hand, when the learning model generation unit 304 determines that the additional learning of the trained model for all the evaluation items has not been completed (NO in S1604), the learning model generation unit 304 returns to the process of S1601 and performs the additional learning of the trained model. Perform the above-mentioned processing for the evaluation items that have not been implemented.

In S1605, the learning model generation unit 304 saves the learning model additionally learned for each evaluation item as described above, and ends the learning model generation process.

<Image correction processing using a learning model (S1407)>
FIG. 17 is a flowchart showing an example of image correction processing using the learning model executed by the image correction unit 305 and the display control unit 306 in S1407 shown in FIG.
In S1701, the display control unit 306 displays a GUI for asking the user to input information necessary for processing.

In S1702, the display control unit 306 detects the user pressing the model selection button 1502. When the display control unit 306 detects that the model selection button 1502 is pressed, the display control unit 306 causes the GUI selection image area 1503 shown in FIG. 15C to display a list of corrected images using the learning model for each evaluation item. For example, the display control unit 306 displays the correction images related to the three evaluation items in the order of the evaluation items in which the difference between the evaluation value of the target image and the evaluation value of the correction target image is large among the plurality of evaluation items. In addition, a plurality of corrected images having different learning models used for correction may be displayed in a list for all the evaluation items so that the degree of correction can be selected for all the evaluation items. Further, in the present embodiment, three images are arranged for each evaluation item in the selection image area 1503. The image corrected by applying the learning model as described in the first embodiment is applied to the left side, and the trained model used for the past conversion acquired by referring to the past conversion history is applied to the correction processing in the center. The corrected image created by the processing of S1406 is placed on the right side of the image. In the present embodiment, three changed images are arranged for each evaluation item, but the number of arranged images is not limited to this. An arbitrary number of images may be arranged by enlarging / reducing according to the number of trained models to be prepared and the display area of the display, or the GUI may be provided with a means for designating the number of images to be arranged.

In S1703, the image processing unit 305 and the display control unit 306 apply the correction process to the image to be corrected using the learning model used for the image selected by the user, and the image to which the correction process is applied is displayed on the touch panel display 105. To display. Specifically, the image correction unit 305 performs correction processing using the learning model used for the selected image in response to a selection instruction from the user of the image displayed in the selection image area 1503. Corrects the image to be corrected. The display control unit 306 displays the image obtained by correcting the correction target image in the correction target image area 522.

The image processing unit 305 and the display control unit 306 perform the processing of S1703 every time the image displayed in the selection image area 1503 is selected by the user until the pressing of the apply button 521 from the user is detected in S1704. Repeat. When the image processing unit 305 and the display control unit 306 detect that the application button 521 is pressed by the user in S1704, the image processing unit 305 and the display control unit 306 consider that the selection of the correction processing for the correction target image is completed, and proceed to the processing of S1705.

In S1705, the image processing unit 305 and the display control unit 306 save the corrected image displayed in the correction target image area 522, and end the image correction processing using the learning model.

According to the third embodiment, it is possible to execute a correction process that brings the impression (texture) of the target image closer to any evaluation item related to the features of the image. In addition, by presenting an image corrected for each of a plurality of different evaluation items based on the past conversion history, it is possible to infer the tendency of the user to correct the image to be corrected to what kind of image the user wants to correct. A corrected image can be presented according to the preference.

[Modification example]
In the above-described embodiment, the hardware configuration of the image processing apparatus 100 is not limited to the configuration shown in FIG. 1A, and may be, for example, the configuration shown in FIG. 1B. The image processing apparatus 100 shown in FIG. 1B has a CPU 101, a ROM 102, a RAM 103, a VC (video card) 111, a general-purpose I / F 114, and a SATA (Serial ATA) I / F 118.

The CPU 101 uses the RAM 103 as a work memory to execute an OS (operating system) and various programs stored in the ROM 102, the external storage device 110, and the like to control the image processing device 100 and perform various processes. For example, the CPU 101 controls each configuration of the image processing device 100 connected to the system bus 108.

An input device 116 such as a mouse or keyboard and an imaging device 117 are connected to the general-purpose I / F 114 via a serial bus 115. An external storage device 110 is connected to the SATAI / F118 via the serial bus 119. A display 113 is connected to the VC 111 via a serial bus 112. The CPU 101 displays a UI (user interface) provided by the program on the display 113, and receives input information representing a user's instruction obtained via the input device 116.

The image processing device 100 shown in FIG. 1B is realized by, for example, a desktop PC. The image processing device 100 may be realized by a digital camera integrated with the image pickup device 117, a PC integrated with the display 113, or the like.

Further, in the above-described embodiment, the image processing device 100 has two cameras, an out-camera 202 and an in-camera 201, as the image pickup unit 106, but the image pickup unit 106 is not limited to this example. For example, the image processing device 100 may have a configuration in which only the out-camera 202 is provided as the image pickup unit 106.

Further, although the external storage device 110 in the above-described embodiment is an HDD, the external storage device 110 is not limited to this. For example, the external storage device 110 may be an SSD (Solid State Drive). Further, the external storage device 110 may be realized by a medium (recording medium) and an external storage drive for accessing the media. As the medium, for example, a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, a flash memory, or the like can be used.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or a device via a network or a storage medium, and one or more processors in the computer of the system or the device read and execute the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100: Image processing device 301: Image acquisition unit 302: Evaluation value acquisition unit 303: Learning data generation unit 304: Learning model generation unit 305: Image correction unit 306: Display control unit

Claims (12)

An evaluation value acquisition means for acquiring evaluation values for each of a plurality of evaluation items for each of the correction target image to be corrected and the target image referred to in the correction processing.
A data generation means for generating learning data based on the acquired evaluation values of the correction target image and the target image, and
A model generation means for generating a learning model related to the correction process for each evaluation item based on the generated learning data, and
An image processing apparatus comprising: an image correction means for correcting an image to be corrected by using the learning model selected from the learning models for each of the generated evaluation items. The image processing apparatus according to claim 1, wherein the image correction means selects the learning model to be used for correction based on the difference value between the evaluation values of the correction target image and the target image. 2. The image correction means is characterized in that the learning model corresponding to the evaluation item having the largest difference value between the evaluation values of the correction target image and the target image is selected as the learning model to be used for correction. The image processing apparatus according to. The image correction means is characterized in that the learning model corresponding to an evaluation item in which the difference value between the evaluation values of the correction target image and the target image exceeds a predetermined threshold value is selected as the learning model to be used for correction. The image processing apparatus according to claim 2. The image processing apparatus according to any one of claims 1 to 4, wherein the plurality of evaluation items include at least two of image color, noise, histogram, gloss, and contrast. The data generation means extracts a plurality of images close to the evaluation value of the correction target image from a plurality of images prepared in advance for each evaluation item, and approximates each of the extracted plurality of images to the evaluation value of the target image. The image processing apparatus according to any one of claims 1 to 5, wherein a set of the extracted image and the converted image is generated as learning data. The image processing apparatus according to any one of claims 1 to 6, wherein the model generation means acquires an image used in the past and additionally learns the learning model using the acquired image. .. Any one of claims 1 to 7, wherein the display means includes a display control means for displaying a plurality of images that are candidates for the correction target image and a plurality of images that are candidates for the target image in a list. The image processing apparatus according to the section. The display control means causes the display means to display a list of corrected images using the learning model for each evaluation item.
8. The image correction means is characterized in that the correction target image is corrected by using the learning model corresponding to the correction image selected from the correction images listed in the display means. The image processing apparatus described. The image processing apparatus according to any one of claims 1 to 9, further comprising an image acquisition means for acquiring an image designated as the correction target image and an image designated as the target image, respectively. An evaluation value acquisition process for acquiring evaluation values for each of a plurality of evaluation items for each of the correction target image to be corrected and the target image referred to in the correction processing.
A data generation step of generating learning data based on the acquired evaluation values of the correction target image and the target image, and
A model generation step of generating a learning model related to the correction process for each evaluation item based on the generated learning data, and
An image processing method comprising an image correction step of correcting the correction target image using the learning model selected from the learning models for each of the generated evaluation items. An evaluation value acquisition step for acquiring evaluation values for each of a plurality of evaluation items for each of the correction target image to be corrected and the target image referred to in the correction processing.
A data generation step of generating learning data based on the acquired evaluation values of the correction target image and the target image, and
A model generation step of generating a learning model related to the correction process for each evaluation item based on the generated learning data, and
A program for causing a computer to perform an image correction step of correcting the image to be corrected using the learning model selected from the learning models for each of the generated evaluation items.

Images