JP2021150855A - Image processing device and control method and program for the same - Google Patents

Abstract

To provide an image processing device that enables the color of an image after the image is converted into color image data to be closer to a color desired by a user, and control method and program for the same.SOLUTION: An image processing method for converting monochrome image data into color image data converts color image data of an object into monochrome image data, and stores the color image data as a color candidate image of the object in association with the converted monochrome image data. When the input image data is monochrome image data and a mode for converting monochrome image data into color image data is set, the object is detected from the input image data, and when the detected object is stored in storage means, the color candidate image associated with the object is displayed, and the input image data is converted to color image data by using a trained model based on the color image data corresponding to the displayed color candidate image.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing device, a control method thereof, and a program.

Many techniques related to image correction using artificial intelligence are known. One example of this is a technique for converting monochrome image data into color image data. This technique is intended to improve the appearance of documents or to make information easier to read. Since the color image data includes not only lightness but also hue and saturation information, it is possible to improve the visibility of the document information by colorizing the image data. Patent Document 1 describes a method of changing the monochrome image data into color image data by using the trained model when it is determined that the read image is a monochrome image.

Japanese Unexamined Patent Publication No. 2011-066734

However, in the conventional method, the color of the image after being converted into the color image data may not always be the color desired by the user. For example, depending on the object to be colorized, the actual color may differ even if the monochrome image data is the same, depending on the type of the object and the like. In such a case, there is a problem that the device that has executed the colorization cannot determine that the colorization has failed.

An object of the present invention is to solve at least one of the problems of the prior art.

An object of the present invention is to provide a technique capable of bringing the color of an image after conversion into color image data closer to a color desired by a user.

The image processing apparatus according to one aspect of the present invention in order to achieve the above object has the following configuration. That is,
A conversion means for converting the color image data of an object into monochrome image data,
A storage means for storing the color image data as a color candidate image of the object in association with the monochrome image data converted by the conversion means.
A detection means that detects an object from the input image data,
A display means for displaying a color candidate image associated with the object when the object detected by the detection means is stored in the storage means.
It is characterized by having a conversion means for converting the input image data into color image data using a trained model based on the color image data selected from the color candidate images displayed on the display means. ..

According to the present invention, there is an effect that the color of the image after being converted into the color image data can be brought close to the color desired by the user.

Other features and advantages of the present invention will become apparent in the following description with reference to the accompanying drawings. In the attached drawings, the same or similar configurations are given the same reference numbers.

The accompanying drawings are included in the specification and are used to form a part thereof, show an embodiment of the present invention, and explain the principle of the present invention together with the description thereof.
The block diagram explaining the hardware configuration of the image forming apparatus which concerns on embodiment. The top view of the operation part of the image forming apparatus which concerns on embodiment. The functional block diagram explaining the function of the image processing unit which concerns on embodiment. The block diagram explaining the hardware configuration of the learning processing part which concerns on embodiment. The functional block diagram explaining the function of the learning processing part which concerns on embodiment. The conceptual diagram which shows the input / output structure using deep learning in the machine learning part which concerns on embodiment. A flowchart illustrating a process of performing learning processing in the machine learning unit of the learning processing unit according to the embodiment and updating the learning model used in the color conversion unit of the image processing unit. The figure which shows the structural example of the learning table which concerns on embodiment. The figure explaining the specific processing of the color conversion part using the learning model which concerns on embodiment. The figure which shows the display example of the operation part of the image forming apparatus which concerns on embodiment. The figure which shows an example of the color tone adjustment screen displayed on the operation part of the image forming apparatus which concerns on embodiment. The flowchart explaining the copy process in the image forming apparatus which concerns on embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted. In the embodiment, an example of the image processing apparatus according to the present invention will be described using the image forming apparatus 10.

FIG. 1 is a block diagram illustrating a hardware configuration of the image forming apparatus 10 according to the embodiment.

The image forming apparatus 10 is a multifunction device (MFP: Multi Function Peripheral) having a plurality of functions such as a print function, a scanner function, a copy function, and a fax function. The image forming apparatus 10 includes an operation unit 150, a fax unit 160, a controller unit 100, a printer unit 120, a scanner unit 130, a power supply unit 110, a switch 145, and a power supply switch 148.

The controller unit (control unit) 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, a learning processing unit 105, a BIOS 106 (nonvolatile ROM), an image processing unit 107, a network I / F 108, and the like. These are connected to the CPU 101 via the system bus 109. The CPU 101 expands the program stored in the HDD 104 into the RAM 103, executes the expanded program, and controls the image forming apparatus 10. The ROM 102 stores, for example, a boot program for executing a process related to the startup of the controller unit 100, fixed parameters, and the like. The RAM 103 is used for storing programs and temporary data when the CPU 101 controls the image forming apparatus 10. The programs and temporary data expanded in the RAM 103 are read from the ROM 102 and the HDD 104 described later. The HDD 104 stores a program executed by the CPU 101, a program management table, and various data. The program to be executed is, for example, a boot program executed by the CPU 101 in order to start the OS when the image forming apparatus 10 is started. Although the HDD is used as the storage here, SSD, eMMC, NAND flash memory, NOR flash memory, or the like may be used. The network I / F 108 transmits / receives data to / from the network 118. Specifically, the data sent from the network 118 is received, the image data obtained by scanning the original with the scanner unit 130, and the data stored in the HDD 104 are sent to a predetermined destination via the network 118. Send.

The power supply unit 110 supplies power to each part of the image forming apparatus 10. When the power is off, the AC power is isolated by the power switch 148. When the power switch 148 is turned on, AC power is supplied to the AC-DC converter 141, and a DC power supply is created. The DC power supply is supplied by three independent power supply systems, and these power supply systems are controlled by the CPU 101. The CPU 101 controls the power supply to the controller unit 100 by turning on / off the switch 142. Further, the CPU 101 controls the supply of electric power to the printer unit 120 by turning on / off the switch 143. Further, the CPU 101 controls the supply of electric power to the scanner unit 130 by turning on / off the switch 144.

The learning processing unit 105 performs deep learning on the image data obtained by scanning the document by the scanner unit 130. In the embodiment, the learning processing unit 105 is provided in the image forming apparatus 10, but a learning server may be placed outside the image forming apparatus 10 and connected to the learning server by a network. The details of the learning processing unit 105 will be described later.

Next, other units and switches will be described.

The FAX unit 160 can transmit and receive digital image data via a telephone line or the like.

The operation unit 150 transmits to the controller unit 100 information necessary for creating job information such as a user name, the number of prints, and output attribute information input by the user using the operation panel 201. The scanner unit 130 reads the original and generates monochrome binary image data or color multivalued image data. The scanner unit 130 is connected to the CPU 101 by a scanner control I / F (not shown). The CPU 101 stores image data input from the scanner unit 130 via the scanner control I / F in, for example, a RAM 103 or the like.

The printer unit 120 prints image data obtained by converting PDL data received by the network I / F 108, image data obtained by the scanner unit 130, or the like on paper (sheet). The printer unit 120 includes a CPU 161 and a fuser 162. The fuser 162 fuses the toner image transferred to the paper to the paper by heat and pressure. Heat is generated in the fuser 162 by supplying electric power from the AC power source via the switch 145. Electric power may be supplied to the fuser 162 via the AC-DC converter 141. The CPU 161 functions as a printer controller. Power is supplied to the CPU 161 via the AC-DC converter 141, and by controlling the switch 145, the power supply to the fuser 162 can be controlled.

Whether or not to supply power to the image forming apparatus 10 is determined by turning on / off the power switch 148. The CPU 101 determines the on / off state of the power switch 148 by the Seesaw signal 149 connecting the power switch 148 and the CPU 101. Here, when the Seesaw signal 149 is at a high level, the power switch 148 is in an on state, and when it is at a low level, it is in an off state.

The BIOS 106 is a non-volatile memory that stores a boot program (BIOS). The image processing unit 107 is connected to the CPU 101, the printer unit 120, and the scanner unit 130. The image processing unit 107 performs image processing such as color space conversion on the digital image data output from the scanner unit 130, and outputs the image data after the image processing to the CPU 101. Further, the image processing unit 107 performs image processing such as color space conversion based on the image data obtained by the scanner unit 130, converts it into bitmap data, and outputs it to the printer unit 130. The details will be described later.

In addition to the copy function, the image forming apparatus 10 can execute the SEND function and the FAX function of storing the image data acquired by the scanner unit 130 in the HDD 104 and transmitting the image data via the network 118 and the FAX line. In the copy function, the scanner unit 130 reads the original and the image data acquired is printed on the paper by the printer unit 120. Further, the image forming apparatus 10 can print image data received from an external device (not shown) such as a PC connected via the network 118, or image data received by the FAX unit 160.

In the storage function of the image forming apparatus 10, the scanner unit 130 reads the original and saves the acquired image data in the HDD 104. The saved image data can be printed by the copy function, transmitted to the network 118 by using the SEND function described later, or transmitted to the FAX line via the FAX unit 160 by using the FAX function. The SEND function is a function of transmitting the image data stored in the HDD 104 and the image data obtained by the scanner unit 130 to a designated destination via the network 118. The FAX function is a function of facsimile transmitting image data stored in the HDD 104 and image data acquired by the scanner unit 130 by reading a document via a FAX line.

FIG. 2 is a top view of the operation unit 150 of the image forming apparatus 10 according to the embodiment.

The operation panel 201 is a combination of a liquid crystal display and a touch panel, displays an operation screen, and sends the information to the controller unit 100 when the display key is pressed by the user. The start key 202 is used when reading a document image, starting a printing operation, or instructing the start of other functions. The start key 202 incorporates two color LEDs, green and red, and when the green LED lights up, it indicates that the start is possible, and when the red LED lights up, it indicates that the start is not possible. The stop key 203 functions to stop the operation during operation. The hard key group 204 is provided with a numeric keypad, a clear key, a reset key, a guide key, and a user mode key. The power saving key 205 is used when shifting from the normal mode in which all the functions of the image forming apparatus 10 can be used to the sleep mode in which the image forming apparatus 10 performs only the minimum necessary operations, or returning to the normal mode. The image forming apparatus 10 shifts to the sleep mode when the power saving key 205 is pressed by the user in the normal mode, and shifts to the normal mode when the power saving key 205 is pressed by the user in the sleep mode.

FIG. 2A shows a home screen displayed in the initial state of the image forming apparatus 10, and on this home screen, buttons for instructing functions such as “copy”, “fax”, and “scan and send” are displayed. Has been done.

FIG. 2B is a diagram showing an example of a screen for instructing a function of using the scanner unit 130, for example, a black-and-white to color conversion, which is displayed when "copy" is selected on the screen of FIG. 2A. be.

Here, the user presses the button 210 when he / she wants to set the mode for converting the monochrome image into the color image. On the other hand, when the mode for converting the monochrome image to the color image is not set, the cancel button 211 is pressed. When any of the buttons is pressed here, in the above example, the screen shifts to the "copy" function setting screen (not shown).

Further, when the "black-and-white → color conversion" button 210 is pressed while the "black-and-white → color conversion" mode is set, the mode for colorizing the monochrome image data is canceled. In this case, even if the read image data is monochrome, the color conversion is not performed and the monochrome image data is output as it is. When the read image data is color image data, the color image data is output as it is, as in the case where the function for colorizing monochrome image data is active.

Next, a specific configuration of the image processing unit 107 will be described with reference to the block diagram of FIG.

FIG. 3 is a functional block diagram illustrating the function of the image processing unit 107 according to the embodiment.

The system bus I / F 301 is connected to the system bus 109, and is connected to each module of the image forming apparatus 10 via the system bus 109. The scan image processing unit 302 performs shading correction processing, MTF (Modification Transfer Function) correction processing, input gamma correction, and filter processing applied to the image data acquired by the scanner unit 130. Further, in addition to these, the scan image processing unit 302 performs image processing such as noise removal, color space conversion, rotation, and compression.

The rendering processing unit 303 expands the image data (PDL code) received from an external device (PC or the like) into bitmap data.

The print image processing unit 304 performs preprocessing for printing image data on the printer unit 120 (specifically, color space conversion processing for converting RGB to CMYK, halftone processing by the dither method or error diffusion method, and gamma correction. Etc.). The image data after image processing is output to the printer unit 120. Here, the print image processing unit 304 temporarily writes the image data to the RAM 103, which is a buffer for timing synchronization, in order to output the image data to the printer unit 120 in accordance with the activation and feeding of the printer unit 120. Then, the print image processing unit 304 reads out the image data from the RAM 103 and outputs the image data to the printer unit 120 in synchronization with the timing of feeding the recording medium.

The color determination unit 305 determines whether the image data processed by the scan image processing unit 302 is color image data or monochrome image data. Here, the tint of the pixel is acquired and integrated from the values of the R, G, and B components constituting the pixel data obtained by scanning the entire image data for each pixel. Then, when the integration result, that is, the cumulative value of the tint exceeds a predetermined value, it is determined that the image data is color image data. On the other hand, when the cumulative value of the tint is not more than a predetermined value, it is determined that the image data is monochrome image data.

The monochrome conversion unit 306 sets the value calculated by calculating the value of the RGB component for each pixel of the color image data processed by the scan image processing unit 302 as the pixel value of the monochrome image data, thereby converting the monochrome image data. Convert to.

The color conversion unit 307 converts the monochrome image data processed by the scan image processing unit 302 into color image data. In the embodiment, a learning model that performs calculation processing based on a machine learning technique composed of CNN (Convolutional Neural Network) is used. Details of the color conversion unit 307 will be described later.

The object detection unit 308 detects an object included in the image data processed by the scan image processing unit 302. Here, the object indicates an image area such as an illustration, a figure, a photograph, or a logo mark other than characters.

The object composition unit 309 writes back the object color-converted by the color conversion unit 307 to the area detected by the object detection unit 308. The above-mentioned function of the image processing unit 107 may be realized by the CPU and memory (both not shown) included in the image processing unit 107, or by executing the program expanded in the RAM 103 by the CPU 101. Is also good.

Next, the learning processing unit 105 will be described.

FIG. 4 is a block diagram illustrating a hardware configuration of the learning processing unit 105 according to the embodiment.

The image processing unit 105 has a CPU 401, a RAM 402, an HDD 404, and a GPU (Graphics Processing Unit) 403, which are connected to each other by a system bus 405, respectively. The CPU 401 generates learning data by reading a necessary program from the HDD 404, developing it in the RAM 402, and executing it. The RAM 402 is a system work memory when the CPU 401 executes a program. The HDD 404 is a hard disk drive that stores software for learning processing and generating learning data, as well as the generated learning data, learned models, and received learning data. The GPU 403 can perform efficient calculations by processing more data in parallel. In the embodiment, since the learning is performed a plurality of times by using deep learning, it is effective to perform the processing on the GPU 403. The details of deep learning will be described later with reference to FIG. The GPU 403 is also used when determining the monochrome image data input at the time of image formation from the learning model stored in the HDD 404.

In the embodiment, GPU 403 is used for learning and for determining image data. Specifically, when executing a learning program including a learning model, the GPU 403 performs learning by performing calculations. When determining the input image data and the training data, the pixel data of the input image data and the pixel data of the monochrome image data stored in the HDD 404 are compared, and the closest one is output. In the learning and determination processing, the CPU 401 or GPU 403 and the CPU 401 may cooperate to perform calculations.

FIG. 5 is a functional block diagram illustrating the function of the learning processing unit 105 according to the embodiment.

The learning processing unit 105 includes a learning data generation unit 501, a machine learning unit 502, a data storage unit 503, and an image discrimination unit 504. These functions are realized by the CPU 401 executing the program expanded in the RAM 402 or by the GPU 403.

The learning data generation unit 501 removes unnecessary data that becomes noise from the received learning data in order to obtain the desired learning effect, and processes the learning data into a form that can obtain an effective learning result. Optimize. The learning data generated in this way is stored in the HDD 404. This learning data is obtained from the image data stored in the HDD 104, the external data server (not shown) connected to the image forming apparatus 10, or the image data acquired by the scanner unit 130 reading the original. .. The learning data is generated by the CPU 401 of the learning processing unit 105.

The machine learning unit 502 inputs the learning data generated by the learning data generation unit 501, performs machine learning, and generates a trained model. This machine learning will be described later. Machine learning is executed by GPU 403 of the learning processing unit 105.

The data storage unit 503 stores the learning data generated by the learning data generation unit 501, the learned model generated by the machine learning unit 502, and the received learning data. Data is stored in HDD 404 or RAM 402.

The image discrimination unit 504 discriminates what kind of image data the monochrome image data input to the image processing unit 107 at the time of image formation is, using the trained model stored in the HDD 404. The image discrimination is executed by GPU 403 of the learning processing unit 105.

FIG. 6 is a conceptual diagram showing an input / output structure using deep learning in the machine learning unit 502 according to the embodiment. Here, a learning model using a neural network is illustrated as an example.

As an example for explaining the features of the image forming apparatus 10 according to the embodiment, let X be the learning data related to the generation of the learning model for predicting what is read by this neural network, and let Y be the output data. There is.

In FIG. 6A, when the image data of the apple is set to X as the learning data, the pixel information of each part is output as the output Y using the neural network. Here, the input data X is monochrome image data indicating an apple, and the output data Y is color image data corresponding to the apple.

Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, the decision tree, and the support vector machine, in addition to the neural network. Another example is deep learning in which features and coupling weighting coefficients for learning are generated by themselves using a neural network. In the embodiment, machine learning is performed using any of these algorithms that can be used as appropriate.

Further, the learning model may include an error detection unit and an update unit.

In FIG. 6B, the error detection unit includes the output data Y602 output from the output layer of the learning model W604 using the neural network and the teacher data T603 according to the input data X601 input to the input layer. Find the error. Then, using the loss function, the loss L representing the error between the output data Y602 from the neural network and the teacher data T603 is calculated. The update unit updates the coupling weighting coefficient between the nodes of the neural network and the like so that the loss L becomes smaller based on the loss L obtained by the error detection unit. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

The learning model W604 prepares a large number of learning data in which "input data whose correct answer value is known" and "correct answer value" are set. Then, the weighting coefficient of the learning model W is adjusted so that the output data when the input data corresponding to the correct answer value is input is as close as possible to the correct answer value. In this way, the work for obtaining the highly accurate learning model W is performed. This is called a learning process, and a learning model adjusted through this learning process is called a trained model.

FIG. 7 is a flowchart illustrating a process of performing learning processing by the machine learning unit 502 of the learning processing unit 105 according to the embodiment and updating the learning model used by the color conversion unit 307 of the image processing unit 107. The process shown in this flowchart is executed by the GPU 403 of the learning processing unit 105 and the monochrome conversion unit 306 of the image processing unit 107. Here, the machine learning unit 502 performs learning processing in cooperation with the image processing unit 107.

First, in S701, the GPU 403 saves the image data of the color object determined by the color determination unit 305 as a color image among the objects detected by the object detection unit 308 of the image processing unit 107 in the HDD 404 as the color candidate image data of the learning sample. .. Next, the process proceeds to S702, and the monochrome conversion unit 306 of the image processing unit 107 reads out the color image data stored in the HDD 404, converts it into monochrome image data, and stores it in the RAM 402. As described above, here, the value calculated by calculating the RGB component for each pixel of the color image data is converted into the monochrome image data by making it the pixel value of the corresponding monochrome image data. When the image data read from the HDD 404 is in a compression format such as JPEG, the image data is decompressed by the compression / decompression processing unit (not shown), and then the monochrome conversion processing is performed.

Next, the process proceeds to S703, and the GPU 403 stores the monochrome image data and the object thereof, which have been converted to monochrome in S702 and stored in the RAM 402, in the HDD 404 as input image data for the learning process. Next, the process proceeds to S704, and the GPU 403 adds the image files stored in S701 and S703 to the learning table.

FIG. 8 is a diagram showing a configuration example of the learning table according to the embodiment.

The learning table has the paths of the above-mentioned image data and is stored in the HDD 404. In this training table, the input image data of the training sample indicates the file name of the monochrome image data saved in S703 and its object, and the candidate image data of the training sample indicates the path of the color image data saved in S701. ..

In this learning table, the input image data and the candidate image data are linked. Then, when monochrome image data is input, it is converted into color image data by referring to this learning table. When storing in this learning table, if the features of the shape of the object in the input image data are common within a certain range and the colors are different, the same object has different varieties. Is determined. Image data of the same object but different varieties determined in this way are associated with the input image data as a plurality of candidate image data and stored. For example, in the example of FIG. 8, three candidate image data (kouho001_001.jpg, kouho002_002.jpg, kouho003_003.jpg, all of which are color image data) with respect to the input image data (picture001.jpg) whose object is (object_001). Is registered.

Next, the process proceeds to S705, and the GPU 403 performs learning using the image data shown in the learning table generated in S704 and the color conversion unit 307. Then, the learning model of the color conversion unit 307 is updated.

FIG. 9 is a diagram illustrating specific processing of the color conversion unit 307 of the image processing unit 107 using the learning model according to the embodiment.

Each convolution layer has a two-dimensional filter function (for example, a 4 × 4 filter) (not shown), and the convolution operation is performed on the input image data of each convolution layer by using the two-dimensional filter function. Further, each pooling layer is provided with a two-dimensional pooling function (not shown), and pixels matching the conditions are extracted from the pixels in the window of the two-dimensional pooling function for the input of each pooling layer. For example, if the window size is a maximum pooling of 2 × 2, the pixel having the maximum pixel value is extracted from the four pixels in the window.

The learning model in the color conversion unit 307 according to the embodiment learns in a configuration in which a plurality of pairs of a convolution layer and a pooling layer are connected in series.

First, a monochrome image having only luminance information is prepared as input data 901. This is input to the convolution layer of the first layer to perform the convolution process, and then the pooling process is performed. In the next convolution layer and pooling layer, the processing result of the previous layer is input and processing is performed. This is repeated by the number of connected pairs, and the data after the pooling process in the last pooling layer is mapped to the ab space of the color space (Lab space) composed of the brightness (L) and the saturation (ab). Generate intermediate color image data. Then, the intermediate color image data and the luminance image data are combined and converted into the color image data 902 to perform the colorization process.

Here, the input data 901 is monochrome data indicating an apple, and the color image data 902 is image data indicating, for example, a red apple.

Subsequently, the behavior of the image forming apparatus 10 when colorizing a monochrome image will be described.

FIG. 10 is a diagram showing a display example of the operation unit 150 of the image forming apparatus 10 according to the embodiment.

As described above with reference to FIG. 2B, when reading image data, it is possible to set a mode for converting a monochrome image into a color image. When the mode for converting a monochrome image to a color image is set in this way, the display of "black and white → color conversion" is activated on the operation panel 201 of the operation unit 150.

In FIG. 10A, when a monochrome image is read and printed, a colorization process is performed by the image forming apparatus 10, and a candidate image of the colorized image data is displayed on the operation panel 201. When the user selects a desired image from these displayed candidate images, color image data based on the selected candidate image is output.

In the example of FIG. 10A, candidate images 1001 to 1003 are displayed. Here, the candidate image 1001 shows a red apple such as "Fuji", the candidate image 1002 shows a yellow apple such as "Kiou", and the candidate image 1003 shows a green apple such as "Ourin". ..

On the other hand, in FIG. 10A, when the desired candidate image is not included in these candidate images, the user selects the “color correction” button 1004. As a result, for example, a screen as shown in FIG. 11 is displayed.

FIG. 10B is a diagram showing a display example of colorized image data when there is no candidate image. FIG. 10B will be described with reference to the flowchart of FIG.

FIG. 11 is a diagram showing an example of a color tint adjustment screen displayed on the operation unit 150 of the image forming apparatus 10 according to the embodiment.

The user can specify the desired part of the color image data 1101 as the color desired by the user by designating the desired part of the color image data 1101 and selecting the color tones in the color map 1101 on this screen. If the color image data is input while the function for colorizing the monochrome image data is active, it is output as it is.

The color set by the user on this screen is added to the learning table, and when the same image data is input next time, the candidate image of the set color is also displayed.

Next, the behavior of colorizing a monochrome image will be described with reference to the flowchart of FIG.

FIG. 12 is a flowchart illustrating a copy process in the image forming apparatus 10 according to the embodiment. The program that executes this process is stored in the HDD 104, is expanded in the RAM 103 at the time of execution, and the CPU 101 executes the expanded program to realize the process shown in this flowchart.

First, in S1201, the CPU 101 sets the document reading mode when the "copy" function is selected on the home screen described above. Then, whether or not the monochrome → color conversion mode is set on the screen of FIG. 2B at this time is stored in the RAM 103. Then, when the start key 202 is pressed, the process proceeds to S1202, and the CPU 101 starts scanning the document by the scanner unit 130. Next, the process proceeds to S1203, and the CPU 101 determines whether the image data obtained by reading the original is monochrome image data. If the image data is not monochrome, the process proceeds to S1212, image formation is executed based on the color image data obtained by reading the data, and this process is completed.

On the other hand, in S1203, when the CPU 101 determines that the image data obtained by reading the original is monochrome image data, the process proceeds to S1204, and determines whether or not the monochrome → color conversion mode is stored in the RAM 103. Here, when the mode is not the monochrome → color conversion mode, the process proceeds to S1212, image formation is executed based on the monochrome image data obtained by reading the data, and this process is completed. On the other hand, when it is determined in S1204 that the monochrome → color conversion mode is set, the process proceeds to S1205, and the CPU 101 extracts the object included in the input monochrome image data and proceeds to S1206. In S1206, the CPU 101 determines whether or not the monochrome image data of the extracted object is stored in the learning table shown in FIG. If it is determined that the data is stored here, the process proceeds to S1207, and the CPU 101 acquires a candidate image associated with the monochrome image data of the object and displays it on the operation panel 201 of the operation unit 150, for example, as shown in FIG. 10 (A). indicate.

Next, the process proceeds to S1208, and the CPU 101 proceeds to S1212 when the user selects one of the candidate images on the screen of FIG. 10A, for example. In S1212, the CPU 101 converts the input monochrome image data into color image data based on the image data of the selected candidate image. At this time, as described above with reference to FIG. 6, the learning model is used to convert the data into color image data. Then, the process proceeds to S1213, and the CPU 101 executes image formation based on the converted color image data and ends this process.

On the other hand, if it is determined in S1208 that the user has pressed the color correction button 1004 in FIG. 10A, the process proceeds to S1209. In S1209, the CPU 101 converts the black-and-white image data input using the trained model into color image data as described above with reference to FIG. Next, the process proceeds to S1210, and the CPU 101 displays, for example, the screen shown in FIG. 11 on the operation panel 201. Then, on that screen, the color of the object is changed according to the color candidates to be colored selected by the user. Then, when the start key 202 is pressed, the process proceeds to S1211, and the CPU 101 updates the learning table according to the selected color. At this time, the color image data obtained in S1209 is also corrected according to the tint selected in S1210. Then, the process proceeds to S1213, and the CPU 101 executes image formation based on the converted color image data and ends this process. When the cancel button is pressed on the screen of FIG. 11, the screen returns to the screen of FIG. 10 (A).

Further, in S1206, when the CPU 101 determines that the extracted object is not stored in the learning table shown in FIG. 8, the process proceeds to S1214. In S1214, the CPU 101 converts the input monochrome image data into color image data. This is converted into color image data using a learning model as described above with reference to FIG. Then, the conversion result is displayed on the operation panel 201 as shown in FIG. 10B, for example.

In FIG. 10B, image 1005 shows the color image data obtained by the color conversion in S1214. Then, the process proceeds to S1215, and the CPU 101 waits for the start key 202 to be pressed or the color correction button 1006 to be pressed in the state shown in FIG. 10 (B). If the color image on this screen is satisfactory to the user, the start key 202 is pressed. As a result, the process proceeds to S1213, and the CPU 101 forms an image according to the color image data and ends this process.

On the other hand, when the color correction button 1006 is pressed in S1215, the process proceeds to S1210, and the CPU 101 displays, for example, the screen shown in FIG. 11 on the operation panel 201 as described above. On that screen, change the color of the object according to the color candidates selected by the user. Then, when the start key 202 is pressed, the process proceeds to S1211, and the CPU 101 updates the learning table according to the selected color and corrects the tint of the color image data obtained in S1214. Then, the process proceeds to S1213, and the CPU 101 executes image formation based on the color image data thus obtained, and ends this process. At this time, when the cancel button is pressed on the screen of FIG. 11, the screen returns to the screen of FIG. 10 (B).

As described above, according to the embodiment, the object is determined from the read image data using the trained model, coloring is performed, and a plurality of coloring candidates are displayed so that the user can select from them. do. As a result, it is possible to perform the desired coloring for the user, and it is possible to improve the convenience.

In the embodiment, an image forming apparatus is given as an example of the image processing apparatus, but if there is a function of determining an object of an object from an input monochrome image using a trained model and colorizing the image forming apparatus, the image forming apparatus is used. It can be applied to all devices, not limited to.

In the embodiment, a plurality of color candidates corresponding to a plurality of varieties of the object are displayed, and the color candidates selected by the user are used as the color of the object, thereby eliminating the determination of the varieties of the object.

On the other hand, for example, the product type of the object may be determined and the color candidates of the product type may be displayed. That is, for example, in FIG. 10A, when an apple variety is determined to be "Fuji", a plurality of color candidate images close to the color of the "Fuji" apple may be displayed.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached.

10 ... Image forming device, 100 ... Controller unit, 101 ... CPU, 104 ... HDD, 105 ... Learning processing unit, 107 ... Image processing unit, 120 ... Printer unit, 130 ... Scanner unit, 150 ... Operation unit

Claims (11)

A conversion means for converting the color image data of an object into monochrome image data,
A storage means for storing the color image data as a color candidate image of the object in association with the monochrome image data converted by the conversion means.
A detection means that detects an object from the input image data,
A display means for displaying a color candidate image associated with the object when the object detected by the detection means is stored in the storage means.
A conversion means for converting the input image data into color image data using a trained model based on the color image data selected from the color candidate images displayed on the display means.
An image processing device characterized by having. The image processing apparatus according to claim 1, wherein the color candidate image is a color image of the same object as the object but of different types of objects. A conversion means for converting the color image data of an object into monochrome image data,
A storage means for storing the color image data as a color candidate image of the object in association with the monochrome image data converted by the conversion means.
The first determination means for determining whether the input image data is monochrome image data,
When the first determination means determines that it is monochrome image data, the second determination means for determining whether or not the mode for converting the monochrome image data into the color image data is set.
When the second determination means determines that the mode is set, the detection means for detecting an object from the input image data and the detection means.
A display means for displaying a color candidate image associated with the object when the object detected by the detection means is stored in the storage means.
A first conversion means for converting the input image data into color image data using a trained model based on the color image data corresponding to the color candidate image displayed on the display means.
An image processing device characterized by having. The image processing apparatus according to claim 3, further comprising a second conversion means for converting the input image data into color image data using the trained model. The image processing apparatus according to claim 4, further comprising an output means for outputting the color image data converted by the first conversion means or the second conversion means. The image processing apparatus according to claim 4, further comprising a correction means for correcting the tint of the color image data obtained by the second conversion means. The image processing apparatus according to any one of claims 3 to 6, wherein the color candidate image is the same object as the object but is a color image of an object of a different kind. Further having a reading means for reading a document,
The image processing apparatus according to any one of claims 1 to 7, wherein the input image data is image data input by reading a document by the reading means. A control method for controlling an image processing device.
A conversion process that converts the color image data of an object into monochrome image data,
A storage step of storing the color image data as a color candidate image of the object in a storage unit in association with the monochrome image data converted by the conversion step.
A detection process that detects an object from the input image data,
A display step of displaying a color candidate image associated with the object when the object detected in the detection step is stored in the storage unit, and a display step of displaying the color candidate image associated with the object.
A conversion step of converting the input image data into color image data using a trained model based on the color image data selected from the color candidate images displayed in the display step.
A control method characterized by having. A control method for controlling an image processing device.
A conversion process that converts the color image data of an object into monochrome image data,
A storage step of storing the color image data as a color candidate image of the object in the storage unit in association with the monochrome image data converted in the conversion step.
The first determination step of determining whether the input image data is monochrome image data,
When the first determination step determines that the monochrome image data is used, the second determination step of determining whether or not the mode for converting the monochrome image data into the color image data is set.
When the second determination step determines that the mode is set, the detection step of detecting an object from the input image data and the detection step.
A display step of displaying a color candidate image associated with the object when the object detected by the detection step is stored in the storage unit, and a display step of displaying the color candidate image associated with the object.
A conversion step of converting the input image data into color image data using a trained model based on the color image data corresponding to the color candidate image displayed in the display step.
A control method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

Images