JP7346767B1 - Learning device and reasoning device - Google Patents

Abstract

[Problem] In data processing using machine learning, even when performing learning by inflating the original teacher data, prepare higher quality data and reliably suppress the decline in learning efficiency of machine learning models. We provide a learning device that allows you to A learning device includes an input unit 110 that receives input of original teacher data, a data processing unit 113 that creates processed teacher data based on the original teacher data, and a machine to be learned. a learning process execution unit 101 for executing a process of learning setting values in the machine learning model 111 to be learned based on teacher data including at least post-processed teacher data using the learning model 111; and a learning process execution unit 101 for storing setting values. A storage unit 12 is provided. With this configuration, the learning device prepares higher quality data even when performing learning by inflating the original teacher data in data processing by machine learning, and improves the learning efficiency of the machine learning model 111. The decline can be suppressed. [Selection diagram] Figure 2

Description

The present invention relates to a learning device and an inference device using a machine learning model, and particularly to a learning device and an inference device using a machine learning model for image processing.

Conventionally, data processing methods using machine learning can perform data processing on arbitrary data by giving a data set that serves as training data to a computer program and learning the program's parameters. A method of generating a trained model is known.

For example, in the process of "input image (teacher data) → learning program → output image (teacher data)" using machine learning, the parameters of the learning program are set so that the error between the input image and the output image is minimized. Calculate and generate a "trained model." By using this "trained model", the output image is converted from the input image to "input image (arbitrary data: e.g. low resolution image) → trained model → output image (inference data: e.g. high resolution image)". An output image can be generated by inferring.

In recent years, among machine learning, machine learning using neural networks has been applied to many fields. Especially in the fields of image recognition and voice recognition, deep learning that uses a multilayered neural network has demonstrated high recognition accuracy. Even in multi-layered deep learning, image processing is performed using a convolutional neural network (CNN) that uses convolutional layers and pooling layers multiple times to extract features of the input.

Image processing using neural networks includes super-resolution devices that increase the resolution of signals (see, for example, Patent Document 1) and highly accurate diagnostic support that facilitates understanding of differences in disease areas. There are diagnostic support devices (for example, see Patent Document 2). Further, a program, an image processing method, and an image processing apparatus that avoid deterioration due to free deformation of digital images are also disclosed (for example, see Patent Document 3).

Japanese Patent Application Publication No. 2020-27557 Japanese Patent Application Publication No. 2018-38789 Patent No. 6570164

In machine learning, the quality and quantity of training data are directly linked to the performance of machine learning models. Preparing a large amount of high-quality training data sets requires a lot of effort, so data augmentation is commonly used to obtain high performance with a small amount of training data. .

For example, in order to efficiently learn from a small amount of training image data, editing such as enlargement/reduction, rotation, diamond/trapezoid deformation, shift, color shift, sharpness/unsharpness, etc. can be applied to the training data to reduce the amount of data. Perform Data Augmentation. Among these, scaling up/down, rotation, rhombic/trapezoidal transformation, and shifting will cause blurring or aliasing if small image data is transformed as is, so if data is increased, blurring or noise will always occur above a certain level. New inflated data is generated. For this reason, in conventional data augmentation, minute high-frequency data included in images with good image quality are discarded, so learning using padded data with degraded image quality will warp the performance of the machine learning model. The problem arises that it gets worse.

Furthermore, the above-mentioned Patent Documents 1 to 3 do not disclose the content of preparing a large amount of high-quality learning data from a small amount of learning data.

The present invention has been made in view of the above-mentioned problems, and even when performing learning by inflating the original teacher data in data processing by machine learning, it is possible to prepare higher quality data and use the machine learning method. It is an object of the present invention to provide a learning device that can reliably suppress a decrease in learning efficiency of a learning model. Another purpose is to provide an inference device using this machine learning model.

In order to achieve the above object, the present invention provides a learning device that learns setting values in a machine learning model based on teaching data, comprising: an input unit that receives input of original teaching data; a data processing unit that creates processed training data based on the processing, a machine learning model to be learned, and the machine learning model to perform the learning based on the training data including at least the processed training data. The data processing unit includes a learning processing execution unit for executing a process of learning setting values in a target machine learning model, and a storage unit for storing the setting values, and the data processing unit calculates the target resolution of the processed teacher data. , obtain a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original teacher data, accept editing including rotation, movement, or deformation of the provisional reference image, and edit the provisional reference image. The tentative standard image after execution is sampled at a sampling rate higher than the predetermined magnification, and the sampled image is converted to an image size having the target resolution, and the processed training data is It is characterized by the following.

In this learning device, the learning device further includes an outpainting model that automatically generates the surroundings of the original teaching data in advance to create larger teaching data, and the data processing unit Preferably, the training data created by the processing model is converted into the processed training data.

In this learning device, the data processing unit determines an oversampling rate at a magnification higher than the predetermined multiple for the provisional reference image after editing, and oversamples at the determined rate. It is preferable to perform sampling and convert the image obtained by oversampling into the processed teacher data.

In this learning device, the learning device further comprises, when the resolution of the original teacher data is less than the predetermined times the target resolution, a super-resolution image of the predetermined times the original teacher data. It is preferable that the data processing unit has a super-resolution model for creating a super-resolution image, and that the data processing unit uses the super-resolution image created by the super-resolution model as the provisional reference image.

Further, in order to achieve the above object, the present invention provides a learning device characterized in that at least a part of a machine learning model that has learned the setting value in a learning processing execution unit of the learning device is used as a loss function. be.

Further, in order to achieve the above object, the present invention provides an inference device that executes a predetermined inference process on target data using a machine learning model, comprising an input unit that receives input of the target data; a machine learning model into which target data is input from the unit; and an inference processing execution unit that performs predetermined inference processing on the target data using the machine learning model that executes inference processing, the machine learning model is a machine learning model in which the setting value is learned by a learning processing execution unit of the learning device.

Further, the present invention is a computer program that causes a computer to operate as the learning device or inference device described above.

Further, in order to achieve the above object, the present invention provides a learning method for learning setting values in a machine learning model based on teacher data, comprising an input step of receiving input of original teacher data; a data processing step of creating processed teacher data processed based on the teacher data, a machine learning model to be learned, and using the machine learning model, based on the teacher data including at least the processed teacher data, The data processing step includes a learning process execution step for performing a process of learning setting values in the machine learning model to be learned, and a storing step for storing the setting values, and in the data processing step, the processed teacher data specify a target resolution of the image, obtain a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original training data, and perform editing including rotation, movement, or transformation on the provisional reference image. After receiving and editing, the provisional reference image is sampled at a sampling rate higher than the predetermined magnification, and the sampled image is converted to an image size having the target resolution, and the image is processed. This is characterized in that it is used as post-supervised data.

Further, in order to achieve the above object, the present invention provides an inference method for performing a predetermined inference process on target data using a machine learning model, comprising: an input step of receiving input of the target data; a machine learning model into which target data is input from the step, and an inference processing execution step of performing a predetermined inference process on the target data using the machine learning model that executes the inference process, the machine learning model is a machine learning model that has learned the setting value in the learning process execution step.

The learning device according to the present invention includes an input unit that receives input of original teacher data, a data processing unit that creates processed teacher data based on the original teacher data, and a machine learning model to be learned. a learning processing execution unit for performing a process of learning setting values in a machine learning model to be learned based on teacher data including at least post-processed teacher data using the learning process; and a storage unit for storing the setting values; Equipped with The data processing unit (1) specifies the target resolution of the processed training data, (2) obtains a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original training data, and ( 3) Accept editing including rotation, movement, or transformation of the provisional reference image; (4) Sample the provisional reference image after editing at a sampling rate higher than a predetermined magnification; and (5) Edit the provisional reference image after sampling. An image converted to an image size having a target resolution is used as processed teacher data. With this configuration, the learning device according to the present invention can prepare higher quality data and use the machine learning model even when performing learning by inflating the original teacher data in data processing by machine learning. It is possible to reliably suppress the decline in learning efficiency.

FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. It is a functional block diagram of the image processing device same as the above. It is a figure which shows the example of creation of the processed teacher data at the time of the learning operation of the image processing apparatus same as the above. It is a figure explaining the data padding at the time of the learning operation of the image processing device same as the above. FIG. 2 is an image diagram of a data set given during a learning operation in the image processing device same as the above. (a) A diagram illustrating a loss function used during a learning operation in a modification of the above image processing device, and (b) a diagram illustrating an example of a composite image generated in the modification.

(Embodiment)
An image processing apparatus according to an embodiment of the present invention will be described with reference to FIGS. 1 to 5. In this embodiment, the image processing device is a learning device that learns setting values in a machine learning model based on predetermined learning image data, or a learning device that performs predetermined inference processing on target image data using a machine learning model. It functions as at least one of the inference devices that executes.

First, each processing section included in the image processing apparatus 1 will be explained with reference to FIG. As shown in FIG. 1, the image processing device 1 includes a control section 10, an image processing section 11, a storage section 12, a communication section 13, a display section 14, an operation section 15, and a reading section 16. Note that although the image processing apparatus 1 and the operations thereof will be described below as one server computer, they may be configured so that processing is distributed among a plurality of computers.

The control unit 10 uses a processor such as a CPU and a memory to control the components of the device and realize various functions. The image processing unit 11 uses a processor such as a GPU or a dedicated circuit and a memory to perform image processing in accordance with control instructions from the control unit 10. The control unit 10 and the image processing unit 11 are configured as a single piece of hardware (SoC: System on a Chip) that integrates a processor such as a CPU and a GPU, a memory, a storage unit 12, and a communication unit 13. Good too.

The storage unit 12 uses a hard disk or flash memory. The storage unit 12 stores an image processing program 1P, a machine learning library 1L that functions as a machine learning model (for example, CNN), a super resolution model 1M, and an outpainting model 1N. Further, the storage unit 12 stores definition data that defines a machine learning model, parameters including set values in the trained machine learning model, and the like. The image processing unit 11 outputs a digital image whose resolution is higher than the original resolution based on the super-resolution model 1M stored in the storage unit 12. Note that the out-painting model 1N and the super-resolution model 1M can be generated by a machine learning model.

The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like. The display unit 14 is capable of displaying images through processing by the image processing unit 11 according to instructions from the control unit 10.

The operation unit 15 includes a user interface such as a keyboard or a mouse. A physical button provided on the housing may also be used. The operation unit 15 may also use software buttons displayed on the display unit 14. The operation unit 15 notifies the control unit 10 of operation information by the user. Specifically, when editing the original teacher data, the operation unit 15 allows the user to modify the teacher data, such as enlarging/reducing, rotating, shifting in the left/right direction, changing brightness/saturation, contrast, etc. Accepts settings for various transformation methods.

The reading unit 16 can read the image processing program 2P, machine learning library 2L, super resolution model 2M, and outpainting model 2N stored in the recording medium 2 using an optical disk or the like using, for example, a disk drive. It is possible. The image processing program 1P, machine learning library 1L, super resolution model 1M, and outpainting model 1N stored in the storage unit 12 are the image processing program 2P and machine learning library read by the reading unit 16 from the recording medium 2. 2L, super-resolution model 2M, and outpainting model 2N may be copied by the control unit 10 into the storage unit 12.

Next, the image processing function of the image processing device 1 will be explained with reference to FIG. 2. The control unit 10 of the image processing device 1 includes a learning process execution unit 101 and an inference process execution unit 102.
The learning processing execution unit 101 functions as a machine learning model (machine learning engine) based on the machine learning library 1L, definition data, and parameter information stored in the storage unit 12. In other words, the learning processing execution unit 101 uses the learning target machine learning model to perform a process of learning setting values (parameters, etc.) in the learning target machine learning model based on the learning image data or the learning text data. Execute. This setting value learning is a process that minimizes the difference between input data and answer data, such as updating parameters using a mini-batch gradient descent method. Furthermore, the learning processing execution unit 101 also functions as an image processing execution unit that edits image data that is input data based on operations using the operation unit 15.

The inference processing execution unit 102 performs image processing based on the image processing program 1P stored in the storage unit 12. That is, the inference processing execution unit 102 executes a predetermined inference process on input target data (target image data or inference target text data) using a machine learning model. The inference processing execution unit 102 also functions as an image processing execution unit that inputs image data, which is input data, to the input unit 110 based on the user's operation using the operation unit 15.

The image processing unit 11 includes an input unit 110, a machine learning model 111, a generator 111a and a discriminator 111b as the machine learning model 111, an output unit 112, a processing unit 113, and a cutting unit 114. For example, a machine learning model such as CNN learns using given training data, and definition data and parameter information that function as a generator 111a and definition data and parameter information that function as a discriminator 111b are stored in the storage unit 12. A model is created.

Examples of machine learning model learning typically include classification tasks (input side teacher image data → machine learning model → output label ⇔ teacher label), image conversion (input side teacher image data → machine learning model → output) These include image data ⇔ output side teacher image data), and image generation (input side teacher text data → machine learning model → output image data ⇔ output side teacher image data). Note that → here indicates the flow of data, and ⇔ indicates comparison during learning.

The input unit 110 receives input of teacher (learning) image data, classification labels, text data, or target image data to be inferred or inference target text data (see 12a in FIG. 2). Further, these data are recorded and held in the storage unit 12.

The data processing unit 113 performs processing for data augmentation of teacher data. Specifically, the data processing unit 113 (1) identifies the target resolution of the processed (inflated) teaching data, and (2) determines the target resolution based on the original teaching data input to the input unit 110. (3) accept editing including rotation, movement, or deformation of the provisional reference image; (4) obtain a provisional reference image that is a predetermined times the size of the image that has been edited; (5) Processing is performed in which the sampled image is converted to an image size having the target resolution and used as processed teacher data.

For example, as shown in FIG. 3, (1) the target resolution (here, same size) of the processed teacher data 304 with respect to the original teacher data 301 is specified. Next, (2) a provisional reference image 302 is prepared at a predetermined times the target resolution (here twice, that is, 224*224 pixels) with respect to the original teacher data 301 (here 112*112 pixels). . Note that in order to obtain the provisional reference image 302, for example, the image data stored in advance in the storage unit 12 or the image processing unit 11 increases the resolution of the original teacher data based on the super-resolution model 1M. A super-resolution image is generated and temporarily stored in the storage unit 12 or the like.

Next, (3) the editing (rotation in this case) for the provisional reference image 302 is accepted, and (4) the edited provisional reference image 303 is set at a sampling rate higher than the predetermined time (in this case, 2 times). Then sample at 3x). (5) The sampled image was reduced to the target resolution (here, the image that protruded from the area due to rotation was cut out and multiplied by 1/3) and converted to the size of the image having the target resolution. This data is used as the processed teacher data 304.

In this way, you can prepare an image (temporary reference image) that is larger (for example, twice the size) than the target image (image with the target resolution), and then rotate or move that image at a high sampling rate. If you then reduce the image to the target size, the image will not deteriorate.

In the description of this embodiment, the target resolution is the magnification of the original training data 301 to the processed training data 304, and the control unit 10 initially sets the resolution to the same resolution as the digital image (original) before processing. may be set as the target resolution (same size). Furthermore, if the editing includes enlarging/reducing data, the resolution of the image data after enlarging/reducing may be set as the target resolution (1 or more or 1 or less). Furthermore, if the editing includes rotation/transformation, the sampling rate may be set according to the rotation angle.

The target resolution does not necessarily have to exactly match the resolution after processing, and may be set to a value close to it. In that case, the resolution will be degraded to compensate for the lack of resolution. For example, if the processed teacher data is close to the size of the original teacher data, the target resolution is equal to the same size, the temporary reference image is twice the size, and the oversampling is three times the same. For example, if the processed training data is to be slightly larger than the original training data, the target resolution is 1.5 times, the temporary reference image is 3 times, and the oversampling is 5 times. This value may be set for each image or for each pixel.

Note that the data processing unit 113 determines the oversampling rate at a magnification higher than a predetermined magnification for the provisional reference image after editing, and performs oversampling at the determined rate. The image obtained by the above may be converted into training data after processing. Further, these settings for data processing may be recorded and held in the storage unit 12.

In the example in Figure 3, an image that is twice the size is prepared and reduced to 1/2, but when rotation is performed as an edit, the sampling rate is different in the vertical and horizontal directions and diagonally, so the sampling rate is different in the vertical and horizontal directions. The number of samples decreases relative to the sampling rate, and in diagonal directions the number of samples exceeds the sampling rate. For this reason, alias noise occurs particularly in diagonal directions. In order to suppress this noise, this noise will not occur if the rotation is performed at a high sampling rate and then reduced to the target size. For example, an image that is twice the target size is further enlarged to 1.5 times, rotated, and reduced to 1/3 (3 times oversampling). .

Also, in the case of image deformation, each pixel becomes larger or smaller, so even if each pixel has a provisional reference image that is larger than the target resolution and is adjusted to have a higher sampling rate than that, good. Without considering each pixel, a provisional reference image of sufficiently high resolution and a sufficiently high sampling rate may be set uniformly so that all pixels satisfy the conditions. For example, a provisional reference image with a high resolution is set to accommodate the enlarged portion, and a high sampling rate is set so as to accommodate the reduced or rotated portion.

Regarding editing, the control unit 10 of the image processing device 1 edits image data based on the image processing program 1P stored in the storage unit 12. In particular, the control unit 10 receives user operations via the operation unit 15 based on the image processing program 1P, and realizes editing of image data. Note that editing here refers to processing such as enlarging/reducing, rotation, diamond/trapezoidal deformation, shift, color shift, sharpness/unsharpness, etc. on the provisional reference image via the operation unit 15 in order to increase the data for learning. It is to administer.

Next, data padding will be explained with reference to FIGS. 4 and 5. As shown in FIG. 4, the data processing unit 113 performs various processing on the original teacher data 401 to generate processed teacher data 402 in which the apparent number of teacher data used during learning is increased. . At this time, by using the data processing unit 113 according to the present embodiment, deterioration of the processed teacher data 402 is effectively suppressed, and the quantity and quality of the padded teacher data are improved, and as a result, The performance of learning models can be dramatically improved.

For example, as shown in Figure 5 (II), in the past, the original training data was directly edited to inflate it, and even though the image remained the same size, it could be rotated or moved (shifted) by a fraction of a pixel. If this is done, the input side teacher image data 503 and output side teacher image data 504 after padding will deteriorate. This is because the image has parts such as boundary surfaces that change suddenly and do not exhibit waveform properties, so if waveform processing such as image transformation is performed, the signal in that part will be lost. . As a result, even if the teacher data is padded, the performance of the learning model deteriorates because the input side teacher image data 503 and the output side teacher image data 504 themselves have deteriorated.

On the other hand, as shown in FIG. 5(I), when the data processing unit 113 according to the present embodiment is used, the quality of input-side teacher image data 501 and output-side teacher image data 502 after padding is maintained, and machine learning High performance of the model can be achieved.

On the other hand, when the image processing device 1 functions as an inference device, it is an inference device that performs predetermined inference processing on target data using a machine learning model, and includes an input unit 110 that receives input of target data, and an input unit 110 that receives input of target data. A machine learning model 111 to which target data is input from the unit 110, and an inference processing execution unit 102 that performs predetermined inference processing on the target data using the machine learning model 111 that executes inference processing, The machine learning model 111 is a machine learning model 111 whose setting values have been learned by the learning processing execution unit 101 of the learning device described above.

When using a learned model, the machine learning model 111 performs data optimization processing (for example, increasing resolution, classifying, noise removal, and image generation) based on already learned parameters. When the machine learning model 111 at the time of inference is a CNN, it may include multiple stages of convolutional layers and pooling layers defined by definition data, and a fully connected layer, and extracts and extracts the feature values of image data. Image processing is performed based on the calculated feature values.

Specifically, during inference by the machine learning model 111, image data to be inferred and text data to be inferred can be input to the machine learning model 111, and post-inference image data can be obtained as an output from the output unit 112. The image data here is image data expressed using YCbCr or RGB. Further, the output is not limited to image data, and in the case of classification, the specified class is output. The output unit 112 outputs image data and classification classes to the storage unit 12. Note that the output data may be drawn as an image in the image processing section 11 and output to the display section 14.

Next, outpainting of image data in the learning device according to this embodiment will be explained. Conventionally, inflating image data basically involves increasing image data of the same size, so when scaling or rotating the image, the image is cut or filled with holes in the cutting unit 114 to restore the original size. Return to In conventional image augmentation operations, the unnaturalness of the image is reduced by performing arithmetic processing using the pixel values of surrounding pixels, such as filling in holes with nearby colors when the image is rotated or reduced after transformation. , which suppressed the deterioration of image quality.

On the other hand, the image processing unit 11 according to the present embodiment automatically generates the surroundings of the original teacher data in advance based on the outpainting model 1N stored in the storage unit 12 to create larger teacher data. The data processing unit 113 then converts the teacher data created by the outpainting model 1N into processed teacher data. In other words, in the conventional method, holes created in the processed image data are filled with nearby data or filled with black, but in this case an unnatural image is generated, but this embodiment In such an image processing device 1, an outpainting model 1N is used to first automatically generate and enlarge a naturally continuous image around the teacher image data, thereby preventing the occurrence of holes. By using this in conjunction with the above-described processing method of inflating the teaching data of the data processing unit 113, it becomes possible to obtain processed teaching data of higher quality.

As described above, when the image processing device 1 functions as a learning device that learns setting values in the machine learning model 111 based on teacher data, the learning device receives input of the original teacher data. The input unit 110, the data processing unit 113 that creates processed teacher data based on the original teacher data, and the machine learning model 111 to be learned are used to create teacher data including at least the processed teacher data. It includes a learning processing execution unit 101 for executing a process of learning setting values in a machine learning model 111 to be learned based on the learning target, and a storage unit 12 for storing the setting values. The data processing unit 113 (1) specifies the target resolution of the processed teacher data, (2) obtains a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original teacher data, (3) Accept edits that include rotation, movement, or transformation on the provisional reference image; (4) Sample the provisional reference image after editing at a sampling rate higher than a predetermined magnification; (5) After sampling. An image converted to an image size having a target resolution is used as processed teacher data. With this configuration, in data processing by machine learning, the learning device prepares higher quality data even when performing learning by inflating the original teacher data, reducing the learning efficiency of the machine learning model. can be reliably suppressed.

That is, when the image processing device 1 functions as a learning device, the data processing unit 113 performs transformation using a sufficiently large provisional reference image such as super-resolution when inflating the original teacher data. In addition, the processed training data is obtained after using oversampling with a large magnification. This makes it possible to maintain suitable image quality of the post-transformed teacher data, and also improves the performance of the machine learning model learned using the data. Further, by using this machine learning model, when the image processing device 1 functions as an inference device, it is possible to generate and classify image data after inference with higher accuracy.

(Modified example)
A modification of the image processing device 1 according to the embodiment of the present invention will be described with reference to FIG. 6. In this modification, the machine learning model 111 learned by the learning processing execution unit 101 is used as a loss function, and the loss functions are, for example, Content loss and Style loss. For example, as shown in FIG. 6(a), a system takes two inputs, a content image 601 that ensures structure and a style image 602 that ensures style, and outputs a composite image that has both the structure of the former and the style of the latter. Style transfer, which is a type of CNN, is known.

In this style transfer, first, a content image 601 and a style image 602 are input to the left and right trained VGG (Visual Geometry Group) networks, respectively, and a vector representing the likeness of each image is extracted from the intermediate feature map to create a loss function 603. Define. Then, optimization is carried out in a manner that minimizes the loss. Regarding the trained VGG, which is a type of machine learning model used here, the content image 601 and style image 602 are processed as the data in the above embodiment. If learning is performed using the undegraded images generated in the section 113, at the time the optimization is completed, the content as illustrated in FIG. 6(b) that captures the characteristics of high frequency components (high resolution) Generation of a composite image 604 combining styles can be expected.

Note that in the hardware configuration of the image processing device 1 according to the present embodiment, the communication section 13, the display section 14, and the reading section 16 are not essential. The communication unit 13 may not be used after being used, for example, when acquiring the image processing program 1P and the machine learning library 1L stored in the storage unit 12 from an external server device. Similarly, the reading unit 16 may not be used after reading and acquiring the image processing program 1P and the machine learning library 1L from the storage medium. The communication unit 13 and the reading unit 16 may be the same device using serial communication such as USB.

The image processing device 1 may function as a Web server and provide the function of the machine learning model 111 described above to a Web client device including a display section and a communication section. In this case, the communication unit 13 is used to receive requests from the Web client device and to transmit processing results.

As the error used during learning, an appropriate function such as a square error, absolute value error, or cross entropy error may be used depending on input/output data and the learning purpose. For example, if the output is a classification, use cross-entropy error. Regardless of using the error function, flexible operations such as using other criteria can be applied. This error function itself may be evaluated using an external machine learning model.

Note that the present invention is not limited to the configuration of the above-described embodiments, and various modifications can be made without changing the spirit of the invention. Furthermore, in order to achieve the object of the present invention, the present invention provides an image processing method (a learning method and an inference method) whose steps are characteristic configuration means included in an image processing device (a learning device and an inference device). , it can also be realized as a program including those characteristic steps. The program can not only be stored in a ROM or the like, but can also be distributed via a recording medium such as a USB memory or a communication network.

Furthermore, the present invention can be implemented as a computer system that transmits input data to an image processing device or computer program, and receives and uses output data from the image processing device or computer program. This system is a processing system that uses data obtained from a machine learning model that has been trained through the processing described above, and can provide various services. The device used in this system is an image processing device equipped with a display section and a communication section, or an information processing device capable of transmitting and receiving information to and from a computer, such as a so-called PC, a smartphone, a mobile terminal, a game device, and the like.

1 Image processing device (learning device and inference device)
10 Control unit 12 Storage unit (learning result storage unit)
15 Operation unit 101 Learning processing execution unit 102 Inference processing execution unit 110 Input unit 111 Machine learning model 111a Generator 111b Discriminator 112 Output unit 113 Data processing unit 114 Extraction unit 301,401 Original teacher data 302 Temporary reference image 303 Temporary reference image after editing 304,402 Teacher data after processing

Claims (11)

A learning device that learns setting values in a machine learning model based on training data,
an input section that accepts input of the original teacher data;
a data processing unit that creates processed teacher data based on the original teacher data;
learning to perform a process of learning setting values in the machine learning model to be learned, based on a machine learning model to be learned, and teacher data including at least the post-processed teacher data using the machine learning model; a processing execution unit;
comprising a storage unit that stores the setting value,
In the data processing department,
specifying a target resolution of the processed training data;
Obtaining a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original teacher data;
Accepting edits including rotation, movement, or transformation of the provisional reference image;
sampling the provisional reference image after editing at a sampling rate higher than the predetermined magnification;
A learning device characterized in that the processed teacher data is obtained by converting the sampled image into the size of the image having the target resolution. The learning device further includes:
It has an outpainting model that automatically generates the surroundings of the original training data in advance to create larger training data,
The learning device according to claim 1, wherein the data processing unit converts the teacher data created by the outpainting model into the processed teacher data. The data processing section is
Determining an oversampling rate at a magnification higher than the predetermined magnification for the provisional reference image after the editing is performed according to the editing;
Perform oversampling at the determined rate,
2. The learning device according to claim 1, further comprising converting an image obtained by oversampling into the processed teacher data. The learning device further includes:
If the resolution of the original teacher data is less than the predetermined times the target resolution, the method includes a super-resolution model that creates a super-resolution image of the predetermined size based on the original teacher data;
2. The learning device according to claim 1, wherein the data processing unit uses the super-resolution image created using the super-resolution model as the provisional reference image. A learning device according to any one of claims 1 to 4, wherein at least a part of a machine learning model that has learned the set value in a learning processing execution unit of the learning device is used as a loss function. An inference device that performs predetermined inference processing on target data using a machine learning model,
an input unit that accepts input of the target data;
a machine learning model into which target data is input from the input section;
an inference processing execution unit that executes a predetermined inference process on the target data using a machine learning model that executes the inference process,
An inference device, wherein the machine learning model is a machine learning model that has learned the setting value in a learning processing execution unit of the learning device according to any one of claims 1 to 4. A computer program that causes a computer to operate as the learning device according to any one of claims 1 to 4. A computer program that causes a computer to operate as the learning device according to claim 5. A computer program that causes a computer to operate as the inference device according to claim 6. A learning method for learning setting values in a machine learning model based on training data, the method comprising:
an input step for accepting input of source teacher data;
a data processing step of creating processed teacher data based on the original teacher data;
learning to perform a process of learning setting values in the machine learning model to be learned, based on a machine learning model to be learned, and teacher data including at least the post-processed teacher data using the machine learning model; a processing execution step;
a storing step of storing the set value;
In the data processing step,
specifying a target resolution of the processed training data;
Obtaining a provisional reference image that is a predetermined times the size of the image having the target resolution based on the original teacher data;
Accepting edits including rotation, movement, or transformation of the provisional reference image;
sampling the provisional reference image after editing at a sampling rate higher than the predetermined magnification;
A learning method characterized in that the sampled image is converted to a size of an image having the target resolution and is used as the processed teacher data. An inference method for performing predetermined inference processing on target data using a machine learning model,
an input step of accepting input of the target data;
a machine learning model into which target data is input from the input step;
an inference process execution step of executing a predetermined inference process on the target data using a machine learning model that executes the inference process,
An inference method characterized in that the machine learning model is a machine learning model that has learned the setting value in the learning process execution step according to claim 10.

Images