JP6347028B1 - UPSAMPLE DEVICE, COMPUTER PROGRAM, COMPUTER SYSTEM, DEVICE, AND UPSAMPLE METHOD - Google Patents

Abstract

High-precision upsampled data is created at high speed. An up-sampling apparatus (1) up-samples a plurality of learning subsets in a first hierarchy including data of a third sample rate higher than a second sample rate from a second sample rate to a third sample rate. A first layer filter acquisition unit that acquires a second filter that up-samples from a first sample rate lower than the first sample rate and the second sample rate to a second sample rate, and second input from the second sample rate input data A reduced data creation unit that creates reduced data at one sample rate, a candidate data creation unit that upsamples the reduced data to create candidate data at the second sample rate using each second filter, and input candidate data Using the first filter corresponding to the second filter that minimizes the difference from the data, And a up sample processing unit for upsampling input data Rureto. [Selection] Figure 1

Description

  The present invention relates to an upsampling apparatus that performs upsampling processing on input data, a computer program that realizes the upsampling apparatus, a computer system that uses the upsampling apparatus, a device that uses the upsampling apparatus, and the upsampling apparatus The present invention relates to an upsampling method used in the above.

  In order to improve the quality of images, sound, and other data, an upsampling process that increases the number of samples per unit scale is performed. In particular, in recent years, a super-resolution processing technique for performing high-resolution processing by applying super-resolution processing to an input image has been put into practical use (see, for example, Patent Document 1).

  As a method of up-sampling processing, input data is interpolated and deformed as necessary, but it is conceivable to use a filter that learns from the input data and output data what interpolation and deformation should be performed. As an example of a filter, in super-resolution processing, an input image is input to a machine-learned convolutional neural network (hereinafter referred to as “CNN”), and the input image is subjected to super-resolution processing. A method for outputting an output image with a high resolution is known. The CNN is configured by performing machine learning on the weight and bias of the neural network in each layer, using the reduced image as an input and the original image as an output based on the original image and the reduced image of the original image.

Special table 2013-532878 gazette

  If the data is an image, the image includes various types such as a portrait photograph, a landscape photograph, and an illustration. For example, illustrations include many high frequency components because they include many edge components, while landscape images include all frequency components evenly.

  If the data is sound, the sound includes various types such as a human voice, music played with a musical instrument (for example, a violin), and recording in sports (for example, baseball). For example, human voices do not contain strong high-frequency components due to utterance restrictions, whereas violin performances contain high-frequency components as harmonics of strings.

  However, ignoring the characteristics of these data, one filter (hereinafter referred to as “CNN”. Even if it is processing other than CNN, for example, processing based on the least square estimation result of parameters used for interpolation) In some cases, satisfactory results may not be obtained.

  For example, when an image containing many high-frequency components is subjected to super-resolution processing by CNN that has been machine-learned using many images containing many low-frequency components, the reproducibility of high-frequency components may be impaired. Further, when an image containing a large amount of low-frequency components is subjected to super-resolution processing by machine learning using a large number of images containing a large amount of high-frequency components, high-frequency noise may be generated.

  Also, for acoustics, upsampling does not affect the low range and will inevitably emphasize or de-emphasize the high range, but it should not emphasize the high range very much (eg bass or baritone singer). When a sound (for example, a performance of a violin) that should emphasize the high frequency range is upsampled by CNN that is machine-learned using a lot of singing), it is conceivable that the high tone does not grow and the tone does not become rich. In addition, when CNN that has been machine-learned using a large amount of sound that should emphasize the high range is subjected to upsampling for sounds that should not emphasize the high range so much, an unnatural timbre will result due to the emphasized high tone. It is possible.

  In addition, it is generally difficult for the user to select an optimal one from a plurality of CNN presets. For this reason, conventionally, in the super-resolution processing of an image, it has been necessary to create a CNN using many types of learning images as described above. When many types of learning images are used, the number of learning images is enormous, so that time is required for machine learning and the configuration of the CNN is complicated. The same applies to data other than images.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an upsample processing apparatus, device, computer program, and upsample method that can create high-precision upsample data at high speed. To do.

In order to solve the above-described problem, an upsampling apparatus according to an aspect of the present invention relates to data representing a value for a one-dimensional or two-dimensional space as a discrete value at a sample point in the space.
An upsampling device for upsampling input data of a second sample rate to obtain output data of a third sample rate,
For each of the plurality of learning subsets in the first hierarchy, which is a subset of the learning set including the data of the third sample rate, the data of the second sample rate that has been machine-learned using the learning subset is the third sample. A first filter for upsampling to rate data;
A first layer filter acquisition unit for acquiring a second filter for up-sampling data of a first sample rate obtained by down-sampling data of the second sample rate to the second sample rate;
A candidate data creation unit that upsamples the data of the first sample rate and creates candidate data of the second sample rate by using the second filters obtained by the first hierarchy filter obtaining unit;
By up-sampling the input data using a first filter corresponding to a second filter that minimizes the difference between the candidate data created by the candidate data creation unit and the input data, a third sample rate And an output upsampling unit for creating output data.

  According to this configuration, the second filter that minimizes the difference between the second sample rate candidate data and the second sample rate input data can be selected from the plurality of second filters. Also, the input data can be upsampled using the first filter corresponding to the selected second filter. Since the second filter is machine-learned using the same learning subset as the first filter, it has the same properties as the first filter, and the sample rate of the input data targeted by the second filter is the first. One filter is smaller than the sample rate of the target input data. For this reason, the second filter having the optimal property for the input data can be selected at high speed, and by performing the upsampling process on the input data using the first filter corresponding to the selected second filter, Accurate upsampled data (output data) can be created at high speed. The “upsampling apparatus” naturally includes “super-resolution processing apparatus”.

  One form of upsampling apparatus of the present invention upsamples values for a three-dimensional space.

  According to this configuration, upsampling can be performed on moving image data, CT image data, and other data relating to a three-dimensional space.

A preferred form of the upsampling device of the present invention is as follows:
For each of a plurality of learning subsets in the second hierarchy, which is a subset of the learning subset in the first hierarchy used for machine learning of the second filter with the smallest difference, using the learning subset A second hierarchical filter acquisition unit that acquires the first and second machine-learned filters;
The candidate data creation unit uses the second filters of the second hierarchy acquired by the second hierarchy filter acquisition unit to upsample the data of the first sample rate to create the candidate data,
The output upsampling unit corresponds to a first filter corresponding to a second filter that minimizes a difference between the candidate data created by the candidate data creation unit using the second filter of the second hierarchy and the input data. Output data at the third sample rate is created by up-sampling the input data using a filter.

  According to this configuration, it is possible to select the second filter in the first hierarchy, and further select the second filter in the second hierarchy based on the selected second filter. Further, the input data can be upsampled using the first filter corresponding to the selected second filter. Thus, since the first filter and the second filter can be hierarchically structured, the second filter and the first filter corresponding to the second filter can be efficiently selected. Thereby, highly accurate output data can be created at high speed.

The computer program of the present invention is:
A computer is operated as the upsampling apparatus of the present invention.

  According to this configuration, the upsampling apparatus of the present invention is implemented as a computer program.

A computer system according to an aspect of the present invention includes:
Comprising the upsampling apparatus or computer program of the present invention ,
The input data is transmitted to the upsampling apparatus or the computer program , and the output data is received and used.

  According to this configuration, the upsampling process can be naturally performed on the data processed by the computer program, and the upsampling apparatus or the computer program of the present invention can be utilized when the computer system needs the upsampling. A computer system using the upsampling apparatus or computer program of the present invention can be constructed.

The device of the present invention
Comprising the upsampling apparatus or computer program of the present invention ,
A device that transmits the input data to the upsampling device or the computer program , such as a television device, a camera, a measurement device, a smartphone, a game device, an audio device, and other devices.

  According to this configuration, various devices operate using the upsampling apparatus of the present invention.

The upsampling method of the present invention comprises:
For data expressing a value for a space of one dimension or two or more dimensions as a discrete value at a sampling point of the space,
A method for causing an upsampling device or computer program to upsample input data at a second sample rate to obtain output data at a third sample rate, comprising:
For each of the plurality of learning subsets in the first hierarchy, which is a subset of the learning set including the data of the third sample rate, the data of the second sample rate that has been machine-learned using the learning subset is the third sample. Obtaining a first filter for upsampling to rate data;
Obtaining a second filter for up-sampling the data of the first sample rate down-sampled to the second sample rate to the second sample rate;
Using the second filters acquired by the first hierarchical filter acquisition unit to upsample the data of the first sample rate to create candidate data of the second sample rate;
Creating output data at a third sample rate by up-sampling the input data using a first filter corresponding to a second filter that minimizes the difference between the candidate data and the input data; Prepare.

  According to this configuration, the processing procedure of the upsampling apparatus described above can be used as a method, and can be mounted as a device that is not necessarily a device (for example, a semiconductor integrated circuit). That is, the second filter that minimizes the difference between the candidate data of the second sample rate and the input data of the second sample rate can be selected from the plurality of second filters. Also, the input data can be upsampled using the first filter corresponding to the selected second filter. Since the second filter is machine-learned using the same learning subset as the first filter, it has the same properties as the first filter, and the input data sample rate targeted by the second filter is the first. Less than the input data sample rate targeted by the filter. For this reason, the second filter having the optimal property for the input data can be selected at high speed, and by performing the upsampling process on the input data using the first filter corresponding to the selected second filter, Accurate upsampled data (output data) can be created at high speed. The “upsampling method” naturally includes a “super-resolution processing method”.

The upsampling method of the present invention comprises:
For each of a plurality of learning subsets in the second hierarchy, which is a subset of the learning subset in the first hierarchy used for machine learning of the second filter with the smallest difference, using the learning subset Obtaining a machine-learned first filter and a second filter;
The step of creating candidate data up-samples the data of the first sample rate by using the second filters of the second hierarchy acquired in the step of acquiring the second filter, make,
Step includes: a difference between the candidates data and the input data that was created using the second filter of the second hierarchy in the step of preparing the candidate data is minimized to create an output data of said third sample rate Using the first filter corresponding to the second filter, the input data is upsampled to generate output data at the third sample rate.

  According to this configuration, it is possible to select the second filter in the first hierarchy, and further select the second filter in the second hierarchy based on the selected second filter. Further, the input data can be upsampled using the first filter corresponding to the selected second filter. Thus, since the first filter and the second filter can be hierarchically structured, the second filter and the first filter corresponding to the second filter can be efficiently selected. As a result, more accurate output data can be created at a higher speed.

  According to the upsampling apparatus, computer program, and upsampling method of the present invention, highly accurate output data can be created at high speed.

  According to the computer system and apparatus of the present invention, upsampled data (output data) can be utilized.

FIG. 1 is a block diagram showing the configuration of the upsample processing apparatus. FIG. 2 is a diagram for explaining the data size and the filter. FIG. 3 is a diagram illustrating an example of a learning set and filter having a hierarchical structure. FIG. 4 is a flowchart showing a processing procedure of filter creation processing executed by the upsample processing apparatus. FIG. 5 is a flowchart showing a processing procedure of input data upsampling processing executed by the upsampling processing apparatus. FIG. 6 is a diagram illustrating a computer system and devices that utilize the upsample processing apparatus.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each of the embodiments described below shows a preferred specific example of the present invention. Numerical values, shapes, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. The invention is specified by the claims. Therefore, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept of the present invention are not necessarily required to achieve the object of the present invention. It will be described as constituting a preferred form.

  FIG. 1 is a block diagram showing the configuration of the upsample processing apparatus according to the embodiment of the present invention. The upsampling processing device 1 is a device that performs upsampling processing on input data and creates data having a higher sample rate than the input data. The upsampling processing device 1 includes an input data acquisition unit 10, a reduced data creation unit 11, and machine learning. Unit 12, first layer filter acquisition unit 13, candidate data creation unit 14, second layer filter acquisition unit 15, upsample processing unit 16, and storage device 17. Hereinafter, processing performed by the upsample processing apparatus 1 will be described.

  The input data acquisition unit 10 acquires input data to be subjected to upsampling processing. For example, the input data acquisition unit 10 reads out data selected by the user from the data stored in the storage device 17 using a keyboard or the like from the storage device 17 to acquire the data as input data. Also good. The input data acquisition unit 10 may acquire the data as input data by downloading the data via a network or the like.

  The reduced data creation unit 11 creates reduced data by down-sampling the input data acquired by the input data acquisition unit 10. Here, down-sampling can be performed in an arbitrary manner and according to the nature of the data. If it is an integer ratio downsampling, it may be simply thinned out (a part of the data is deleted), and if it is a non-integer ratio downsampling, interpolation (eg linear interpolation) may be performed between the data. (For example, in the case of acoustic data, the frequency characteristics below the sampling frequency after down-sampling may be the same as the data before down-sampling).

  In addition to the input data, the reduced data creation unit 11 downsamples learning data prepared for machine learning of a filter for upsampling processing.

  It is assumed that a learning set including a plurality of pieces of learning data at the third sample rate is stored in the storage device 17 in advance. The learning set includes various types of learning data. The reduced data creation unit 11 reads the learning set from the storage device 17, reduces the learning data included in the learning set to downsampled to the second sample rate, and reduced downsampled to the first sample rate. Create data and.

  Hereinafter, the reduced data of the second sample rate and the reduced data of the second sample rate created from the learning data are also referred to as “learning data”. In addition, the learning set includes not only a plurality of learning images at the third sample rate but also a learning image at the second sample rate and a learning image at the first sample rate created from them.

  The machine learning unit 12 creates a filter for performing the upsampling process by machine learning using data included in the learning set. Note that the machine learning unit 12 hierarchically structures the data included in the learning set, and creates a filter in each hierarchy.

  FIG. 3 is a diagram illustrating an example of a learning set and filter having a hierarchical structure. In other words, the machine learning unit 12 creates learning sets and filters that are hierarchically structured in the following procedure.

  First, the machine learning unit 12 uses the second sample rate learning data and the third sample rate learning data included in the learning set to convert the second sample rate data into the third sample rate data. A CNN (Convolutional Neural Network) for upsampling processing is created. As shown in FIG. 2, the CNN that upsamples the data of the second sample rate into the data of the third sample rate is referred to as a first CNN or a first filter. The machine learning unit 12 uses the learning data of the second sample rate as the input of the CNN, the learning data of the third sample rate as the output of the CNN, and performs machine learning of the weights and biases in each layer of the CNN, thereby performing the first CNN. Create The first CNN created using the learning set is referred to as the first CNN0 as shown in FIG.

  In the present embodiment, the machine learning unit 12 will be described using CNN as an example of a filter for upsampling processing, but the filter is not limited to CNN, and machine learning is performed from a plurality of learning data. The present invention can also be applied to other filters.

  As a filter other than CNN, for example, an approximate curve having parameters such as moving average and polynomial approximation may be used, and the value of the parameter may be learned.

  Next, the machine learning unit 12 classifies the learning set into three subsets (learning subsets 1 to 3) using the first CNN0. For example, the machine learning unit 12 performs the classification based on the difference between the data up-sampled by the first CNN0 and the learning data.

  Specifically, the machine learning unit 12 creates data of the third sample rate by inputting each learning data of the second sample rate included in the learning set to the first CNN0.

  Next, the machine learning unit 12 calculates a difference between the third sample rate data created by the first CNN0 and the third sample rate learning data used to create the second sample rate learning data. calculate. The difference between the data can be obtained by, for example, a sum of squares or a sum of absolute values of differences between individual data values. In addition, you may use the absolute value of a cube sum, and another reference | standard.

  Here, the difference between the data does not necessarily have to be obtained based on the difference between the individual data values. For example, in the case of acoustic data, it is conceivable to obtain it based on a difference in frequency characteristics as a result of performing a Fourier transform by appropriately applying a window (for example, Hanning window). In short, a difference between data may be arbitrarily defined according to the purpose of use of the data.

  Next, the machine learning unit 12 classifies the learning data at the third sample rate into one of the three subsets based on the calculated difference.

  For example, the machine learning unit 12 classifies the learning data having the third sample rate whose difference is larger than the first threshold into the learning subset 1. In addition, the machine learning unit 12 uses learning data for learning at a third sample rate whose difference is greater than a second threshold (where the second threshold is smaller than the first threshold) and equal to or less than the first threshold. Classify into subset 2. Further, the machine learning unit 12 classifies the learning data having the third sample rate whose difference is equal to or smaller than the second threshold into the learning subset 3. The machine learning unit 12 also learns the second sample rate learning data and the first sample rate learning data created by the reduced data creation unit 11 from the third sample rate learning data. Classify into the same learning subset as the training data. Thereby, as shown in FIG. 3, the learning set is classified into learning subsets 1 to 3.

  Since the above difference indicates an error of the upsampled data, the learning set is classified into a plurality of learning subsets based on the error.

  The machine learning unit 12 uses each of the learning subsets 1 to 3 to perform a first CNN and a CNN (second CNN or second filter) that upsamples the first sample rate data to the second sample rate data. ).

  For example, the machine learning unit 12 uses the learning data of the second sample rate included in the learning subset 1 as the input of the CNN, and the learning data of the third sample rate included in the learning subset 1 as the output of the CNN. A first CNN is created by machine learning of weights and biases in each layer of the CNN.

  Further, the machine learning unit 12 uses the learning data of the first sample rate included in the learning subset 1 as the input of the CNN, and the learning data of the second sample rate included in the learning subset 1 as the output of the CNN. A second CNN is created by machine learning of weights and biases in each layer of the CNN.

  The machine learning unit 12 similarly creates the first CNN and the second CNN for the learning subsets 2 and 3 as well.

  The first CNN and the second CNN created from the learning subset i are described as the first CNNi and the second CNNi, respectively (i = 1 to 3).

  The machine learning unit 12 further classifies each of the learning subsets 1 to 3 into three subsets.

  For example, the machine learning unit 12 classifies the learning subset 1 into three subsets (learning subsets 1-1 to 1-3) using the first CNN1. The classification method is the same as that of the learning set except that the classification target is different. Therefore, detailed description thereof will not be repeated here.

  In addition, the machine learning unit 12 creates a first CNN and a second CNN for each of the learning subsets 1-1 to 1-3. The creation method of the CNN is the same as the creation method of the first CNN and the second CNN using the learning subset 1, except that the learning subset to be used is different. Therefore, detailed description thereof will not be repeated here.

  The machine learning unit 12 also performs learning subset classification processing and CNN creation processing using the classified learning subsets for the learning subsets 2 and 3.

  A learning subset obtained by classifying the learning subset i is referred to as a learning subset i-j (i = 1 to 3, j = 1 to 3). Further, the first CNN and the second CNN created from the learning subset i-j are referred to as a first CNNi-j and a second CNNi-j, respectively.

  By the processing executed by the machine learning unit 12, a learning set and a CNN classification tree having a resolution structure as shown in FIG. 3 are created. Assuming that the learning set and the first CNN0 are the zeroth hierarchy, the first hierarchy includes the learning set i, the first CNNi, and the second CNNi. The second hierarchy includes a learning set i-j, a first CNNi-j, and a second CNNi-j.

  The first hierarchical filter acquisition unit 13 to the upsampling processing unit 16 described below are processing units for upsampling input data.

  The first hierarchy filter acquisition unit 13 acquires the first CNN and the second CNN for each of the learning subsets 1 to 3 in the first hierarchy.

  The candidate data creation unit 14 uses the second CNNi obtained by the first hierarchical filter obtaining unit 13 to upsample the reduced data of the first sample rate obtained by down-sampling the input data, and creates data of the second sample rate. To do. The created second sample rate data is referred to as candidate data. In other words, the candidate data creation unit 14 creates three pieces of second sample rate candidate data by inputting the reduced data of the first sample rate to the second CNNi.

  The second hierarchy filter acquisition unit 15 acquires the first CNNi-j and the second CNNi-j machine-learned using the learning subset i-j for each of the plurality of learning subsets i-j in the second hierarchy. (I = 1-3, j = 1-3).

  The candidate data creation unit 14 further uses the second CNNi-j acquired by the second hierarchical filter acquisition unit 15 to upsample the reduced data of the first sample rate obtained by down-sampling the input data, thereby obtaining the second sample rate. Data (hereinafter referred to as “candidate data”). That is, the candidate data creation unit 14 creates candidate data for the second sample rate by inputting the reduced data for the first sample rate to the second CNNi-j.

  The upsample processing unit 16 calculates a difference between each of the three pieces of second sample rate candidate data created from the second CNNi by the candidate data creation unit 14 and the input data of the second sample rate. The method for calculating the difference between data is as described above.

  The upsampling processing unit 16 specifies the second CNNm used to create candidate data with the smallest calculated difference, and specifies the learning subset m used for machine learning of the specified second CNNm.

  The upsample processing unit 16 specifies a learning subset mj (j = 1 to 3) that is a subset of the specified learning subset m. For example, when m = 1, learning subsets 1-1 to 1-3 are specified.

  The up-sampling processing unit 16 further selects input data of the second sample rate from among the three candidate data created using the second CNNm-j of the learning subset m-j (j = 1 to 3). The candidate data with the smallest difference is identified.

  The up-sampling processing unit 16 uses the first CNNm-n corresponding to the second CNNm-n used to create the identified candidate data to upsample the second sample rate input data, thereby Create sample rate upsampled data.

The storage device 17 is a storage device for storing data and various types of data, and includes a hard disk drive (HDD), a nonvolatile memory, a volatile memory, or the like.
Next, a procedure of processing executed by the upsample processing apparatus 1 will be described.

  FIG. 4 is a flowchart showing the processing procedure of the filter creation process executed by the upsample processing apparatus 1 according to the embodiment of the present invention.

  As shown in FIG. 4, the reduced data creation unit 11 divides the data from the learning data of the third sample rate included in the learning set stored in the storage device 17 by 1/2 each of the vertical and horizontal data. The learning data of the second sample rate down-sampled to 1 and the learning data of the first sample rate down-sampled to 1/4 each of the vertical and horizontal are created (S1).

  The machine learning unit 12 converts the second sample rate data to the third sample rate data by machine learning based on the third sample rate learning data and the second sample rate reduced data included in the learning set. A first filter (first CNN0) for converting to (S2) is created.

  The machine learning unit 12 creates data of the third sample rate by inputting each learning data of the second sample rate included in the learning set to the first CNN0. Next, the machine learning unit 12 calculates a difference between the third sample rate data created by the first CNN0 and the third sample rate learning data used to create the second sample rate learning data. calculate. Based on the calculated difference, the machine learning unit 12 classifies the learning data at the third sample rate into one of the three learning subsets 1 to 3 (S3).

  The machine learning unit 12 creates the first CNNi and the second CNNi using the learning subset i (i = 1 to 3) (S4). The machine learning unit 12 stores the created CNN in the storage device 17. The machine learning unit 12 repeatedly executes machine learning while appropriately replacing the learning subset i (i = 1 to 3), so that the error with each learning subset is reduced. 2CNNi may be created.

  For each of the learning subsets 1 to 3, the machine learning unit 12 inputs each learning data of the second sample rate included in the learning subset i (i = 1 to 3) into the first CNNi, thereby Create sample rate data. Next, the machine learning unit 12 calculates a difference between the third sample rate data created by the first CNNi and the third sample rate learning data used to create the second sample rate learning data. calculate. The machine learning unit 12 classifies the learning data at the third sample rate into one of the three learning subsets ij (j = 1 to 3) based on the calculated difference (S5). Thereby, each of the learning subsets 1 to 3 is further classified into three learning subsets.

  The machine learning unit 12 creates the first CNNi-j and the second CNNi-j by machine learning using the learning subset ij (i = 1 to 3, j = 1 to 3) (S6). The machine learning unit 12 stores the created CNN in the storage device 17. Note that the machine learning unit 12 repeatedly performs machine learning while appropriately replacing the learning subsets ij (i = 1 to 3, j = 1 to 3), thereby causing an error from each learning subset. The first CNNi-j and the second CNNi-j may be created to be smaller.

  By the filter creation process shown in FIG. 4, a hierarchically structured CNN as shown in FIG. 3 is created.

  FIG. 5 is a flowchart showing the processing procedure of the input data upsampling process executed by the upsampling apparatus 1 according to the embodiment of the present invention.

  As shown in FIG. 5, the input data acquisition unit 10 acquires input data of the second sample rate that is the target of the upsampling process (S11).

  The reduced data creation unit 11 creates reduced data of the first sample rate by reducing the input data of the second sample rate acquired by the input data acquisition unit 10 (S12).

  The first hierarchy filter acquisition unit 13 acquires the first hierarchy filter of the classification tree shown in FIG. 3 (S13). That is, the first hierarchy filter acquisition unit 13 acquires the first CNN 1 to 3 and the second CNN 1 to 3 from the storage device 17.

  The candidate data creation unit 14 creates three second sample rate candidate data by inputting the reduced data of the first sample rate obtained by down-sampling the input data to the second CNN 1 to 3 (S14).

  The upsample processing unit 16 calculates a difference between each of the three pieces of candidate data of the second sample rate created in step S14 and the input data of the second sample rate (S15).

  The upsample processing unit 16 specifies the second CNNm used to create candidate data that minimizes the difference calculated in step S15 (S16). For example, the second CNN used to create candidate data with the smallest difference is specified as the second CNN1.

  The input to the second CNN 1 to 3 is not limited to the reduced data at the first sample rate. For example, a part of data cut out from the input data may be input to the second CNN 1 to 3. For example, when the input data is an image, the partial data may be created by cutting out small regions at equal intervals, or the partial data may be obtained by thinning out pixels of the input data at equal intervals. You may create it. Further, the partial data may be created by cutting out a part of the partial data from a plurality of data assumed to have the same properties as the input data. For example, when the input data is a part of the moving image data, a part of the data may be extracted from the moving image data of the same scene as the input data, and a part of the extracted data may be cut out.

  Further, the second CNNm may be specified in the same manner as described above from the difference between the candidate data created by the second CNN 1 to 3 based on the partial data and the partial data. By such processing, the time required to specify the second CNNm can be shortened.

  The second hierarchy filter acquisition unit 15 acquires a filter of the second hierarchy, which is a hierarchy lower than the second CNNm of the first hierarchy specified in step S16 (S17). That is, the second hierarchy filter acquisition unit 15 reads the first CNNm-j and the second CNNm-j in the classification tree shown in FIG. 3 from the storage device 17 (j = 1 to 3). For example, the second hierarchy filter acquisition unit 15 reads the first CNN 1-1 to 1-3 and the second CNN 1-1 to 1-3 from the storage device 17.

  The candidate data creation unit 14 inputs the reduced data of the first sample rate obtained by down-sampling the input data to the second CNNm-j (j = 1 to 3), thereby obtaining the three second sample rate candidate data. Create (S18).

  The upsample processing unit 16 calculates the difference between each of the three pieces of candidate data of the second sample rate created in step S18 and the input data of the second sample rate (S19).

  The upsample processing unit 16 identifies the second CNNm-n used to create candidate data that minimizes the difference calculated in step S19 (S20). For example, the second CNN used to create the candidate data with the smallest difference is specified as the second CNN 1-2.

  Note that the input to the second CNNm-j (j = 1 to 3) is not limited to the reduced data of the first sample rate. For example, a part of data cut out from the input data may be input to the second CNNm-j (j = 1 to 3). For example, when the input data is an image, the partial data may be created by cutting out small regions at equal intervals, or the partial data may be obtained by thinning out pixels of the input data at equal intervals. You may create it. Further, the partial data may be created by cutting out a part of the partial data from a plurality of data assumed to have the same properties as the input data. For example, when the input data is a part of the moving image data, a part of the data may be extracted from the moving image data of the same scene as the input data, and a part of the extracted data may be cut out.

  Further, the second CNNm-n is identified in the same manner as described above from the difference between the candidate data created by the second CNNm-j (j = 1 to 3) based on the partial data and the partial data. Also good. By such processing, the time required for specifying the second CNNm-n can be shortened.

  The up-sampling processing unit 16 performs the up-sampling process on the input data of the second sample rate using the first CNNm-n corresponding to the second CNNm-n of the second hierarchy specified in step S20, so that the third sample The rate super-resolution data is created (S21). In the above example, the upsampling process is performed using the first CNN 1-2.

  As described above, according to the processing performed by the upsample processing device 1, the second CNN that minimizes the error when the downsampled data of the first sample rate obtained by downsampling the input data of the second sample rate is minimized. The search can be made using the classification tree. The second CNN has the same properties as the first CNN, and the sample rate of the input data targeted by the second CNN is smaller than the sample rate of the input data targeted by the first CNN. For this reason, the second CNN can be searched efficiently. Further, by performing upsampling processing on input data using the first CNN corresponding to the searched second CNN, it is possible to create highly accurate upsampled data with few errors.

  That is, the first CNN and the second CNN paired with the first CNN are subjected to machine learning using the same learning data having different sizes. For this reason, it can be expected that data that can be accurately upsampled by the second CNN is also accurately upsampled by the first CNN paired with the second CNN. As a result, the first CNN paired with the second NN that minimizes the error is selected, and the input data is upsampled using the selected first CNN, thereby reducing the error when the input data is upsampled. I can expect.

  In addition, since the configuration of the second CNN is made smaller than that of the first CNN and the second CNN is selected and then the first CNN corresponding to the selected second CNN is selected, the processing speed of the selection of the first CNN is improved. be able to. However, the configuration of the second CNN is not necessarily smaller than the configuration of the first CNN.

  The learning set and the learning subset are each classified into three learning subsets, but the number of learning subsets to be classified is not limited to three, and may be two or more. Design according to the nature of the input data.

  Further, as shown in FIG. 3, the learning subset has two hierarchies, but may have one hierarchy or three or more hierarchies.

  In the above-described embodiment, the learning set and the learning subset are classified based on the difference between the upsampled data and the learning data. However, the classification method is not limited to this. . For example, if the type of data (in the case of an image, illustration, landscape photograph, portrait photograph, etc.) is known, the classification may be performed with further consideration thereof.

  Further, the first CNN and the second CNN have been described as receiving the entire data area as an input and performing the upsampling process. However, each CNN receives a plurality of block areas into which the data is divided as an input, and receives each block area. It may be configured to merge after upsampling. In addition, each block area may overlap with an adjacent block area, or the size may be different between the block areas. In this way, by configuring the CNN, it is possible to learn the CNN using only a part of the area without inputting the entire area of the learning data.

  The processing performed by the upsample processing apparatus 1 can be considered not a function of the apparatus but a method for causing the apparatus to function or a method for leaving the apparatus.

  It is also possible to implement these methods as a computer program realized by a computer. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  You may comprise as a computer system comprised from a microprocessor, ROM, RAM, a hard-disk drive, a display unit, a keyboard, a mouse | mouth, etc. A computer program is stored in the RAM or hard disk drive. The upsampling processing apparatus 1 achieves its functions by the microprocessor operating according to the computer program.

  Furthermore, some or all of the constituent elements constituting the super-resolution processing apparatus 1 may be configured as an IC card that can be attached to and detached from the up-sampling processing apparatus 1 or a single module. The IC card or module is a computer system that includes a microprocessor, ROM, RAM, and the like. The IC card or the module may include an LSI (Large Scale Integration: a large-scale integrated circuit: an ultra-multifunctional LSI manufactured by integrating a plurality of components on one chip). The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  Further, some or all of the constituent elements constituting the upsampling processing apparatus 1 may be configured by one system LSI. The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

(Image data processing)
Data to be processed by the upsample processing apparatus 1 can be image data such as illustrations and photographs. The data is pixel values (for example, three values of RGB) for a two-dimensional space in the vertical and horizontal directions. Thus, “data” is not limited to a numerical value of 1 (scalar value), but may be a numerical value of 2 or more (vector value).

  For example, input data is an image of 2K size (1920 × 1080 pixels: second sample rate), and an upsampling process is performed on the input data to create an image of 4K size (3840 × 2160 pixels: third sample rate). Can do. The reduced data may be QHD (Quarter High Definition) size (960 × 540 pixels: first sample rate). However, the size of the image targeted by the upsample processing apparatus 1 is not limited to these.

  Although the data size and filter have been described with reference to FIG. 2, rectangles representing reduced data, input data, and output data are shown in accordance with the above-described image size.

  The size of the input image is the same as the learning image included in the learning set, but the size of both is not necessarily the same, and the size of the learning image may be different from the size of the input image. Good.

(Data other than images)
The application target of the present invention is not limited to image data. For example, the acoustic data is a sound pressure value for a one-dimensional space of time. The moving image data is pixel values for three dimensions of the vertical direction, the horizontal direction, and time. Any of these data can be upsampled / downsampled and becomes input data of the upsample processing apparatus 1, and the upsampling method of the present invention can be applied.

  Similarly, a distribution map of measurement data such as medical scan data such as MRI (Magnetic Resonance Imaging) data and CT (Computed Tomography) data can be processed. This is because the same processing as in the case of an image is possible. Also, a vector structure, a three-dimensional structure such as a voxel image, and the like can be processed. That is, any data that can be processed by upsampling / downsampling becomes input data of the upsampling processing apparatus 1, and the upsampling method of the present invention can be applied.

  For data in a space of two or more dimensions, the ratio between the first sample rate and the second sample rate and the ratio between the second sample rate and the third sample rate depend on each dimension that defines the space. May be different. For example, in a moving image, processing may be performed separately for a sample rate (resolution of an image) for a two-dimensional space in the vertical direction and the horizontal direction, and a sample rate for another one-dimensional time (a so-called “frame rate”). Conceivable.

(Utilization of upsample processing equipment)
FIG. 6 is a diagram illustrating a computer system and devices that utilize the upsample processing apparatus.

  6A and 6B show a computer system. It is assumed that the upsample processing apparatus 1 is implemented as a computer program. The business computer program 21 provides input data to the upsample processing apparatus 1 and receives output data from the upsample processing apparatus 1. A computer system 2 is configured by combining the upsample processing apparatus 1 and the business computer program 21.

  Here, the up-sample processing apparatus 1 and the business computer program 21 may be implemented in one computer as shown in (A), or the up-sample processing apparatus 1 is used as a server as shown in (B). In addition, transmission / reception of input data and output data to / from the business computer program 21 may be performed via a network.

  The business computer program 21 may be a program for processing a still image or a moving image (for example, a program for performing rendering or CODEC processing), a program for processing sound (for example, a program for recording), or the like. It should be noted that “business” includes economic activities such as personal hobbies (“business” is understood as “a general act of performing some kind of processing”).

  6C and 6D show the device. The upsample processing apparatus 1 may be implemented as a computer program, or may be separately implemented as an IC card, LSI, or apparatus.

  In FIG. 6C, the upsample processing device 1 is mounted inside the device 3. For example, the device 3 includes the upsampling processing apparatus 1 as a computer program. The functional element 31 of the device 3 gives input data to the upsample processing apparatus 1 and receives output data from the upsample processing apparatus 1.

  In FIG. 6D, the upsample processing apparatus 1 is provided in a server, and input data and output data are transmitted to and received from the device 3 (the functional element 31 therein) via a network.

  Here, the server provided with the upsampling processing apparatus 1 may be common in FIGS. 6B and 6D. That is, one server can cope with both of FIGS. 6B and 6D.

  The device 3 (functional element 31) includes a television (image output to a monitor), a camera (storage of imaging data), a CT, an MRI, an X-ray, an echo, a radar and other measuring devices (display of measurement results), a smartphone ( Image output to a screen, sound output to a speaker), game machine (image output to a screen, sound output to a speaker), audio equipment (acoustic output to an amplifier or a speaker), and others can be considered.

  The embodiments disclosed in this specification should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the meanings described above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  The present invention is useful when used in an up-sample processing apparatus or the like that creates data in which the sampling rate of input data is increased in resolution.

1 Upsample Processing Device 10 Input Data Acquisition Unit 11 Reduced Data Creation Unit 12 Machine Learning Unit 13 First Hierarchy Filter Acquisition Unit 14 Candidate Data Creation Unit 15 Second Hierarchy Filter Acquisition Unit 16 Upsample Processing Unit 17 Storage Device 2 Computer System 21 Commercial computer program 3 Equipment 31 Functional elements

Claims (8)

For data expressing a value for a space of one dimension or two or more dimensions as a discrete value at a sampling point of the space,
An upsampling device for upsampling input data of a second sample rate to obtain output data of a third sample rate,
For each of the plurality of learning subsets in the first hierarchy, which is a subset of the learning set including the data of the third sample rate, the data of the second sample rate that has been machine-learned using the learning subset is the third sample. A first filter for upsampling to rate data;
A first layer filter acquisition unit for acquiring a second filter for up-sampling data of a first sample rate obtained by down-sampling data of the second sample rate to the second sample rate;
A candidate data creation unit that upsamples the data of the first sample rate and creates candidate data of the second sample rate by using the second filters obtained by the first hierarchy filter obtaining unit;
By up-sampling the input data using a first filter corresponding to a second filter that minimizes the difference between the candidate data created by the candidate data creation unit and the input data, a third sample rate An upsampling device comprising: an output upsampling unit that creates output data.   The upsampling apparatus according to claim 1, wherein the space is a three-dimensional space. For each of a plurality of learning subsets in the second hierarchy, which is a subset of the learning subset in the first hierarchy used for machine learning of the second filter with the smallest difference, using the learning subset A second hierarchical filter acquisition unit that acquires the first and second machine-learned filters;
The candidate data creation unit uses the second filters of the second hierarchy acquired by the second hierarchy filter acquisition unit to upsample the data of the first sample rate to create the candidate data,
The output upsampling unit corresponds to a first filter corresponding to a second filter that minimizes a difference between the candidate data created by the candidate data creation unit using the second filter of the second hierarchy and the input data. The upsampling apparatus according to claim 1 or 2, wherein output data at a third sample rate is created by upsampling the input data using a filter.   A computer program for causing a computer to operate as the upsampling apparatus according to any one of claims 1 to 3. It is provided with the upsampling device according to any one of claims 1 to 3 or the computer program according to claim 4 ,
A computer system , wherein the input data is transmitted to the upsampling apparatus or the computer program , and the output data is received and used. It is provided with the upsampling device according to any one of claims 1 to 3 or the computer program according to claim 4 ,
A device for transmitting the input data to the upsampling device or the computer program ,
A device characterized by being a television device, a camera, a measuring device, a smartphone, a game device, an audio device, and other devices. For data expressing a value for a space of one dimension or two or more dimensions as a discrete value at a sampling point of the space,
A method for causing an upsampling device or computer program to upsample input data at a second sample rate to obtain output data at a third sample rate, comprising:
For each of the plurality of learning subsets in the first hierarchy, which is a subset of the learning set including the data of the third sample rate, the data of the second sample rate that has been machine-learned using the learning subset is the third sample. Obtaining a first filter for upsampling to rate data;
Obtaining a second filter for up-sampling the data of the first sample rate down-sampled to the second sample rate to the second sample rate;
Using the second filters acquired by the first hierarchical filter acquisition unit to upsample the data of the first sample rate to create candidate data of the second sample rate;
Creating output data at a third sample rate by up-sampling the input data using a first filter corresponding to a second filter that minimizes the difference between the candidate data and the input data; An up-sampling method comprising: For each of a plurality of learning subsets in the second hierarchy, which is a subset of the learning subset in the first hierarchy used for machine learning of the second filter with the smallest difference, using the learning subset Obtaining a machine-learned first filter and a second filter;
The step of creating the candidate data up-samples the data of the first sample rate using the second filters of the second hierarchy acquired in the step of acquiring the second filter, and the candidate data Create
Step includes: a difference between the candidates data and the input data that was created using the second filter of the second hierarchy in the step of preparing the candidate data is minimized to create an output data of said third sample rate The upsampling method according to claim 7, wherein output data at a third sample rate is created by upsampling the input data using a first filter corresponding to the second filter.