JP2017059193A - Time series image compensation device, time series image generation method, and program for time series image compensation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically generate an image in time series and precisely compensate it.SOLUTION: A time series image compensation device 1, which generates a new image based on two or more images, reads a calculation model 50 in which two or more images in time series are taken as an input for outputting a new image. Elements of the two or more images 104 and 105 are read, and the two or more images that have been read are accepted as an input. A calculation that is based on the calculation model 50 that has been read is added to the elements of the two or more images having been accepted as the input, and a result of the calculation is outputted as a compensation image 106.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a time-series image complementing device, an intermediate image generating method, and a program for a time-series image complementing device that generate a time-series image using machine learning technology.

  Today, with the development and popularization of information technology, a large number of stored data are analyzed and utilized. In the field of machine learning technology, research on advanced discriminative and computational models, also called artificial intelligence, has been advanced beyond conventional simple statistical processing, and in recent years, remarkable progress has been made especially in image processing. It is accomplished.

  By the way, in general, a moving image is established using an illusion that a still image appears to be moving, that is, a so-called apparent motion, by continuously switching a plurality of still images and projecting them at high speed. For example, in animation, it is necessary to create around 24 still images in order to shoot a one-second moving image in order to show a motion to an observer.

  As an example, when considering television broadcasting, since a program is organized weekly for commercial purposes, about 36,000 still images are created every week for a 25-minute television program. . In principle, these images cannot be reused between other animations, so there is a problem that production costs are too high and efficiency is required.

  In order to deal with such a problem, an animation creation method that can produce an animation on a computer using an original image without manually creating a moving image has been disclosed.

Japanese Patent Publication No. 2000-172865

  According to Patent Document 1, a first hand-drawn original image is read, the original image is traced by a polygon or the like, color, transparency, texture, etc. are set to the polygon, and the subsequent i and subsequent images are set. An animation creation method is disclosed in which the first original image is read, the original image trace is transformed to create the original image trace, and the intermediate data is generated by interpolating the vertices of the traced polygon. Has been.

  However, since the technique disclosed in Patent Document 1 is performed by polygon tracing, there is a correspondence between polygons before and after deformation, particularly between lines before and after deformation, between original images to be complemented. If not done correctly, there remains a problem that an intermediate image cannot be generated correctly.

  Therefore, the inventor of the present invention uses a machine learning technique, in particular, a neural network, to construct a learning model that uses the complementary image as correct data from the images before and after the complementary image. We focused on the possibility of making it possible.

  In addition, the inventor of the present invention can perform up-conversion by using the above-described method to complement not only an animation that is an artificial image but also a general-purpose moving image in general and missing frames. Pay attention.

  Further, the inventors of the present invention have focused on the fact that a more advanced learning model can be constructed and a further improvement in accuracy can be achieved by constructing a learning model using a technique in the technical field called Deep Learning.

  The present invention relates to a time-series image complementing device that automatically generates and outputs a corresponding complementary image for two or more time-series images using a machine learning model obtained by learning a time-series image as correct answer data, A time-series image generation method and a program for a time-series image complementing apparatus are provided.

  The present invention provides the following solutions.

  The invention according to the first feature is a time-series image complementing device that generates a new image based on two or more images, wherein two or more images on the time-series are input and a new image is output. An arithmetic model reading means for reading the arithmetic model, an image reading means for reading the elements of the two or more images, an original image receiving means for receiving the two or more read images as input, and an operation based on the read arithmetic model There is provided a time-series image complementing device comprising: image element computing means for adding to two or more image elements received as input; and complementary image output means for outputting the result of the computation as a complemented image.

  According to the first aspect of the invention, the time-series image complementing device that generates a new image based on two or more images inputs two or more images on the time series and outputs a new image. Read the operation model, read the elements of the two or more images, accept the two or more read images as input, and add the operation based on the read operation model to the elements of the two or more images received as the input The result of the calculation is output as a complementary image.

  The invention according to the first feature is the category of the time-series image complementing device, but the same effect is obtained according to the category in the category of the time-series image complementing method and the program for the time-series image complementing device.

The invention according to the second feature is an intermediate image learning in which two or more images prepared in advance and a time-series intermediate image are input with the two or more images and the supervised learning is performed with the intermediate image as an output. Means, and
There is provided a time-series image complementing apparatus according to the first aspect of the invention, wherein the arithmetic model reading means reads an arithmetic model obtained as a result of the learning.

  According to the invention relating to the second feature, the time-series image complementing device which is the invention relating to the first feature relates to the two or more images prepared in advance and the two or more images and the intermediate images on the time-series. Supervised learning is performed using an image as input and the intermediate image as output, and the calculation model reading means reads a calculation model obtained as a result of the learning.

In the invention according to the third feature, with respect to two or more images prepared in advance and a preceding or succeeding image in time series, the two or more images are input, and the preceding or following image is output as a supervised teacher. And before and after image learning means for performing learning,
There is provided a time-series image complementing apparatus which is an invention according to a first feature characterized in that the predetermined computation is performed by a computation model obtained as a result of the learning.

  According to the invention relating to the third feature, the time-series image complementing device which is the invention relating to the first feature relates to the two or more images prepared in advance and the previous or subsequent image on the time series. Two or more images are input, supervised learning is performed using the previous or subsequent image as an output, and the predetermined calculation is performed according to a calculation model obtained as a result of the learning.

The invention according to a fourth feature comprises meta tag addition means for extracting and adding high-order information representing the feature of the image from the read image,
There is provided a time-series image complementing apparatus which is an invention according to any one of the first to third features, wherein the extracted high-order information is handled as an element of the read image.

  According to the fourth feature of the invention, the time-series image complementing device according to any one of the first to third features extracts high-order information representing the feature of the image from the read image. Then, the extracted higher-order information is treated as an element of the read image.

  An invention according to a fifth feature provides a time-series image complementing device according to any one of the first to fourth features, wherein the arithmetic model is expressed by a neural network.

  According to a fifth feature, in the time-series image complementing device according to any one of the first to fourth features, the calculation model is expressed by a neural network.

  The invention according to a sixth feature provides a time-series image complementing device according to any one of the first to fifth features, wherein the arithmetic model is expressed by a convolutional neural network.

  According to the sixth aspect of the invention, in the time-series image complementing apparatus according to any one of the first to fifth aspects, the operation model is expressed by a convolutional neural network.

  An invention according to a seventh feature provides a time-series image complementing device, which is an invention according to the fifth or sixth feature, wherein the learning is performed by deep learning.

  According to the seventh aspect of the invention, in the time-series image complementing apparatus which is the invention of the fifth or sixth aspect, the learning is performed by deep learning.

The invention according to an eighth feature comprises contour extracting means for extracting a contour in reading the image,
There is provided a time-series image complementing apparatus which is an invention according to any one of the first to seventh features, wherein the extracted contour is handled as an element of the image.

  According to the eighth aspect of the invention, the time-series image complementing apparatus according to any one of the first to seventh aspects of the present invention extracts a contour in reading the image, and the extracted contour is Treat as an element of the image.

  The invention according to a ninth feature includes an output image aggregating unit that, in the predetermined calculation, performs an operation using one or more calculation models, and outputs an image obtained by aggregating the calculation results as a final intermediate image. There is provided a time-series image complementing apparatus which is an invention according to any one of the first to eighth characteristics.

  According to the ninth aspect of the invention, the time-series image complementing apparatus according to any one of the first to eighth aspects of the present invention performs each of the predetermined calculations using one or more calculation models. Then, an image obtained by collecting the calculation results is output as a final intermediate image.

The invention according to a tenth feature is a time-series image complementing method executed by a time-series image complementing apparatus that generates a new image based on two or more images,
Reading an arithmetic model that takes two or more images in time series as input and outputs a new image;
Reading the elements of the two or more images;
Receiving the two or more read images as input;
Adding an operation based on the read operation model to two or more image elements received as the input;
Outputting the result of the calculation as a complementary image;
A time-series image complementing method is provided.

The invention according to the eleventh feature is a time-series image complementing device that generates a new image based on two or more images.
Reading an arithmetic model that takes two or more images in time series as input and outputs a new image;
Reading the elements of the two or more images;
Receiving two or more read images as input;
Adding an operation based on the read operation model to two or more image elements received as the input;
Outputting the result of the calculation as a complementary image;
A program for a time-series image complementing apparatus for executing the above is provided.

  A time-series image complementing device that automatically generates and outputs a corresponding complementary image for two or more time-series images using a machine learning model in which a time-series image is learned as correct data according to the present invention, It is possible to provide a time-series image generation method and a program for a time-series image complementing apparatus.

FIG. 1 is a diagram showing an outline of the time-series image complementing apparatus 1. FIG. 2 is a schematic diagram of a general neural network. FIG. 3 is a functional block diagram of the time-series image complementing apparatus 1. FIG. 4 is a flowchart of complementary image output processing executed by the time-series image complementing apparatus 1. FIG. 5 is a flowchart of the complementary image output model learning process executed by the time-series image complementing apparatus 1. FIG. 6 is an example of a time-series image that the time-series image complementing apparatus 1 learns and outputs.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. This is merely an example, and the technical scope of the present invention is not limited to this.

[Outline of time-series image complementing apparatus 1]
FIG. 1 is a diagram showing an outline of the time-series image complementing apparatus 1. In FIG. 1, in the machine learning model 50 that outputs a complementary image, the upper half area of the image represents the outline of learning, and the lower half area represents the outline of output.

  First, the outline of learning will be described. In FIG. 1, the model being learned is composed of a machine learning model 50, a learning data image 101, a learning data image 102, and a teacher data image 103.

  Here, as an example of learning in machine learning, an outline will be described using a neural network that is a representative model suitable for this purpose.

  FIG. 2 is a schematic diagram of a general neural network. The neural network can be said to be an operation model that adds a predetermined operation learned in advance to input data. Here, as an example, a neural network having a single intermediate layer is illustrated.

  The neural network includes an input layer such as node 201, an intermediate layer such as node 202, and an output layer such as node 203. The intermediate layer may be a multilayer, and a plurality of nodes may be interposed between the node 202 and the node 203. However, although the number of layers increases, the degree of freedom of the mathematical expression that can be expressed is improved, but the time and data amount required for learning increase.

  Edges 204 connecting nodes have weights independently. The weight of the edge 204 is multiplied by the numerical value input to the node 201 that is the starting point of the edge 204 and is passed to the node 202 that is the ending point. Since the node 202 is the end point of a plurality of edges, the sum of these values is passed as the final input to the node 202.

  For example, when the input 207 to the input layer is 2 and the weight of the edge 204 is 1.5, 3 is delivered to the node 202. Similarly, the nodes 205 and 206 located in the input layer also have independent inputs and edge weights, so the sum of the three numbers becomes the input to the node 202.

  In this way, the input to each node in the intermediate layer can be calculated. In addition, since the nodes of the intermediate layer and the nodes of the output layer are also connected by edges having independent weights, the input to each of the nodes of the output layer depends on the input to the intermediate layer and the weight of the edge. Will be calculated. This result is nothing but the output of the entire neural network.

  Here, since the number of layers and the number of nodes are given at the time of designing the neural network, it can be said that the act of learning is an act of optimizing the weight of each edge.

  Next, learning will be described. There are various learning methods depending on the machine learning model to be used, but a neural network generally uses a method called backpropagation. In learning, first, input data and an ideal value when the input data is input are prepared. This ideal value is called teacher data. The input data is sometimes called learning data.

  In the back propagation method, first, input data is input to a neural network to be learned, and an output at that time is obtained. Then, for each output node, the difference between the output and the ideal value is determined, and as the cause that caused the difference, the edge having the largest weight among the previous edges is determined, and the weight of the edge is determined as the ideal value. Adjust the direction so that it comes out. The steepest descent method is used for adjustment. When this adjustment is completed, the same adjustment is performed before that. By doing so, the weight adjustment is back-propagated from the output layer to the input layer.

  By repeating this adjustment for a large amount of teacher data, the weight of each edge is continuously adjusted so as to give a value close to the ideal value, and finally a neural network suitable for the purpose is obtained. Note that learning of a neural network does not necessarily converge, so that the completion of learning is practically determined by a threshold value based on accuracy or the amount of teacher data.

  When constructing a machine learning model using an image as an input as in the present case, a convolutional neural network is often used instead of a simple neural network. A convolutional neural network has a structure imitating a human visual cortex, and is known to contribute to improving accuracy particularly in the field of image processing.

  That is, in a simple neural network, an image is handled as one-dimensional data, and thus a parallel moved or rotated image is often separated as an element. The convolutional neural network can handle the input as it is in two dimensions, and thus has an advantage that it easily absorbs the image deformation.

  The above is the outline of the neural network.

  Returning to FIG. 1, the learning data image 101 is converted into numerical data called an element (feature) to be handled as an input of a machine learning model. The elements are arbitrarily selected according to the learning algorithm and learning purpose, and are not fixed. For example, one pixel at a time may be treated as an element, or if a pixel is converted to grayscale as an element, or a characteristic amount such as a gradient when a pixel is regarded as vector data is treated as an element. In some cases.

  As described above, machine learning is nothing but an act of optimizing an arithmetic algorithm by a combination of learning data and teacher data. In this example, the learning data is a combination of the elements of the learning data image 101 and the elements of the learning data image 102, and is input to the machine learning model 50 being learned (step S01). The teacher data image 103 is used for learning as teacher data (step S02). In addition, when the machine learning model 50 outputs secondary information that can restore an image instead of directly outputting an image, the secondary information becomes teacher data.

  The machine learning model 50 thus completed learning is used for output (step S03).

  In output, the input data image 104 and the input data image 105 are input to the machine learning model 50 as input data (step S04). At this time, each input data image is converted into an element and input as in the learning data 101 and the like.

  The machine learning model 50 performs the above-described calculation and outputs an output image 106 (step S05). The image thus obtained is an image output from the input data based on the relationship represented by the learning data and the teacher data in learning. That is, if learning is performed to complement the time-series intermediate images, an intermediate image of two input images can be obtained as an output.

  In this figure, the learning data 101 and the learning data 102 have a relationship in which the teacher data image 103 is sandwiched between them in the time series, but the learning data 101 and the learning data 102 are followed by the teacher data image 103. Even in such a relationship, learning is made accordingly, and there is no significant difference in algorithm. In that case, the output image also corresponds to the relationship. Note that selection of elements and selection of parameters in learning can be optimized according to the nature of the output to be obtained.

  The use of complementary images as described above can be used in a variety of ways. When a manually created time-series image such as an original animation is complemented, so-called intermediate images can be automatically output. Can make a significant contribution to cost reduction. In general, the frame rate is limited by the capacity and performance of moving images, but the so-called up-conversion function can generate intermediate images and increase the number of frames to enable smoother moving image playback. Can be provided.

  The above is the outline of the time-series image complementing apparatus 1.

[System configuration of time-series image complementing apparatus 1]
The time-series image complementing apparatus 1 is an apparatus that can operate in a stand-alone manner in principle. Note that when input data and teacher data are stored in an external storage or when parallel calculation is performed, the time-series image complementing apparatus 1 may be physically configured by a plurality of apparatuses.

  The time-series image complementing apparatus 1 is a household or business appliance having a function described later. The time-series image complementing apparatus 1 is, for example, an information home appliance such as a smartphone, a tablet terminal, a netbook terminal, a slate terminal, an electronic book terminal, a portable music player, in addition to a personal computer, a server device, a mobile phone, and a portable information terminal. It may be.

[Description of each function]
The configuration of each device will be described with reference to FIG.

  The time-series image complementing apparatus 1 includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and the like as the control unit 11, and the data controlled by the control unit as the input / output unit 12 In addition to a touch panel, a keyboard, a mouse, and the like that receive an input from a user or supporter, an input / output device that captures an image captured by an external device may be provided.

  The time-series image complementing apparatus 1 also includes a data storage unit such as a hard disk, a semiconductor memory, a recording medium, a memory card, or the like as the storage unit 14 that stores data and files.

  In the time-series image complementing apparatus 1, the control unit 11 reads a predetermined program, thereby realizing an image reading module 14, a complementary image output module 15, and an output image aggregation module 16 in cooperation with the input / output unit 12. Further, in the time-series image complementing apparatus 1, the control unit 11 reads a predetermined program, so that in cooperation with the storage unit 13, the calculation model reading module 17, the original image receiving module 18, the image element calculation module 19, and the intermediate An image learning module 20, a front-rear image learning module 21, a meta tag addition module 22, and a contour extraction module 23 are realized.

[Complementary image output processing]
FIG. 4 is a flowchart of complementary image output processing executed by the time-series image complementing apparatus 1. The processing performed by the module of each device described above will be described together with this processing.

  First, in order to learn the machine learning model 50 used for output, the time-series image complementing apparatus 1 performs a complementary image output model learning process.

[Complementary image output model learning process]
FIG. 5 is a flowchart of the complementary image output model learning process executed by the time-series image complementing apparatus 1. The processing performed by the module of each device described above will be described together with this processing.

  First, the image reading module 14 of the time-series image complementing apparatus 1 reads elements of the Nth time-series image (step S21). Here, the element is a numerical value when the image is converted into numerical data. For example, one pixel at a time may be treated as an element, or if a pixel is converted to grayscale as an element, or a characteristic amount such as a gradient when a pixel is regarded as vector data is treated as an element. In some cases. Which element is taken depends on the type and purpose of the machine learning model because the final accuracy may depend on how the element is taken.

  Next, the image reading module 14 of the time-series image complementing apparatus 1 reads the elements of the M-th time-series image as with the N-th (step S22).

  Here, for example, landscape images, portraits, animal drawings, live-action images, animations, or high-order information representing the contents of images such as morning, noon, and night are assigned to the image from which the elements are read, and processing is divided accordingly. As a result, it is possible to improve the accuracy of output in addition to calculations specialized for each type of image. Therefore, the meta tag addition module 21 of the time-series image complementing apparatus 1 adds higher-order information to the time-series image according to the read image and its elements (step S23).

  This higher-order information may be incorporated into the learning algorithm as an element of the image, or the machine learning model itself used for learning or output may be changed according to the higher-order information. In addition, a method for adding higher-order information may be set by a human on a rule basis, or a machine learning model for that purpose may be used. It can also be used together with higher order information manually added by a human.

  Next, the original image receiving module 18 of the time-series image complementing apparatus 1 receives the elements of the Nth and Mth time-series images as learning data (step S24).

  Then, the intermediate image learning module 20 or the front and rear image learning module 21 of the time-series image complementing apparatus 1 accepts the Lth time-series image as teacher data (step S25). Here, which module is used is determined by the context of L, M, and N. That is, if L is located between M and N, the intermediate image learning module 20 is used, and if not, the front and rear image learning module 21 is used. Note that the teacher data has the same dimensions as the final output, and therefore does not necessarily have to be treated as the same element as the learning data.

  Then, the intermediate image learning module 20 or the front and rear image learning module 21 performs learning on the machine learning model 50 using the received learning data and teacher data (step S26).

  FIG. 6 is an example of a time-series image that the time-series image complementing apparatus 1 learns and outputs. In FIG. 6, an image 601, an image 602, and an image 603 are continuous images in time series. Here, for the sake of explanation, it is assumed that time has passed in the order of image 601, image 602, and image 603, and learning a scene of waving using image 601 and image 603 as learning data and image 602 as teacher data. And

  In the image 601 to the image 603, a movement of a person shaking his / her hand is shown. That is, the arm 605 is listed in the image 601, whereas the arm 607 is swung down to an angle close to parallel to the ground in the image 603. Here, by using the image 601 and the image 603 as learning data and using the image 602 as teacher data, an arm 606 having an intermediate angle like the image 602 is in an intermediate state between the image 601 and the image 603. Let's learn to be.

  By learning a number of sets of scenes of swinging arms, such as a series of images, an image 602 is generated even when only the images 601 and 603 having no scene while hitting the image 602 are given. The model which can do is learned. Here, only the scene where the arm is shaken has been taken as an example, but by learning using a large amount of data, it is possible to generate a model that generates an intermediate image regardless of the type of movement.

  In the above example, an example in which an intermediate image is learned using a time-series image has been described. However, by using the image 601 and the image 602 as learning data and the image 603 as teacher data, the previous image in time series is used. It is also possible to learn an arithmetic model that uses two images to output a subsequent image in time series. It is also possible to learn an arithmetic model that outputs the previous image using the image 601 as teacher data.

  By the way, the learning method varies depending on the type of machine learning model, and if it is a neural network, the above-mentioned error back propagation method may be used, and it is also suitable for each machine learning model such as logistic regression, decision tree, and random forest. Learning methods may be used.

  Here, the design and learning of a model using a multilayer neural network having four or more layers among neural networks is called deep learning. As the number of layers of the neural network increases, the degree of freedom of expression is improved and a more complex model can be represented.

  The accuracy is determined using the above as a unit of learning (step S27). The definition of accuracy may be arbitrarily determined according to the output. Generally, when the difference between the teacher data and the output data is below a certain level (step S27: “YES”), it is assumed that sufficient learning has been performed. The process ends.

  If the accuracy is not yet sufficient (step S27: “NO”), it is determined whether there is a data set of learning data and teacher data not yet used for learning (step S28). If there is, the above learning is repeated (step S28: "YES"). However, if there is no more data set, further learning cannot be performed even if the accuracy is not sufficient, and learning is terminated (in the case of “NO” in step S28).

  In general, when the accuracy is measured with a data set used for learning, accuracy is increased in addition to unknown data. Therefore, it is common to prepare a test data set separately. Further, when measuring the accuracy of the machine learning model itself rather than the learning result, cross-validation using the same data separately for learning and testing is widely used. Further, there is a case where learning is unconditionally performed on all prepared data without setting a threshold for accuracy.

  Finally, the intermediate image learning module 20 or the front and rear image learning module 21 stores the result of the above learning as a learned model in the storage unit (step S29). This is because the learning process generally takes longer time than the output.

  The above is the processing procedure of the complementary image output model learning process. Returning to the complementary image output process, the arithmetic model reading module 17 of the complementary image output device 1 reads in order to use the learning model as described above (step S12). Here, “reading” refers to making the contents of the calculation as a learning result available.

  Next, the image reading module 14 of the time-series image complementing apparatus 1 reads elements of two or more time-series images that are the original images (step S13).

  Here, when reading an element, the contour extraction module 23 of the time-series image complementing apparatus 1 may extract the contour of the original image and treat it as the original image or an element of the original image (step S14). By doing so, while the image can be simplified and the accuracy can be improved, there is a commercial advantage that the output obtained by the middle division or the like is sufficiently valuable even if the output is a contour image. The contour extraction method may be a general method such as a canny method or Robert-cross.

  Further, the meta tag addition module 22 of the time-series image complementing apparatus 1 may add higher-order information indicating the type of image to the original image and handle it as an element of the original image (step S15).

  Next, the original image receiving module 18 of the time-series image complementing apparatus 1 receives two or more read time-series image elements as input data (step S16).

  Next, the image element calculation module 19 of the time-series image complementing apparatus 1 adds a calculation to the received input data using the read calculation model (step S17). In the example of the neural network described above, the calculation refers to obtaining specific data output as the output layer from the input layer via the intermediate layer.

  Next, the complementary image output module 15 of the time-series image complementing apparatus 1 outputs the calculation result as a complementary image (step S18). Here, in principle, it is assumed that the output data is pixel or vector data that can be expressed as an image as it is, but any format can be used as long as the result of calculation is vector data.

  When there are a plurality of learning models, the output image aggregation module 16 of the time-series image complementation apparatus 1 aggregates the results of the calculations by the models and outputs the result as a final complement image (step S19). This is a measure for improving accuracy by aggregating a plurality of outputs. The above is the processing procedure of the complementary image output process.

  The means and functions described above are realized by a computer (including a CPU, an information processing apparatus, and various terminals) reading and executing a predetermined program. The program is provided in a form recorded on a computer-readable recording medium such as a flexible disk, CD (CD-ROM, etc.), DVD (DVD-ROM, DVD-RAM, etc.), for example. In this case, the computer reads the program from the recording medium, transfers it to the internal storage device or the external storage device, stores it, and executes it. The program may be recorded in advance in a storage device (recording medium) such as a magnetic disk, an optical disk, or a magneto-optical disk, and provided from the storage device to a computer via a communication line.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to these embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

1 Time series image complement device, 50 machine learning model

Claims (11)

A time-series image complementing device that generates a new image based on two or more images,
Arithmetic model reading means for reading an arithmetic model that takes two or more images in time series as input and outputs a new image;
Image reading means for reading the elements of the two or more images;
Original image receiving means for receiving the two or more read images as input;
Image element calculation means for adding an operation based on the read operation model to two or more image elements received as the input;
Complementary image output means for outputting the result of the calculation as a complementary image;
A time-series image complementing device. Two or more images prepared in advance, and an intermediate image learning means for performing supervised learning using the two or more images as input and the intermediate image as an output for intermediate images on a time series,
The time series image complementing apparatus according to claim 1, wherein the arithmetic model reading unit reads an arithmetic model obtained as a result of the learning. Two or more images prepared in advance, and before or after the time-series images, input the two or more images, and before and after image learning means for performing supervised learning with the preceding or following images as outputs, With
The time-series image complementing apparatus according to claim 1, wherein the predetermined calculation is performed using an arithmetic model obtained as a result of the learning. Meta tag adding means for extracting and adding higher order information representing the characteristics of the image from the read image;
4. The time-series image complementing apparatus according to claim 1, wherein the extracted higher-order information is handled as an element of the read image. 5.   The time-series image complementing device according to claim 1, wherein the calculation model is expressed by a neural network.   The time-series image complementing apparatus according to claim 1, wherein the calculation model is expressed by a convolutional neural network.   The time-series image complementation apparatus according to claim 5, wherein the learning is performed by deep learning. A contour extracting means for extracting a contour in reading the image,
The time-series image complementing apparatus according to claim 1, wherein the extracted contour is handled as an element of the image.   An output image aggregating unit that performs each of the predetermined computations using one or more computation models and outputs an image obtained by aggregating the computation results as a final intermediate image. The time-series image complementing apparatus according to any one of claims 8 to 9. A time-series image complementing method executed by a time-series image complementing apparatus that generates a new image based on two or more images,
Reading an arithmetic model that takes two or more images in time series as input and outputs a new image;
Reading the elements of the two or more images;
Receiving the two or more read images as input;
Adding an operation based on the read operation model to two or more image elements received as the input;
Outputting the result of the calculation as a complementary image;
A time-series image complementing method comprising: In a time-series image complementing device that generates a new image based on two or more images,
Reading an arithmetic model that takes two or more images in time series as input and outputs a new image;
Reading the elements of the two or more images;
Receiving two or more read images as input;
Adding an operation based on the read operation model to two or more image elements received as the input;
Outputting the result of the calculation as a complementary image;
A program for a time-series image complementing apparatus for executing

Images