KR102198303B1 - Method and apparatus for training artificial neural network to generate slow motion image - Google Patents

Abstract

The present invention relates to a method for learning an artificial neural network for generating a slow motion image of a second speed faster than a first speed from an original image of a first speed. The method comprises the steps of: generating a learning data set including a start frame, an end frame, one or more intermediate frames, and a relative time of the one or more intermediate frames with respect to the start frame and the end frame; and learning to output a frame at each of one or more times between the time of the first frame and the time of the second frame.

Description

Learning method of artificial neural network to generate slow motion image {Method and apparatus for training artificial neural network to generate slow motion image}

An embodiment of the present invention relates to a method of learning an artificial neural network that generates a slow motion image of a second speed faster than the first speed from an original image having a first speed.

The concept of artificial intelligence first appeared in 1956 at the Dartmouth Conference held by Professor John McCarthy at Dartmouth University in the United States, and has been growing explosively in recent years.

In particular, since 2015, it is accelerating with the introduction of GPUs that provide fast and powerful parallel processing performance, and in the era of'big data', the explosively increasing storage capacity and data in all areas such as images, text, and mapping data overflowed. The advent also had a great influence on this growth.

Machine learning basically analyzes data using algorithms, learns through analysis, makes judgments or predictions based on what has been learned, and ultimately, rather than coding specific guidelines for decision criteria directly into software. , It aims to learn how to perform tasks by'learning' the computer itself through a large amount of data and algorithms.

Machine learning comes from concepts directly advocated by early artificial intelligence researchers, and algorithmic methods include decision tree learning, inductive logic programming, clustering, reinforcement learning, and Bayesian networks.

The present invention intends to naturally convert a low-speed image into a high-speed image using a learned artificial neural network.

In addition, the present invention intends to create only some images of content including a plurality of images using the learned artificial neural network, and generate the remaining images from the created images.

In addition, the present invention intends to make the transition between a plurality of images for one subject more natural by using the learned artificial neural network.

A method of generating a slow motion image of a second speed faster than the first speed from an original image of a first speed according to an embodiment of the present invention is, in which a viewpoint of the slow motion image from the original image Selecting a first frame and a second frame that is an end point of the slow-motion image; Generating at least one third frame describing a process of transitioning from the first frame to the second frame using the learned artificial neural network; And generating the slow motion image including the first frame, the at least one third frame, and the second frame.

At this time, the artificial neural network includes a view frame included in the training data, an endpoint frame included in the training data, and the at least one time between a view point and an end point, and the frame at each of the at least one view point. It may be a neural network that learns a correlation between one or more times and frames at each of the one or more times.

The method for generating a slow motion image may further include training the artificial neural network using training data before selecting the first frame and the second frame.

The training of the artificial neural network may include generating a sampling training image by sampling a training image having a third speed as an image having a fourth speed slower than the third speed; Selecting two frames included in the sampling training image as a view frame and an end frame, respectively; Extracting one or more intermediate frames between a frame corresponding to the viewpoint frame and a frame corresponding to the end point frame and a relative time of each of the one or more intermediate frames from the training image; And training the artificial neural network using the relative time of each of the view frame, the end frame, the one or more intermediate frames, and the one or more intermediate frames.

The relative time of each of the one or more intermediate frames may be set at equal intervals within a time period defined by the time of the viewpoint frame and the time of the end frame.

In the generating of the slow motion image, the slow motion image may be generated so that the at least one third frame is displayed at a frame rate corresponding to the first speed between the first frame and the second frame. .

The method for generating a slow motion image according to an embodiment of the present invention may further include determining the number of third frames to be generated prior to the step of generating the at least one third frame. In this case, generating the at least one third frame may generate the at least one third frame by referring to the quantity.

A method of generating content including M (M is a natural number, M>=N) images using sequential N (N is a natural number) images according to an embodiment of the present invention is included in the N images Selecting a first image and a second image adjacent to each other; Generating at least one third image describing a process of conversion from the first image to the second image using the learned artificial neural network; And the first image, the at least one third image, and the second image.

In this case, the artificial neural network includes a viewpoint image included in the training data, an endpoint image included in the training data, and the at least one time between a viewpoint and an end point, and the image at each of the at least one time, It may be a neural network that learns a correlation between one or more views and images at each of the one or more views.

In an artificial neural network learning method for generating a slow motion image according to an embodiment of the present invention, a view frame, an end point frame, one or more intermediate frames, and the view frame of the one or more intermediate frames and the relative time of the end frame are determined. Generating a training data set including, wherein the view frame, the end point frame, and the one or more intermediate frames are frames included in the same training image; And training the artificial neural network to output a frame at each of one or more times between the time of the first frame and the time of the second frame according to the input of the first frame and the second frame using the training data set. Step; may include.

The generating of the training data set may include generating a sampling training image by sampling the training image having a third speed as an image having a fourth speed slower than the third speed; Selecting two frames included in the sampling training image as the start frame and end frame, respectively; And extracting one or more intermediate frames between a frame corresponding to the viewpoint frame and a frame corresponding to the end point frame and a relative time of each of the one or more intermediate frames from the training image.

The artificial neural network is based on the learning data set in which at least one time between a viewpoint and an end point and a frame at each of the at least one time are marked, a viewpoint frame included in the learning data set, an end point frame, the one or more viewpoints, and It may be a neural network that has learned the correlation between frames at each of the one or more times.

According to the present invention, a low-speed image can be naturally converted into a high-speed image.

In addition, only some images of content including a plurality of images may be created, and the remaining images may be automatically generated from the created images.

In addition, it is possible to make the transition between various images on one subject more natural.

1 is a diagram schematically showing the configuration of a system according to an embodiment of the present invention.
2 is a diagram schematically showing the configuration of an image generating apparatus 110 provided in the image generating server 100 according to an embodiment of the present invention.
3 is a diagram for explaining an exemplary structure of an artificial neural network learned by the image generating device 110.
4 is a time-series diagram illustrating frames included in a training image having a third speed according to an embodiment of the present invention.
5 is a diagram illustrating frames included in a sampling training image at a fourth rate in time series.
6 is a diagram for explaining one generated learning data set 511.
7 is a diagram for describing a process of learning an artificial neural network 520 using the training data 510.
8 is a diagram for explaining a method of generating a slow motion image by the controller 112 according to an embodiment of the present invention.
9 is a diagram for explaining the generated slow motion.
10 is a diagram illustrating a method of generating a slow motion image performed by the image generating apparatus 110 according to an embodiment of the present invention.
11 is a diagram for describing images included in image content.
12 is a diagram for explaining a content generation method performed by the image generating apparatus 110 according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will be apparent with reference to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

In the following embodiments, terms such as first and second are not used in a limiting meaning, but are used for the purpose of distinguishing one component from another component. In the following examples, the singular expression includes the plural expression unless the context clearly indicates otherwise. In the following embodiments, terms such as include or have means that the features or elements described in the specification are present, and do not preclude the possibility of adding one or more other features or elements in advance. In the drawings, components may be exaggerated or reduced in size for convenience of description. For example, the size and shape of each component shown in the drawings are arbitrarily shown for convenience of description, and thus the present invention is not necessarily limited to what is shown.

1 is a diagram schematically showing the configuration of a system according to an embodiment of the present invention.

Referring to FIG. 1, a system according to an embodiment of the present invention may include an image generation server 100, a user terminal 200, a service server 300, and a communication network 400.

The system according to an embodiment of the present invention may generate a slow-motion image of a second speed faster than the first speed from an original image having a first speed using the learned artificial neural network.

In addition, the system according to another embodiment of the present invention may generate content including M (M>=N) images from N sequentially using the learned artificial neural network.

In the present invention, the'speed' of an image may be a value proportional to the number of frames included per unit time. Accordingly, in the present invention, an image with a relatively large number of frames included per unit time may be expressed as an image having a high speed, an image having a high frame rate, or a slow motion image. In addition, an image with a relatively small number of frames included per unit time may be expressed as a slow image or a low frame rate image.

In the present invention, the'playback speed' or'display speed' of an image is a concept that is distinguished from the'speed' of the image described above, and may mean the number of frames displayed per unit time when the image is reproduced. Therefore, despite the high speed of the video, the display speed (or playback speed) of the video may be slow, and the display speed (or playback speed) of the video may be high despite the slow speed of the video.

In the present invention, a'frame' may mean a scene, an image cut, and an image included in an image. Accordingly, in the present invention,'image' is an object composed of a set of a plurality of scenes (or image cuts, images), and may mean an object in which a plurality of scenes are arranged and displayed according to a predetermined time interval. For example,'video' may mean a video clip in which a frame rate is defined as a specific value.

In the present invention,'content' is an object consisting of a set of a plurality of scenes (or image cuts, images) similar to the above-described image, and unlike an image, it may mean an object without limitation of display time. For example,'content' may refer to web content including a plurality of images, and in which images are arranged in a virtual two-dimensional space (eg, a web page).

In the present invention, the'artificial neural network' is a neural network that has been trained appropriately for the generation of the aforementioned'image' or the aforementioned'content', and means a neural network that is learned by machine learning or deep learning. can do. The structure of such a neural network will be described later with reference to FIG. 3.

The user terminal 200 according to an embodiment of the present invention includes the user and the image generation server 100 and/or so that the user can use various services provided by the image generation server 100 and/or the service server 300. Alternatively, it may mean various types of devices that mediate the service server 300.

In other words, the user terminal 200 according to an embodiment of the present invention may mean various devices that transmit and receive data with the image generation server 100 and/or the service server 300.

As illustrated in FIG. 1, the user terminal 200 may refer to portable terminals 201, 202, and 203, or may refer to a computer 204.

Meanwhile, the user terminal 200 may include a display means for displaying content or the like in order to perform the above-described functions, and an input means for acquiring a user's input for such content. In this case, the input means and the display means may be configured in various ways. For example, the input means may include a keyboard, a mouse, a trackball, a microphone, a button, and a touch panel, but are not limited thereto.

The communication network 400 according to an embodiment of the present invention may mean a communication network that mediates data transmission/reception between components of the system. For example, the communication network 400 is wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, satellite communication, etc. It may cover wireless networks of, but the scope of the present invention is not limited thereto.

The service server 300 according to an embodiment of the present invention may refer to a device that provides a slow-motion image generation service or a device that provides a content generation service using an artificial neural network learned by the server 100. As shown in FIG. 1, the service server 300 may be a separate device separated from the image generation server 100, or integrally configured with the image generation server 100 (ie, the image generation server 100 ). It may be a device included in).

The server 100 according to an embodiment of the present invention may generate a slow-motion image of a second speed faster than the first speed from the original image of the first speed using the learned artificial neural network. In addition, the server 100 according to another embodiment of the present invention may generate content including M (M>=N) images from sequential N images using the learned artificial neural network.

2 is a diagram schematically showing the configuration of an image generating device 110 provided in the image generating server 100 according to an embodiment of the present invention.

Referring to FIG. 2, an image generating apparatus 110 according to an embodiment of the present invention may include a communication unit 111, a control unit 112, and a memory 113. In addition, although not shown in the drawings, the image generating apparatus 110 according to the present embodiment may further include an input/output unit, a program storage unit, and the like.

The communication unit 111 includes hardware required for the image generating device 110 to transmit and receive signals such as control signals or data signals through wired or wireless connection with other network devices such as the user terminal 200 and/or the service server 300, and It may be a device including software.

The controller 112 may include all types of devices capable of processing data, such as a processor. Here, the'processor' may refer to a data processing device embedded in hardware having a circuit physically structured to perform a function expressed by, for example, a code or command included in a program. As an example of such a data processing device built into the hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific integrated (ASIC) Circuit) and processing devices such as a Field Programmable Gate Array (FPGA) may be covered, but the scope of the present invention is not limited thereto.

The memory 113 temporarily or permanently stores data processed by the image generating device 110. The memory may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto. For example, the memory 113 may temporarily and/or permanently store data (eg, coefficients) constituting an artificial neural network. Of course, the memory 113 may also store training data for learning an artificial neural network. However, this is merely an example and the spirit of the present invention is not limited thereto.

3 is a diagram for explaining an exemplary structure of an artificial neural network learned by the image generating apparatus 110 of the present invention.

The neural network according to an embodiment of the present invention may be a neural network according to a convolutional neural network (CNN) model as shown in FIG. 3. At this time, the CNN model may be a hierarchical model used to finally extract features of input data by alternately performing a plurality of computational layers (Convolutional Layer, Pooling Layer).

The control unit 112 of the image generating apparatus 110 according to an embodiment of the present invention may build or train a neural network model by processing training data according to a supervised learning technique.

The control unit 112 according to an embodiment of the present invention creates a convolution layer for extracting a feature value of input data and a pooling layer constituting a feature map by combining the extracted feature values. can do.

In addition, the control unit 112 according to an embodiment of the present invention combines the generated feature maps to generate a fully connected layer that prepares to determine the probability that the input data corresponds to each of a plurality of items. I can.

Finally, the controller 112 may calculate an output layer including a probability that the input data corresponds to each of the plurality of items.

In the example shown in FIG. 3, the input data is image-type data (e.g., original image), divided into 5X7-type blocks, a 5X3-type unit block is used to generate the convolution layer, and 1X4 is used to create the pooling layer. Or, although it is shown that a 1X2 type unit block is used, this is exemplary and the spirit of the present invention is not limited thereto. Accordingly, the type of input data and/or the size of each block may be configured in various ways.

Meanwhile, the neural network may be stored in the above-described memory 113 in the form of coefficients of at least one node constituting the neural network, a weight of the node, and coefficients of a function defining a relationship between a plurality of layers constituting the neural network. Of course, the structure of the artificial neural network may also be stored in the memory 113 in the form of source codes and/or programs.

The type and/or structure of the neural network described in FIG. 3 is exemplary, and the spirit of the present invention is not limited thereto. Therefore, a neural network of various types may correspond to a'neural network' described through a specification.

In the present invention, the artificial neural network may be trained to generate a slow-motion image or may be trained to generate content including a plurality of images according to an embodiment. Hereinafter, a method for the controller 112 to generate a slow motion image will be described first, and a method for generating image content will be described later.

<How to create slow motion video>

The controller 112 according to an embodiment of the present invention may generate a slow motion image of a second speed faster than the first speed from the original image of the first speed using the learned artificial neural network.

To this end, the control unit 112 according to an embodiment of the present invention may train an artificial neural network. In this embodiment, the artificial neural network may be a view frame, an end point frame, a neural network that learns correlations between frames at one or more times, and at least one view point.

In this case,'start point' may mean a start point of an image,'end point' means an end point of an image, and'one or more times' may mean an arbitrary time between a view point and an end point. In addition, the frames at one or more times and at least one time may be used as cover data of the learning data.

With reference to FIGS. 4 to 7 together, a process in which the controller 112 learns an artificial neural network will be described in more detail.

FIG. 4 is a time-series diagram illustrating frames (Study Frame 1 to Study Frame 12) included in a learning image having a third speed according to an embodiment of the present invention. FIG. 5 is a diagram illustrating a time series of frames (Sampled Frame 1 to Sampled Frame 4) included in a sampling training image of a fourth rate. 6 is a diagram for explaining one generated learning data set 511. 7 is a diagram for describing a process of learning an artificial neural network 520 using the training data 510.

The controller 112 according to an embodiment of the present invention may generate a sampling learning image by sampling a learning image having a third speed as an image having a fourth speed slower than the third speed.

As described above, in the present invention, the'speed' of an image may be a value proportional to the number of frames included per unit time. Therefore, the training image having the third rate may include more frames within a unit time than the sampling training image having the fourth rate. For example, a training image having a third rate may include 12 frames within a unit time as shown in FIG. 4, and a sampling training image having a fourth rate includes 4 frames within a unit time as shown in FIG. can do.

Meanwhile, the frames included in the sampling training image may correspond to some of the frames included in the training image. For example, the first frame (Sampled Frame 1) of the sampling training image may correspond to the first frame (Study Frame 1) of the training image, and the second frame (Sampled Frame 2) of the sampling training image is the fourth frame of the training image. It may correspond to (Study Frame 4). Of course, the third frame (Sampled Frame 3) and the fourth frame (Sampled Frame 4) of the sampling training image may correspond to the seventh frame (Study Frame 7) and the tenth frame (Study Frame 10), respectively.

The controller 112 according to an embodiment of the present invention may select two frames from among frames included in the sampled training image as a view frame and an end frame, respectively. In this case, the view frame may mean the first frame of the image, and the end frame may mean the last frame of the image. For example, as illustrated in FIG. 6, the controller 112 may select a first frame (Sampled Frame 1) of the sampling training image as a view frame and a second frame (Sampled Frame 2) as an end point frame. However, such selection is exemplary, and the spirit of the present invention is not limited thereto, and the frame selection interval may be greater than one frame interval.

The controller 112 according to an embodiment of the present invention may extract one or more intermediate frames between a frame corresponding to a selected view frame and a frame corresponding to the selected end frame from the training image. For example, as shown in FIG. 6, when the first frame (Sampled Frame 1) of the sampling training image is selected as the view frame and the second frame (Sampled Frame 2) is selected as the end point frame, the controller 112 The frame corresponding to the frame (i.e., Study Frame 1) and the frame corresponding to the end frame (i.e., Study Frame 4) are identified, and one or more frames (i.e., Study Frame 2, Study Frame 3) between them are extracted from the training image. I can.

In addition, the controller 112 may extract a relative time of each of the extracted one or more intermediate frames. For example, as shown in FIG. 6, assuming that the viewpoint frame is relative time 0 and the end point frame is relative time 1, the control unit 112 can extract the relative times of the extracted two frames as 1/3 time intervals. have.

In an embodiment of the present invention, the controller 112 may set the relative time of each of one or more intermediate frames to equal intervals (for example, 1/3) within a time period defined as the time of the viewpoint frame and the time of the end frame. have. However, this is merely an example and the spirit of the present invention is not limited thereto.

The control unit 112 according to an embodiment of the present invention includes a selected view frame (Sampled Frame 1), a selected end point frame (Sampled Frame 2), extracted intermediate frames (Study Frames 2, 3), and relative time 511A between each frame. A training data set 511 including) may be generated.

The control unit 112 according to an embodiment of the present invention generates a plurality of training data sets according to the above-described process, generates training data 510 including the generated plurality of training data sets, and uses them to generate an artificial neural network. You can learn (520).

The control unit 112 according to an embodiment of the present invention trains the artificial neural network 520 using the training data 510, so that the artificial neural network 520 It can be trained to output a frame at each of one or more times.

During the learning process using the training data 510, coefficients of at least one node constituting the artificial neural network 520, the weight of the node, and coefficients of the function defining the relationship between the plurality of layers constituting the neural network are updated (or updated). ), learning can proceed.

Meanwhile, at least one time between the viewpoint and the end point and the frame at each time may be a frame having a predetermined correlation with the viewpoint frame and the end point frame.

Hereinafter, a method of generating a slow motion image will be described with reference to FIG. 8 on the premise that the artificial neural network is learned according to the above-described process.

The control unit 112 according to an embodiment of the present invention may select a first frame (Input frame 1) as a starting point of a slow-motion image and a second frame (Input frame 2) as an end point of a slow-motion image from the original image. have. In this case, the'original image' may refer to an image obtained in various ways as an image having a slower speed than a slow motion image. For example, the original image may refer to an image transmitted to the server 100 by the user of the user terminal 200 by using the user terminal 200 to generate a slow motion image. In addition, the original image may be a third party providing a slow motion service transmitting an image received from a customer terminal (not shown) for its own service to the server 100. However, this is only an example, and the original image is sufficient if the target image to be converted into a slow motion image.

Meanwhile, the first frame (Input frame 1) and the second frame (Input frame 2) may be selected by receiving a user input or may be selected according to a predetermined rule. In the case of selection based on the user's input, the user may select a start point and an end point of a section to be generated in slow motion within the original image and transmit it to the server 100. Meanwhile, when the first frame (Input frame 1) and the second frame (Input frame 2) are selected according to a predetermined rule, the control unit 112 determines the start and end points of the section to be generated in slow motion according to a preset rule. You can choose. For example, the controller 112 may select the first frame and the last frame of the original image as a first frame (Input frame 1) and a second frame (Input frame 2), respectively. However, this is merely an example and the spirit of the present invention is not limited thereto.

The control unit 112 according to an embodiment of the present invention describes at least a process of switching from the selected first frame (Input frame 1) to the selected second frame (Input frame 2) using the learned artificial neural network 520 One third frame (Output frame 1 to Output frame 3) may be generated, and a display time (relative view) of each of the third frames may be determined.

In another embodiment of the present invention, the control unit 112 may generate the third frame (Output frame 1 to Output frame 3) by further taking into account the third frame (Output frame 1 to Output frame 3) generation conditions (Conditions). May be. At this time, the conditions may include at least one of the number of generations of the third frame (eg, three) and the generation interval (eg, 1/4 relative time). In this case, the artificial neural network 520 may be trained further including a third frame generation condition.

As shown in FIG. 9, the controller 112 according to an embodiment of the present invention includes a first frame (Input Frame 1), at least one third frame (Output Frame 1 to Output Frame 3), and a second frame (Input Frame 1). A slow motion image 610 including Frame 2) may be generated.

At this time, the control unit 112 according to an embodiment of the present invention includes at least one third frame (Output Frame 1 to Output Frame 3) between the first frame (Input Frame 1) and the second frame (Input Frame 2). The slow motion image 610 may be generated to be displayed at a frame rate corresponding to the first speed.

For example, assume that the first speed, which is the speed of the original video, is one frame per unit time. In this case, only one frame may be displayed per unit time. Meanwhile, when three third frames are generated by the above-described process, four frames per current time, that is, one frame of the original image and the generated three third frames must be displayed to make the image natural (that is, the movement speed of the object in the image). May be displayed in the same way as the original image).

Meanwhile, the speed at which four frames per unit time are played (same speed as the second speed) may be faster than the first speed, which is the speed of the original video, and if four frames per unit time are played according to the second speed, the video Apart from this more natural connection, the motion of the object in the video is not described slowly like in slow motion.

Therefore, the control unit 112 according to an embodiment of the present invention allows the generated slow-motion image 610 to be displayed at a frame rate (or playback speed) corresponding to the first speed (that is, the speed of the original image) to switch between frames. At the same time, the motion of the object in the image can be slowly described. Accordingly, the present invention can provide a more lively image by generating a high-speed photographed image from an image photographed at a low speed.

10 is a diagram for explaining a method of generating a slow motion image performed by the image generating apparatus 110 according to an embodiment of the present invention. Hereinafter, descriptions of contents overlapping with those described in FIGS. 1 to 9 will be omitted, but will be described with reference to FIGS. 1 to 9.

The image generating apparatus 110 according to an embodiment of the present invention may train an artificial neural network. (S121) In this embodiment, the artificial neural network is a view frame, an end point frame, one or more time frames, and one or more time frames. It may be a neural network that has learned the correlation between the two.

In this case,'start point' may mean a start point of an image,'end point' means an end point of an image, and'one or more times' may mean an arbitrary time between a view point and an end point. In addition, the frames at one or more times and at least one time may be used as cover data of the learning data.

With reference to FIGS. 4 to 7 together, a process in which the image generating device 110 learns an artificial neural network will be described in more detail.

The image generating apparatus 110 according to an embodiment of the present invention may generate a sampling learning image by sampling a learning image having a third speed as an image having a fourth speed slower than the third speed.

As described above, in the present invention, the'speed' of an image may be a value proportional to the number of frames included per unit time. Therefore, the training image having the third rate may include more frames within a unit time than the sampling training image having the fourth rate. For example, a training image having a third rate may include 12 frames within a unit time as shown in FIG. 4, and a sampling training image having a fourth rate includes 4 frames within a unit time as shown in FIG. can do.

Meanwhile, the frames included in the sampling training image may correspond to some of the frames included in the training image. For example, the first frame (Sampled Frame 1) of the sampling training image may correspond to the first frame (Study Frame 1) of the training image, and the second frame (Sampled Frame 2) of the sampling training image is the fourth frame of the training image. It may correspond to (Study Frame 4). Of course, the third frame (Sampled Frame 3) and the fourth frame (Sampled Frame 4) of the sampling training image may correspond to the seventh frame (Study Frame 7) and the tenth frame (Study Frame 10), respectively.

The image generating apparatus 110 according to an embodiment of the present invention may select two frames from among frames included in the sampled training image as a view frame and an end frame, respectively. In this case, the view frame may mean the first frame of the image, and the end frame may mean the last frame of the image. For example, as illustrated in FIG. 6, the image generating apparatus 110 may select a first frame (Sampled Frame 1) of the sampling training image as a viewpoint frame and a second frame (Sampled Frame 2) as an end frame. However, such selection is exemplary, and the spirit of the present invention is not limited thereto, and the frame selection interval may be greater than one frame interval.

The image generating apparatus 110 according to an embodiment of the present invention may extract one or more intermediate frames between a frame corresponding to a selected view frame and a frame corresponding to the selected end frame from the training image. For example, as shown in FIG. 6, when the first frame (Sampled Frame 1) of the sampling training image is selected as the view frame and the second frame (Sampled Frame 2) is selected as the end point frame, the image generating apparatus 110 Identifies the frame corresponding to the view frame (i.e., Study Frame 1) and the frame (i.e., Study Frame 4) corresponding to the view frame, and one or more frames between them (i.e., Study Frame 2, Study Frame 3) from the training image. Can be extracted.

Also, the image generating device 110 may extract a relative time of each of the extracted one or more intermediate frames. For example, as shown in FIG. 6, assuming that the viewpoint frame is the relative time 0 and the end frame is the relative time 1, the image generating device 110 extracts the relative times of the extracted two frames as 1/3 time intervals. can do.

In an embodiment of the present invention, the image generating device 110 sets the relative time of each of one or more intermediate frames at equal intervals (for example, 1/3) within a time period defined as the time of the viewpoint frame and the time of the end frame. Can be set. However, this is merely an example and the spirit of the present invention is not limited thereto.

The image generating apparatus 110 according to an embodiment of the present invention includes a selected view frame (Sampled Frame 1), a selected end point frame (Sampled Frame 2), extracted intermediate frames (Study Frames 2, 3), and a relative time between each frame. A training data set 511 including 511A may be generated.

The image generating apparatus 110 according to an embodiment of the present invention generates a plurality of training data sets according to the above-described process, generates training data 510 including the generated plurality of training data sets, and uses them. The artificial neural network 520 may be trained.

The image generating apparatus 110 according to an embodiment of the present invention learns the artificial neural network 520 using the training data 510, so that the artificial neural network 520 is configured to provide a view and an end point according to the input of the view frame and the end frame. It can learn to output a frame at each of one or more times in between.

During the learning process using the training data 510, coefficients of at least one node constituting the artificial neural network 520, the weight of the node, and coefficients of the function defining the relationship between the plurality of layers constituting the neural network are updated (or updated). ), learning can proceed.

Meanwhile, at least one time between the viewpoint and the end point and the frame at each time may be a frame having a predetermined correlation with the viewpoint frame and the end point frame.

Hereinafter, a method of generating a slow motion image will be described with reference to FIG. 8 on the premise that the artificial neural network is learned according to the above-described process.

The image generating apparatus 110 according to an embodiment of the present invention includes a first frame (Input frame 1) as a viewpoint of a slow-motion image from an original image and a second frame (Input frame 2) as an end point of a slow-motion image. You can choose. (S122)

In this case, the'original image' may refer to an image obtained in various ways as an image having a slower speed than a slow motion image. For example, the original image may refer to an image transmitted to the server 100 by the user of the user terminal 200 by using the user terminal 200 to generate a slow motion image. In addition, the original image may be a third party providing a slow motion service transmitting an image received from a customer terminal (not shown) for its own service to the server 100. However, this is only an example, and the original image is sufficient if the target image to be converted into a slow motion image.

Meanwhile, the first frame (Input frame 1) and the second frame (Input frame 2) may be selected by receiving a user input or may be selected according to a predetermined rule. In the case of selection based on the user's input, the user may select a start point and an end point of a section to be generated in slow motion within the original image and transmit it to the server 100. On the other hand, when the first frame (Input frame 1) and the second frame (Input frame 2) are selected according to a predetermined rule, the image generating device 110 is the time point of the section to be generated in slow motion according to a preset rule. End point can be selected. For example, the image generating apparatus 110 may select the first frame and the last frame of the original image as a first frame (Input frame 1) and a second frame (Input frame 2), respectively. However, this is merely an example and the spirit of the present invention is not limited thereto.

The image generating apparatus 110 according to an embodiment of the present invention depicts a process of converting from the selected first frame (Input frame 1) to the selected second frame (Input frame 2) using the learned artificial neural network 520 At least one third frame (Output frame 1 to Output frame 3) may be generated, and a display time (relative view) of each of the third frames may be determined (S123).

In one embodiment of the present invention, the image generating device 110 further considers the third frame (Output frame 1 to Output frame 3) generation conditions (Conditions), and the third frame (Output frame 1 to Output frame 3). You can also create it. At this time, the conditions may include at least one of the number of generations of the third frame (eg, three) and the generation interval (eg, 1/4 relative time). In this case, the artificial neural network 520 may be trained further including a third frame generation condition.

The image generating apparatus 110 according to an embodiment of the present invention includes a first frame (Input Frame 1), at least one third frame (Output Frame 1 to Output Frame 3), and a second frame as shown in FIG. 9. A slow motion image 610 including (Input Frame 2) may be generated (S124).

At this time, the image generating apparatus 110 according to an embodiment of the present invention includes at least one third frame (Output Frame 1 to Output Frame 3) between the first frame (Input Frame 1) and the second frame (Input Frame 2). The slow motion image 610 may be generated so that) is displayed at a frame rate corresponding to the first speed.

For example, assume that the first speed, which is the speed of the original video, is one frame per unit time. In this case, only one frame may be displayed per unit time. Meanwhile, when three third frames are generated by the above-described process, four frames per current time, that is, one frame of the original image and the generated three third frames must be displayed to make the image natural (that is, the movement speed of the object in the image). May be displayed in the same way as the original image).

Meanwhile, the speed at which four frames per unit time are played (same speed as the second speed) may be faster than the first speed, which is the speed of the original video, and if four frames per unit time are played according to the second speed, the video Apart from this more natural connection, the motion of the object in the video is not described slowly like in slow motion.

Therefore, the image generating apparatus 110 according to an embodiment of the present invention allows the generated slow-motion image 610 to be displayed at a frame rate (or playback speed) corresponding to the first speed (that is, the speed of the original image), While allowing the transition between them to occur naturally, the movement of objects in the image can be slowly described. Accordingly, the present invention can provide a more lively image by generating a high-speed photographed image from an image photographed at a low speed.

<How to create content>

In the present invention,'content' is an object consisting of a set of a plurality of scenes (or image cuts, images) similar to the above-described image, and unlike an image, it may mean an object without limitation of display time. For example,'content' may refer to web content including a plurality of images, and in which images are arranged in a virtual two-dimensional space (eg, a web page).

Therefore, the content generation method according to an embodiment of the present invention may be substantially the same as the above-described slow-motion image generation method, except for the factor related to the playback speed.

The controller 112 according to an embodiment of the present invention may train an artificial neural network in the same manner as the artificial neural network learning method described in FIGS. 4 to 7. Therefore, the control unit 112 according to an embodiment of the present invention can generate training data according to the method described in FIGS. 4 to 6 and train the artificial neural network using the training data generated according to the method described in FIG. 7. have.

The control unit 112 according to an embodiment of the present invention uses the learned artificial neural network and N sequentially (N is a natural number) images to provide content including M (M is a natural number, M>=N) images. Can be generated.

Looking at this in more detail with reference to FIG. 11, the controller 112 according to an embodiment of the present invention may select a first image (Image 1) and a second image (Image 2) included in N images and adjacent to each other. have.

At this time, the'first image (Image 1)' is the'first frame' in the slow motion generation method, the'second image (Image 2)' is the'second frame' in the slow motion generation method, and the'third' The images (Images 3-1 and 3-2) may be concepts respectively corresponding to the “third frame” in the slow motion generation method. Meanwhile, in the present invention, when images are'adjacent', it may mean that identification numbers (or sequence numbers) assigned to each image are adjacent according to a predetermined rule.

In the present invention, the N images are images required for content generation, and may mean a part of a plurality of images constituting content generated by a user of the user terminal 200 to generate the entire content.

In other words, the user can complete the entire content more quickly by creating only images (ie, N images) for a large scene change, and (automatically) creating an image for a specific scene change process using the created image.

Meanwhile, the N images are images selected by the controller 112 according to a predetermined rule, and may refer to images used to generate content for representing one subject. In this case, the predetermined rule may be images selected based on the location information, the shooting (creation) time, the related event, or the similarity of the objects included in the image by referring to the metadata of the image. have. However, this is merely an example and the spirit of the present invention is not limited thereto.

The controller 112 according to an embodiment of the present invention uses at least one third image describing a process of converting from the first image (Image 1) to the second image (Image 2) using the learned artificial neural network. 3-1, 3-2) can be created.

At this time, the control unit 112 according to an embodiment of the present invention may generate a third image by further considering the third image generation condition. At this time, the conditions may include the number of generations (for example, two) of the third images (Images 3-1 and 3-2). In this case, the artificial neural network may be learned by further including a third image generation condition. However, this is merely an example and the spirit of the present invention is not limited thereto.

The control unit 112 according to an embodiment of the present invention provides content including a first image (Image 1), at least one third image (Image 3-1, 3-2), and a second image (Image 2). Can be generated. At this time, the control unit 112 according to an embodiment of the present invention includes a first image (Image 1), at least one third image (Image 3-1, 3-2), and a second image in a two-dimensional space defined as a web page. 2 You can create web content that sequentially lists images (Image 2).

Accordingly, according to the present invention, only an image for a large scene change is created, and an image for a specific scene change process is generated using the learned artificial neural network, thereby enabling the entire content to be completed more quickly.

12 is a diagram for explaining a content generation method performed by the image generating apparatus 110 according to an embodiment of the present invention. Hereinafter, descriptions of contents overlapping with those described in FIG. 11 will be omitted, but will be described with reference to FIG. 11.

The image generating device 110 according to an embodiment of the present invention may train an artificial neural network. (S131) The image generating device 110 according to an embodiment of the present invention is performed according to the method described in FIGS. 4 to 6. Accordingly, training data may be generated, and an artificial neural network may be trained using the training data generated according to the method described in FIG. 7. Detailed description of learning of artificial neural networks will be omitted.

The image generating apparatus 110 according to an embodiment of the present invention may select a first image (Image 1) and a second image (Image 2) included in N images and adjacent to each other. (S132)

At this time, the'first image (Image 1)' is the'first frame' in the slow motion generation method, the'second image (Image 2)' is the'second frame' in the slow motion generation method, and the'third' The images (Images 3-1 and 3-2) may be concepts respectively corresponding to the “third frame” in the slow motion generation method. Meanwhile, in the present invention, when images are'adjacent', it may mean that identification numbers (or sequence numbers) assigned to each image are adjacent according to a predetermined rule.

In the present invention, the N images are images required for content generation, and may mean a part of a plurality of images constituting content generated by a user of the user terminal 200 to generate the entire content.

In other words, the user can complete the entire content more quickly by creating only images (ie, N images) for a large scene change, and (automatically) creating an image for a specific scene change process using the created image.

Meanwhile, the N images are images selected by the image generating device 110 according to a predetermined rule, and may refer to images used to generate content for representing one subject. In this case, the predetermined rule may be images selected based on the location information, the shooting (creation) time, the related event, or the similarity of the objects included in the image by referring to the metadata of the image. have. However, this is merely an example and the spirit of the present invention is not limited thereto.

The image generating apparatus 110 according to an embodiment of the present invention uses at least one third image describing a process of converting from a first image (Image 1) to a second image (Image 2) using a learned artificial neural network. (Image 3-1, 3-2) can be created (S133)

At this time, the image generating apparatus 110 according to an embodiment of the present invention may generate a third image by further considering the third image generating condition. At this time, the conditions may include the number of generations (for example, two) of the third images (Images 3-1 and 3-2). In this case, the artificial neural network may be learned by further including a third image generation condition. However, this is merely an example and the spirit of the present invention is not limited thereto.

The image generating apparatus 110 according to an embodiment of the present invention includes a first image (Image 1), at least one third image (Image 3-1, 3-2), and a second image (Image 2). Content can be created (S134) At this time, the image generating device 110 according to an embodiment of the present invention includes a first image (Image 1) and at least one third image in a two-dimensional space defined as a web page. It is possible to create web content in which (Image 3-1, 3-2) and the second image (Image 2) are sequentially arranged.

Accordingly, according to the present invention, only an image for a large scene change is created, and an image for a specific scene change process is generated using the learned artificial neural network, thereby enabling the entire content to be completed more quickly.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may store a program executable by a computer. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And ROM, RAM, flash memory, and the like may be configured to store program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of the computer program may include not only machine language codes produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

The specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections. Connections, or circuit connections, may be represented. In addition, if there is no specific mention such as "essential", "important", etc., it may not be an essential component for the application of the present invention.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

100: server
110: image generating device
111: communication department
112: control unit
113: memory
200: user terminal
300: service server
400: communication network

Claims (3)

In the learning method of an artificial neural network that generates a slow motion image,
Generating a learning data set including a view frame, an end frame, at least one intermediate frame, and the view frame of the at least one intermediate frame and a relative time to the end frame, wherein the view frame, the end frame and the one The above intermediate frames are frames included in the same training image; And
Learning the artificial neural network to output a frame at each of one or more times between the time of the first frame and the time of the second frame according to the input of the first frame and the second frame using the training data set A learning method of an artificial neural network that generates a slow motion image, including. The method according to claim 1
Generating the training data set
Generating a sampling training image by sampling the training image having a third speed as an image having a fourth speed slower than the third speed;
Selecting two frames included in the sampling training image as the start frame and end frame, respectively; And
In the training image, extracting at least one intermediate frame between a frame corresponding to the view frame and a frame corresponding to the end frame and a relative time of each of the at least one intermediate frame; generating a slow motion image including Learning method of artificial neural network. The method according to claim 1
The artificial neural network is
A view frame included in the learning data set, an end point frame, the one or more times, and the one or more times, based on the learning data set in which one or more times between a view point and an end point and a frame at each of the one or more times are marked Learning method of artificial neural network that generates slow-motion image, which is a neural network that learns the correlation between frames in each.

Images