JP2021052414A - Image transmitting/receiving system, data transmitting/receiving system, transmitting/receiving method, computer program, image transmitting system, image receiving device, transmitting system, and receiving device - Google Patents

Abstract

To provide a system 1 for delivering a moving image content from a server 2-2 to a viewer terminal 11, which reduces a delivery capacity to reduce a load on a transmission path and improve an image quality for viewing.SOLUTION: Content data 31 based on an image encoded to a low bit rate and model data 32 for obtaining an image close to an original image 30 from a low bit rate encoded image, which is data of a conversion matrix in a neural network are distributed from a moving image content distribution server 2-2 to a viewer terminal 11, and the viewer terminal 11 can use these pieces of data to obtain an improved moving image content 33.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an image transmission / reception system, a data transmission / reception system, a transmission / reception method, a computer program, an image transmission system, an image reception device, a transmission system, and a reception device. The present invention relates to a configuration suitable for a moving image distribution system for distribution.

Viewers can use video content (also referred to as "image program", "program", and "content", which will also be used below) in combination with moving images or still images and audio as necessary from the distribution server. An image distribution system is used in which the image content is transmitted to the client terminal to be used and the viewer views the image content displayed on the display screen of the client terminal.

For example, in Patent Document 1 below, there are a plurality of terminal devices 400 connected to a streaming server 300 for distributing video data via a network 500 in paragraphs 1 and 0012 to 0016 and paragraphs 0032 to 0035. However, the configuration of a system capable of selecting desired video data from a plurality of video data according to a user's selection operation and receiving distribution is disclosed.

Japanese Patent No. 5956761 Publication of patent application Japanese Patent Application Laid-Open No. 2017-123649 Publication of patent application Japanese Patent Application Laid-Open No. 2017-49686 Publication of patent application Japanese Patent Application Laid-Open No. 2017-158067 Publication of patent application Japanese Patent Application Laid-Open No. 2015-201819

In particular, moving image content has a large amount of data to be transmitted, and when delivering moving image content via a communication path such as a wide area network including an Internet communication network from the content distribution server of the delivery source to the viewing terminal of the delivery destination, communication is performed. If the load on the road increases, the number of distribution destination terminals increases, or distribution is concentrated during a certain period of time, data congestion and distribution interruption may occur.

On the other hand, the video encoding system disclosed in Patent Document 2 below is for viewing video via an Internet communication network having only a limited bandwidth, as described in paragraphs 0024 to 0025. In systems that send and receive streaming, it is necessary to use efficient digital video encoding that can substantially reduce the data ratio of digital video signals for the purpose of video data compression. The encoder provided in the Patent Document 2 disclosure system first divides the video stream into a plurality of scenes, and for each scene, the scene types are, for example, "high-speed movement", "stationary", "talking head", "character", and so on. Determine one of "scroll credits", "almost black images", "short scenes with 5 or less image frames", etc., and pre-defined video encoding parameters (image coding parameters) for each scene type. ) Is used to output an encoded video stream.

On the other hand, as one option for band-compressing the moving image content with higher efficiency, there is a method of reducing the transmission rate (bit rate) for transmitting the moving image content data and transmitting the moving image content data as a small amount of data. However, with this method, the amount of data contained in the moving image content data is reduced, and the image quality is inferior, that is, the detail information is missing, and the image display including block noise and mosquito noise tends to be performed. The person (user) is dissatisfied.

On the other hand, although the configuration is not intended to be applied to a moving image content distribution system, such image data lacking details is modified to improve the resolution and generate an image closer to the original image. Several proposals have been made to do so, including those that utilize machine learning techniques, including deep learning.

For example, in Patent Document 3 below, in a technique for restoring a high-quality image from a low-quality image (referred to as "super-resolution technology"), the entire process first creates a dictionary database used for restoration. It is divided into a learning process, which is, and a restoration process, which restores a high-quality image from a low-quality image using this dictionary database (paragraph 0043). Then, in the learning process, a pair of a minute-sized high-resolution image derived from the same local region of the same learning image and a deteriorated image created by degrading the image quality of this high-resolution image is created. A patch image is cut out from a low-quality image, a learned minute-sized degraded image in a dictionary database similar to the patch image is identified, and a micro-sized high-resolution image paired with the degraded image is identified. It is said that it will use a learning-type super-resolution technology that restores images with high image quality by assembling them.

Further, in the following Patent Document 4, which also attempts to restore a high-resolution image using deep learning, more accurate monitoring can be performed in a scene where a plurality of types of shooting objects may appear. In order to provide a monitoring system (paragraph 0004), as described in paragraphs 0015, 0029 to 0041, super-resolution processing is performed using dictionary data 64 corresponding to the type of object, and super-resolution processing is performed. The dictionary data 64 including the coefficients required when executing the convolution operation in order to acquire the image processed image is, for example, a method such as deep learning by combining a large number of correct data, high resolution data and low resolution data. The latter-stage image processing unit 54 uses the dictionary data 64 generated by this learning to execute a convolution operation on the actually acquired image, and the high-resolution image (enlarged) is generated. Image) is to be acquired.

Further, Patent Document 5 below, which also attempts to restore a high-resolution image using deep learning, discloses a high-quality image system for deteriorated images recorded on an analog recording medium (video tape, film, etc.). ing.

However, in the configuration disclosed by each of these patent documents shown above, when a large amount of image data such as moving image content is distributed from a distribution source to a distribution destination as described above, a communication path or the like is used. We have not disclosed or even suggested a configuration for delivering moving image content that reduces the load on the screen and has an appropriate image quality.

As described above, the present invention provides video streaming for viewing moving image content via a transmission line such as an Internet communication network having only a limited bandwidth, which is a problem that each conventional technique has not yet solved. An image transmission / reception system and data that can efficiently perform transmission / reception system compression and image restoration with a resolution close to that of the original image while reducing the burden on the operator. It is an object of the present invention to provide a transmission / reception system, a transmission / reception method, a computer program, an image transmission system, an image reception device, a transmission system, and a reception device.

The present invention provides an image transmission / reception system, a data transmission / reception system, a transmission / reception method, a computer program, an image transmission system, an image reception device, a transmission system, and a reception device described in the following items in order to solve the above problems. To do.
1)
Machine learning of model data for at least one of a single or multiple transmitters to generate an improved image closer to the original image from a low bit rate encoded image that encodes the original image to a lower bit rate. Equipped with a machine learning unit generated by
At least one of a single or multiple transmitters comprises a transmitter that transmits low bit rate encoded images and model data to the outside of the device.
An image transmission / reception system, wherein the receiving device has an improved image generation unit that generates an improved image of the low bit rate encoded image from the received low bit rate encoded image and model data.
2)
The image transmission / reception system according to 1), wherein the data used for machine learning further includes meta information of a low bit rate encoded image.
3)
The meta information of the low bit rate encoded image is at least one of the coding block quantization parameter (QP), the prediction error coefficient, the prediction mode information, and the motion vector information in the image coding technique. The image transmission / reception system according to 2).
4)
Model data transmitted with the low bitrate encoded image by at least one or more transmitters, based on information about any of the low bitrate encoded images transmitted by the transmitter. The image transmission / reception system according to any one of 1) to 3), which has a model data selection unit for selecting from a plurality of models.
5)
Machine learning of model data for at least one or more transmitters to generate improved data closer to the original data from the low bit rate encoded data that encoded the original data to a lower bit rate. Equipped with a machine learning unit generated by
At least one of the single or plural transmitters comprises a transmitter that transmits low bit rate encoded data and model data to the outside of the device.
A data transmission / reception system, wherein the receiving device has an improved data generation unit that generates improved data of the low bit rate encoded data from the received low bit rate encoded data and model data.
6)
How to send and receive images
A model for a machine learning unit of at least one of a single or a plurality of transmitters to generate an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate. Steps to generate data by machine learning and
A step in which a transmitter of at least one of a single or a plurality of transmitters transmits a low bit rate encoded image and model data to the outside of the device.
A transmission / reception method, wherein the improved image generation unit of the receiving device includes a step of generating an improved image of the low bit rate encoded image from the received low bit rate encoded image and model data.
7)
The transmission / reception method according to 6), wherein the data used for machine learning further includes meta information of a low bit rate encoded image.
8)
The meta information of the low bit rate encoded image is at least one of the coding block quantization parameter (QP), the prediction error coefficient, the prediction mode information, and the motion vector information in the image coding technique. , 7).
9)
At least one of the single or plural transmitters further comprises the model data transmitted with the low bit rate encoded image based on the information about the low bit rate encoded image transmitted from the transmitter. The transmission / reception method according to any one of claims 6 to 8, further comprising a model data selection unit for selecting from a plurality of items.
10)
A model for a machine learning unit of at least one of a single or a plurality of transmitters to generate improved data closer to the original data from the low bit rate encoded data obtained by encoding the original data to a low bit rate. Steps to generate data by machine learning and
A step in which a transmitter of at least one of a single or a plurality of transmitters transmits low bit rate encoded data and model data to the outside of the device.
A transmission / reception method, wherein the improved data generation unit of the receiving device includes a step of generating improved data of the low bit rate encoded data from the received low bit rate encoded data and model data.
11)
A computer program for executing the transmission / reception method according to any one of 6) to 10).
12)
Model data for generating an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate provided in at least one of a single or a plurality of transmitters. , Machine learning unit generated by machine learning,
An image comprising a low bit rate encoded image and a transmitter for transmitting model data to the outside of the system, provided in at least one of a single or a plurality of transmitters. Transmission system.
13)
The image transmission system according to 12), wherein the data to be used for machine learning is meta information of a low bit rate encoded image.
14)
The meta information of the low bit rate converted image is at least one of a coded block quantization parameter (QP), a prediction error coefficient, a prediction mode information, and a motion vector information in an image coding technique. The image transmission system according to 13).
15)
Further, it is characterized by having a model data selection unit that selects from a plurality of model data to be transmitted together with the low bit rate encoded image based on the information about the low bit rate encoded image transmitted from the transmission unit. , 12) to 14). The image transmission system according to any one of the following items.
16)
Model data for generating improved data that is closer to the original data from the low bit rate encoded data that encodes the original data to a low bit rate, which is provided in at least one of one or more transmitters. , Machine learning unit generated by machine learning,
A transmission system including a transmission unit provided in at least one of a single or a plurality of transmission devices, which transmits low bit rate encoded data and model data to the outside of the device.
17)
Model data for generating an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate. Model data generated by machine learning and low bit rate encoding. A receiver that receives the completed image from the image transmission system,
An image receiving device including an improved image generation unit that generates an improved image of the low bit rate encoded image from the received low bit rate encoded image and model data.
18)
The image receiving device according to 17), wherein the data to be used for machine learning is meta information of a low bit rate encoded image.
19)
The meta information of the low bit rate encoded image is at least one of the coding block quantization parameter (QP), the prediction error coefficient, the prediction mode information, and the motion vector information in the image coding technique. , 18).
20)
The image receiving device according to 17), wherein the model data received by the receiving unit is selected from a plurality of models based on the information regarding the low bit rate encoded image received together.
21)
Model data for generating improved data closer to the original data from low bit rate encoded data obtained by encoding the original data to a low bit rate. Model data generated by machine learning and low bit rate encoding. A receiver that receives completed data from the transmission system,
A receiving device including an improved data generation unit that generates improved data of the low bit rate encoded data from the received low bit rate encoded data and model data.

By having the above-mentioned configuration, the present invention is efficient in a system for transmitting and receiving video streaming in order to view moving image contents via a transmission line such as an Internet communication network having only a limited bandwidth. Image transmission / reception system, data transmission / reception system, transmission / reception method, computer that can efficiently reduce the burden on the operator and efficiently perform compression of the transmission band and image restoration with a resolution close to that of the original image. A program, an image transmitting system, an image receiving device, a transmitting system, and a receiving device can be provided.

It is a conceptual diagram of the high image quality processing common to each embodiment of this invention. It is an overall block diagram of the moving image content distribution system which concerns on this invention. It is a conceptual diagram which shows the flow of the distribution signal in 1st Embodiment of this invention. It is a conceptual diagram of the neural network used by the 1st Embodiment of this invention. It is a figure which shows the structure of the model data creation server and the moving image content distribution server common to each embodiment of this invention. It is a figure which shows the structure of the 1st viewer terminal common to each embodiment of this invention. It is a schematic appearance diagram of the 1st viewer terminal common to each embodiment of this invention. It is a screen transition schematic diagram of the moving image distribution site common to each embodiment of this invention. It is a sequence chart of the high image quality processing in the 1st Embodiment of this invention. FIG. 5 is a flow chart for improving the image quality of an image, which is executed by the first viewer terminal according to the first embodiment of the present invention.

[Structure common to each embodiment of the present invention]
In each embodiment of the present invention, the content (program) distributed from the moving image content distribution server 2-2, particularly each image included in the moving image content, is an original image (cat) as shown in FIG. 1 (A). (Illustrated in the image of), but for the purpose of reducing the transmission capacity, an image encoded to a low bit rate as shown in FIG. 1 (B) (also illustrated in an image encoded to a low bit rate of a cat). ) Is generated, and the moving image content for transmission consisting of each low bit rate image is distributed to the viewer terminals 11, 12, and 13.

Each of the distributed viewer terminals 11, 12, and 13 visually approaches the original image as shown in FIG. 1 (C) by the respective configurations and methods described in the following embodiments. Images (also referred to as "high image quality") are generated (also exemplified by high image quality images of cats), and each of these high image quality images is collected to improve high image quality moving image content. Is generated and used for viewing by the viewer.

Therefore, as a configuration common to each embodiment of the present invention, as shown in FIG. 2, the moving image content distribution system 1 is a model data creation server 2-1 realized by a server computer or the like, and also a server computer. A personal computer that connects a signal between the moving image content distribution server 2-2 realized by the above and the servers 2-1 and 2-2 via a transmission path 3 exemplified by an Internet communication network or the like. It includes a first viewer 11, a second viewer terminal 12, a third viewer terminal 13, and the like, which are realized by a smartphone or a mobile information terminal. The number of viewer terminals in the implementation is not limited to the above example. Further, in each of the following explanations, the first viewer terminal 11 will be described as a representative of the viewer terminal, but the configuration and operation are the same in other viewer terminals.

Here, if a qualitative explanation is given about the "high image quality" that the present invention intends to carry out, or the generation of an image that is visually closer to the original image, in the prior art, the number of pixels is simply increased, or analog. Whereas the configuration only removes noise, the embodiment of the present invention is characterized in that a low bit rate moving image is converted into an image that humans feel as if it were a decoded image of a high bit rate moving image. .. Further, the high image quality in the present invention may include not only the spatial high image quality processing in the still image but also the time high image quality processing in the moving image.

[First Embodiment / Outline]
Hereinafter, the moving image content distribution system 1 which is the first embodiment according to the present invention will be described with reference to the drawings of FIGS. 1 to 10. Not limited to this embodiment, each embodiment described in the present specification is merely an example of the embodiment of the present invention, and various modifications and combinations with other techniques are possible, and these are also included in the present invention. included.

The system 1 of the present embodiment is based on the configuration of FIG. 2 described above, and further, as shown in FIG. 3, the model data creation server 2-1 is distributed (transmitted) by the first viewer terminal 11. It is machine-learned model data using a low-bit rate image, which is input data for machine learning, and an original image before the low bit rate, which corresponds to the moving image content that is desired. The conversion matrices Q and R that do

When the video content distribution server 2-2 receives a distribution request for the video content that the user wants to distribute from the first viewer terminal 11 (step S1 in FIG. 9), the video content distribution server 2-2 receives the video content distribution server 2-2. First, each original image of the driving image content requested to be distributed, including the original image 30, is encoded to a low bit rate, and each low bit rate encoded image 31 generated by the processing is collected. Generate low bit rate video content. Alternatively, low bit rate encoded content for these contents may be created before receiving the content distribution request.

Next, the moving image content distribution server 2-2 is model data in machine learning suitable for improving the image quality of the moving image content requested to be distributed by machine learning, for example, the conversion matrices Q and R in the neural network technology. Distribution is requested to the model data creation server 2-1 (step S2 in FIG. 9), and the moving image content for which distribution is requested together with the conversion matrices Q and R32 which are machine-learned model data obtained in response to the request. The moving image content 31 having a low bit rate is transmitted to the first viewer terminal 11 via the transmission line 3 (FIGS. 9 steps S3 and S4).

The first viewer terminal 11 that received the distribution visually approached the original image using the machine-learned model data 32 by the operation and method described below for each low bit rate encoded image 31. The image 33 is generated, and each of the high-quality images is collected to generate a moving image content having an improved sense of resolution, which is provided for viewing by the viewer.

[Acquisition of model data using machine learning]
In this embodiment, in machine learning, a method of obtaining an optimum model using teacher data is used when obtaining a multidimensional output from a multidimensional input using a neural network.

It should be noted that the application of machine learning using these neural networks is only an example, and it is also possible to perform high image quality processing using other machine learning methods, and such a configuration is also included in the present invention. ..

As shown in FIG. 4, which is a conceptual diagram of machine learning for improving image quality using a neural network used in the first embodiment of the present invention, a low bit rate encoded image is used as input data in the neural network technology. For example, for a target frame image, input data parameter 1, input data parameter 2, ..., Which are a plurality of (m) parameters that are pixel values (brightness, color tone) in a plurality of sample pixels. -, Each has an input data parameter m as a concrete numerical value, while as a teacher data (output data) in the neural network technology, similarly, for the original image, for example, for the target frame image. Specific examples of the teacher data parameter 1, the teacher data parameter 2, ..., The teacher data parameter d, which are a plurality of (d) parameters that are pixel values (brightness, color tone) in a plurality of sample pixels. I have it as a numerical value. The set of parameters for each of these input data and teacher data (output data) may be referred to as a "parameter vector" below. Further, each parameter of the input data and each parameter of the output data (teacher data) may partially or completely overlap.
The input data parameter vector w (Equation (1)) for the low bit rate image described above forms the input layer (m dimension) 41, and the teacher data parameter for the original image also described above. The output data parameter vector x (Equation (2)), which is d-dimensional like the vector β, forms the output layer 43.

A k-dimensional vector y (formula (3), also referred to as intermediate data) forms an intermediate layer 42 between the input layer 41 and the output layer 43.

The data of the input layer 41 is converted into the intermediate layer 42 by the linear transformation by the transformation matrix Q, and the data of the intermediate layer 42 is linearly transformed by another transformation matrix R and output as the data of the output layer 43. Inside each layer, there is no connection between each data and they are independent.

As described above, instead of directly converting the input data parameter vector w to the output data parameter vector x, a two-step conversion is performed as shown in the equation (4).

In equation (4), Q and R are matrices representing the linear transformation described above. Then, after performing the respective linear transformations Q and R, the transformation is performed for each variable by a non-linear function. The function is called an activation function, and in this embodiment, the logistic sigmoid function σ (a) shown in the equation (5) is used.

Using this logistic sigmoid function σ (a), the transformation of each data described above is expressed in four steps as in Eq. (6).

At the time of learning, the teacher data t (Equation (7)), which is the pixel value of the original image, which is the target data of the output variable, is given in advance. Then, each parameter of the neural network is determined by performing the following "estimation" so that the output value is close to the teacher data t.

Now, when the matrix of k rows and m columns that converts the input data parameter vector w into the variable vector y representing the intermediate layer 42 is represented by Q = [q _hj ] (q _hj is an element of h rows and j columns), Y = Qw, and when expressed in terms of elements, it is as shown in equation (8).

Further, the variable vector y transformed according to the equation (8) is non-linearly transformed as in the equation (9) by the logistic sigmoid function σ (a) described above.

Similarly, the variable vector α from the intermediate layer 42 is used as the variable vector x in the output layer, and the matrix R = [ _rih ] of _{d rows and k columns (rih} is an element of i rows and h columns), and x = Convert to Rα. Expressed as an element, it becomes as shown in equation (10).

Similar to the transformation in the intermediate layer 42, the transformed variable vector x is further transformed by the logistic sigmoid function σ (a) as shown in equation (11).

Next, we move on to the process of estimating the two matrices Q and R, which is the learning process. For this estimation, the present embodiment uses a method called the error back propagation method described below.

That is, first, the error between the teacher data t, which is a parameter in the original image, and the output β is calculated, and the amount for changing the transformation matrix of the intermediate layer 42 and the output layer 43 is obtained using the error. Next, the amount that changes the transformation matrix of the input layer 41 and the intermediate layer 42 is obtained. In estimating the element parameters of each transformation matrix, the estimation is performed to minimize the sum of squares of the error, but since the nonlinear transformation is included in the middle, the stochastic gradient descent method is used. This is a method of changing the element parameters of the matrix by an amount proportional to the gradient of the error so as to reduce the sum of squares of the error for each sample of the training data.

Since each element of the transformation matrix Q and R could be estimated according to each of the above processes, when the learning process is completed and the low bit rate image to be converted is given, each of the low bit rate images is given. The parameters (pixel values representing the brightness and color tone of each pixel, each parameter in the image coding technique, or other parameters) are converted according to the equation (6), and the output data vector x By obtaining, parameters for drawing a high-quality image can be obtained.

[Configuration of model data creation server 2-1 and video content distribution server 2-2]
As shown in the configuration diagram in FIG. 5 (1), the model data creation server 2-1 included in the system 1 of the present embodiment is realized by a computer for a server or the like, and data connection between inside and outside the server is established. Program storage that readablely stores the execution programs executed by the input / output interface 2-1a, the control unit 2-1b, which is the CPU (central processing unit) that controls the server 2-1 and the server 2-1. Part 2-1f, the input data and the teacher data for use in machine learning based on the neural network described above, are used as, for example, a low bit rate image of moving image content data for each category and an original image. Alternatively, the machine learning content recording unit 2-1g, which is recorded in another mode, and the machine learning unit 2-1h, which estimates the conversion matrices Q and R, which is machine learning based on the neural network described above. , A bus 2-1i for connecting data between each configuration in the server 2-1 is provided.

Further, as shown in FIG. 5 (2), the moving image content distribution server 2-2 is realized by a server computer or the like, and controls the input / output of information communication to and from the outside of the server 2-2. It includes an input / output interface 2-2a, a control unit 2-2b that controls and controls the entire server 2-2, and a content recording unit 2-2c that records and stores moving image contents to be distributed. The content handled by the server 2-2 is not limited to the moving image content, but may be content having other specifications such as still image content and audio content, or a combination of these various contents.

In addition, the content recording unit 2-2c records the "comment", which is the text data posted by the viewer for each content, together with the playback time (information of the time measured from the beginning of the content) of the posting. ing.

Further, the server 2-2 receives the request communication from the outside and sends the video content to the viewer terminal 11 or the like that made the request in response to the request, the content distribution unit 2-2d, the server 2-2. The program storage unit 2-2f, which stores the computer program to be executed by the server, records information about the viewer or the viewer terminal, such as whether the viewer terminal requesting the content distribution is a member of a video distribution site, for example. It is provided with a user management unit 2-2g for management and a bus 2-2i for communicating and connecting each configuration in the server 2-2.

As described above, the moving image content distribution server 2-2 distributes the moving image content, while the model data creation server 2-1 which is another server performs machine learning to generate the model data. The configuration is only an example, and it is not necessary to limit the configuration to this configuration. That is, in carrying out the present invention, one or a plurality of servers, that is, transmission devices 2-1 and 2-2 are provided in the system 1, and one of these servers has a configuration for distributing moving image contents. However, it is also possible to have one of these servers configured to perform machine learning to generate model data. Further, the configuration provided on the server side in the system 1 of the present invention is not limited to the configuration for performing machine learning and the configuration for distributing moving image contents, and the configuration provided on the server side is at least one or a plurality of servers, that is, a transmission device. It may be provided in either one, or similarly, each configuration provided on the viewer terminal side may be distributed to a plurality of viewer terminals. That is, it can be said that a single or a plurality of servers, that is, a transmitting device constitutes a transmitting system, and similarly, a single or a plurality of viewer terminals, that is, a receiving device constitutes a receiving system. .. These configurations are the same in other embodiments of the present invention.

[Structure of the first viewer terminal 11]
Hereinafter, the configuration of the first viewer terminal 11 will be described, but the second viewer terminal 12 and the third viewer terminal 13 also have the same configuration.

As shown in FIG. 6, the first viewer terminal 11 is a terminal device used by a viewer, which is realized by a personal computer, a smartphone, a mobile information terminal, or the like, and controls an input / output interface inside and outside the terminal. Output interface 11a, control unit 11b that controls and controls the entire terminal, image restoration unit 11c that restores a low-bit rate encoded image to a high-quality image using a machine-learned model, content of moving image content Is realized by the display unit 11f realized by the liquid crystal screen and its control unit, the keyboard, the mouse, etc., which displays the operation screen and the like of the moving image site, and the viewer operates the viewer terminal 11. The operation unit 11g used for this purpose, the program storage unit 11h for storing the computer program running on the terminal 11, the moving image content by the low bit rate image received from the server 2-2, or the resolution restored by the image restoration unit is improved. A data recording unit 11i that records moving image contents and the like based on the images, a comment posting unit 11k for posting comments to the moving image content distribution server 2-2, and each configuration inside the terminal 11, as will be described later. Each has a bus 11m for communication connection between them.

FIG. 7 schematically shows the appearance of the first viewer terminal 11. The terminal 11 has a display panel 11-1, a mouse cursor 11-2 displayed in the display panel 11-1, and a mouse 11. -3, keyboard 11-4 is provided.

FIG. 7 shows a situation in which a certain moving image content is reproduced and displayed. A moving image display screen 11-1a is displayed on the display panel 11-1, and a person 11-1b is displayed as the content of the moving image content. , Trees 11-1n, Houses 11-1o are displayed.

Further, the comment "good weather" 11-1r and "fast running www" 11-1r are displayed on the display panel 11-1, and the comment 11-1r creates the moving image content and distributes the moving image content. Viewers or other viewers using the first viewer terminal 11 who saw this content, not created by the poster (or also referred to as the "distributor" for convenience) who posted to the server 2-2. Is the text information posted to the video content distribution server 2-2 at any time during playback, so that the viewer can clearly understand that it is different from the original content. A part of the moving image display screen 11-1a is projected so as to be displayed outside.

Similarly, on the display panel 11-1, the display is switched to the portal screen (entrance screen) of the video distribution site as the screen display of the video distribution site displayed by communicating with the video content distribution server 2-2. Home button 11-1e for, stop button 11-1f for ending video playback, pause button 11-1g to pause video playback, play button 11-1h to start playing the paused content, post a comment A comment posting button 11-1i for posting a comment, a seek bar 11-1k for displaying the playback time at a relative position from the start point to the end point, and a seek button 11-1m are displayed, respectively.

The video distribution site provided by the video content distribution server 2-2 explained that each viewer can post a comment 11-1r to the video content, but the posted comment is the content playback time. It is displayed when another viewer plays this content with the same playback time as the posting time of the comment in (for example, 1 minute when posting in 1 minute from the start in the content of 3 minutes). To. Therefore, when posting a comment, the information on the posting time at which the comment was posted is transmitted from the viewer terminal to the server 2-2 together with the character information that is the content of the comment, and the server 2-2 records and stores the information. Then, when another viewer tries to play the same content and issues a playback transmission request signal to the server 2-2, the server 2-2 transmits the comment information with the posting time information to the viewer terminal together with the program content. , Each viewer terminal can read the comment with the same screen as the background at the same playback time posted by the poster.

FIG. 8 illustrates the original use of the “tag”, which is a content search item, which will be described later by explaining the screen transition of the moving image distribution site provided by the moving image content distribution server 2-2. It is a schematic diagram to try. Since the tag is also related to the user interface of the content distribution site screen, the explanation will be given in relation to the screen display.

On the portal screen (Fig. 8 (A)) that is first connected to the video distribution site and displayed, the site name 80 is first displayed as "cat cat video", and the tab 81 is "home" ( "Portal screen") and "Entertainment", "Life", and "Anime" are displayed as the categories (category tags) explained above. A plurality of thumbnail images 82 are displayed as recommended moving images at the bottom of the portal screen, and when the viewer clicks and selects a desired thumbnail image 82 from these with the mouse, playback of the program content is started.

FIG. 8B is a display screen when the viewer clicks and selects the “life” category displayed in FIG. 8 (A), and a plurality of tags 83 (“cow pot”, which belong to the category “life”). "Highball", "fishing", "cat", "cooking video", "camping", "nostalgic CM") are displayed on the screen and can be selected by the viewer.

FIG. 8C is a diagram showing a screen displayed when the tag “cooking video” is selected in FIG. 8B, and the selected tag name “cooking video” is displayed at the upper part of the screen. At the bottom of the screen, thumbnail images 85 of a plurality of video contents with the tag "cooking video" and captions (descriptions) 86 of the contents are displayed. Since the viewer can play the content by clicking and selecting the thumbnail 85 of the content that he / she likes, the tag that guides the viewer's selection is extremely useful. In addition, although not shown, it is also possible to search for a tag name corresponding to a desired word and display it in a list on another keyword selection screen.

[Process to generate high-quality images using machine-learned models]
Using the sequence chart of FIG. 9 and the flowchart of FIG. 10, the image quality encoded to a low bit rate was improved by using the transformation matrices Q and R, which are the machine-learned model data 32 described above. The process of obtaining an image will be explained again. The first viewer terminal 11 described above may be referred to as a viewer terminal 11.

First, the moving image content distribution server 2-2 stores a plurality of moving image contents composed of original images or moving image contents in which the original images are encoded at a low bit rate, and the viewer can distribute the contents described above. When you decide the content you want to watch from various images and other information on the site and click the content thumbnail button display displayed on the display screen of the viewer terminal 11, you can watch the distribution request signal of the corresponding content. It is transmitted from the user terminal 11 to the moving image content distribution server 2-2, and is received by the server 2-2 (FIG. 9, step S1).

On the other hand, the model data creation server 2-1 has the transformation matrices Q and R described above, which are machine-learned model data 32 corresponding to the contents instructed to be distributed to the moving image content distribution server 2-2. Are kept in record.

The model data corresponding to each content is, for example, in the case of moving image content related to "cat", a category of moving image content called "animal" is prepared in advance, and the original image belonging to this animal category is used as teacher data. The transformation matrices Q and R may be estimated and obtained by machine learning using an image obtained by encoding the original image at a low bit rate as an input image. Then, the model data creation server 2-1 or the moving image content distribution server 2-2 knows the moving image content that the user has requested to be distributed using the viewer terminal 11, and is suitable for improving the image of this content. The machine-learned model data may be selected from a plurality of prepared model data and distributed to the viewer terminal 11 via the moving image content distribution server 2-2 (FIG. 9, step S2, S3).

Alternatively, there is also a method of directly performing machine learning using the image in the moving image content to be distributed to obtain model data. That is, when performing machine learning using a neural network, the values (brightness, color tone) of the pixels (brightness, color tone) included in the low bit rate encoded image and the original image in the moving image content to be transmitted to the viewer terminal 11. ) May be used as input data and teacher data. With this configuration, the model data 32 becomes data content close to the content to be transmitted, and the quality of the high-quality image using the machine-learned model data 32 can be made high, but on the other hand, viewing It is necessary to carry out machine learning and prepare model data for all the contents that may be delivered to the personal terminal 11.

Therefore, based on the above points, the method of creating model data by machine learning using images belonging to the category or related field in which the content is included, instead of the content to be distributed, has been described above, for example. For the moving image content of the "cat", machine learning may be performed using the images included in the content of the "animal" category to generate model data. With such a configuration, the frequency of having to perform machine learning is reduced, and the number of content titles for distribution can be increased freely and quickly.

By the way, as explained earlier, there is also a method of using model data obtained by machine learning using images belonging to the "animal" category for moving image content related to "cat", but it is related to "cat". Determining whether the video content belongs to the "animal" category may also have to be made by the manipulating person. Furthermore, in order to obtain an image closer to the content to be distributed, that is, to obtain an image closer to the original image when high image quality processing is performed, the type of content to be distributed, the content being shot, the title, and the like. The model data may be divided according to the photographer, genre, etc., and the model data of the type adapted to each may be distributed together with the content, or these "types of content to be distributed, content being photographed, title" may be distributed. , Photographer, genre, etc., or a plurality of other items may be combined to select appropriate model data.

Therefore, for example, the following items are closely related to the content of each content, and the characteristics of the images included in the content can be appropriately classified. Therefore, the model data is automatically calculated according to these items. It is also effective to classify and prepare the content and distribute it together with the low bit rate encoded content of the moving image content requested to be distributed.

Therefore, the model data creation server 2-1 or the moving image content distribution server 2-2 described above provides the optimum model data for improving the image quality of the moving image content requested to be distributed. It may have a configuration for selecting from a plurality of prepared model data. The operation of selecting is such that when the following items are included in the moving image content to be distributed, model data suitable for high image quality processing is automatically selected from these items. You may.
-Comment information posted by viewers who viewed the content-Explanatory text information explaining the content-Information about the creator of the content-Information about the name or series name of the content-Information about the distributor who distributes the content

As described above, another item closely related to the content of the moving image content is "tag" information.

Here, the "tag" is a search keyword attached to each moving image content and indicates the content of the moving image, and for example, up to 10 tags can be registered for one content. The tag makes it easy for the viewer to find the desired video or a video similar to a certain video.

The tag can be freely registered not only by the video poster who posts the moving image content to the server 2, but also by the viewer (also referred to as the viewer) of the content. Originally used as a search function, there are many tags related to the content of videos and tags specific to video distribution sites. As for the actual usage, it may play a role of teaching viewers the highlights of the video rather than classification for search, and it may be used for communication between viewers using tags. Viewers voluntarily invented tags for videos dealing with the same material (for example, innumerable sub-genres belonging to popular genres such as "I tried to sing" and "Idolmaster") and videos by the same contributor. There is also an aspect that meets the needs of deeper search. (Partly, Wikipedia "Nico Nico Douga" https://ja.wikipedia.org/wiki/%E3%83%8B%E3%82%B3%E3%83%8B%E3%82%B3%E5%8B% Quoted from 95% E7% 94% BB.)

The applicant is the video distribution site "Nico Nico Douga"
https: // www. nicovideo. jp / video_top? ref = nicotop_video is operated.

The following examples are examples of tags actually used on this "Nico Nico Douga" site.

In the classification of "category" (also called "category tag"), which is a higher classification of tags, "entertainment / music" includes "VOICEROID theater", "original song", "virtual YouTube", "idle club", and "idle club". "Nijisanji", "Anison full", "Working BGM", "Fate / MMD", "MMD sword dance", "Nikosuro", "SCP commentary", "Pachisuro", "SCP", "Vocaloid karaoke DB" , "Slow commentary", "Voice actor live", "RAB", "Pachinko", "BGM for work without animation color", "Singing voiceroid", "VOCALOID", "With legend", "Cosplay" "I tried dancing with", "Nikopachi", "VOCALOID Hall of Fame", "Uchiiku TV", "Micra Liver Test", "Slowly Kaidan", "Halo Pro", "Western Music Masterpieces", "Become a Novelist" , "That song I was looking for", "Western music" are used, for example.

Similarly, in the category of "general life / sports", "Japan-US baseball", "Norich", "RTA (real mountain climbing attack)", "slow commentary", "VOICEROID in-vehicle", "WWE", "Kotsumekawauso", "Figure skating", "World traffic situation", "Bike", "Drive recorder", "Different friendship video link", "Fucking company", "Slow chat", "VOICEROID commentary", "Professional baseball", "Murder" "Hairball", "Lost Wild", "Boy Sake Lloyd", "Highball People", "Introduction of Bizarre / Weirdo / Great People in the World", "Slow Commentary Video", "Play Collection of the Active Age of Baseball World OB" , "Shiba dog", "Barbecue", "Battle race", "F1", "Niko Niko overseas travel", "Nuko Nuko video", "Wild liberation", "Outdoor cooking", "Ramen", "Military", "Military" "Home Run Collection", "Road Race", "Nostalgic CM", "Dog", "Zarashi", "Toast", "Slowly Car", "Baseball", "Yokohama DeNA BayStars", "Cat", "Screaming Beaver" , "Dog and cat" and the like are used, for example.

Similarly, in the category of "science and technology", "dust", "aviation accident", "gun", "documentary", "revolver", "military", "space dangerous", "rokuro-turning series", "hydrogen" "Sound of", "Handgun", "Figure", "Rare weapon", "Let's go by stray plane series", "Stray train derivative series", "Men of Naples", "Plastic model", "Japanese sword", " "Space", "Impact Video", "Military Training NG Collection", "Plastic Model", "Retro PC", "Mini 4WD", "Niko Niko Weapon Development Bureau", "JAXA", "Subaru", "Niko Niko Fancy" "Faculty of Science", "Size Comparison Series", "Black Hole", "Vehicle Approach Notification Device Series", "F-22", "World Traffic Conditions", "Flap Machine", "Science Hoi Hoi", "Mathematics" Etc. are used, for example.

As a result, there are the following special effects.

First, since the tag is given by the poster or the viewer of the content, there is no need for the man-hours to be given by the operator or administrator of the system 1, and the tag is given by the poster or viewer who is familiar with the contents of the content. So the grant is accurate.

In addition, as described above, tags are subdivided unlike simple categories, and posters and viewers who know existing tags give the same tags, so moving image content belonging to the same tag is Since the contents can be expected to be extremely close, the learning process in machine learning can be executed accurately.

As described above, the transformation matrices Q and R, which are model data 32, use the low bit rate encoded image as input and the corresponding original image as output teacher data for the image included in this content. It is obtained by estimation based on machine learning using the neural network described above.

The moving image content distribution server 2-2 transmits the model data 32 suitable for the content and the content data consisting of the low bit rate encoded image, which is the content data for which distribution is requested, to the viewer terminal 11 (step). S4).

The viewer terminal 11 receives the above model data 32 and the low bit rate encoded content data (step S11), and thereafter, for each frame of each low bit rate encoded image forming the content data, According to the equation (6) described above, each pixel value is obtained as output data in the neural network, and an image frame with high image quality based on the value is obtained (step S12). Then, by assembling the obtained image frames with high image quality on the time axis, the content data with high image quality is obtained (step S13).

[Second Embodiment]
As data used for machine learning, apart from or in addition to the low bit rate encoded image frame and pixel values (brightness, color tone) of the original image described above, the following image coding technology At least one of the items in the above may include meta information of low bit rate encoded moving image content for which high image quality is desired, such as the following, and other configurations may be explained first. It is possible to use a second embodiment configured to conform to the first embodiment of the present invention.
・ Coding block quantization parameter ・ Prediction error coefficient ・ Prediction mode information ・ Motion vector information

With such a configuration, it can be expected that the accuracy of estimation in machine learning will be further improved.

[Third Embodiment-Application to various data formats]
In each of the above embodiments, the implementation of the present invention has been described focusing on the distribution of moving image contents, but the present invention can be carried out not only for moving image contents but also for various data types such as still images and audio data. .. The configuration of the present embodiment applies the configurations of the first and second embodiments described above mutatis mutandis, and at least one of a single or a plurality of transmitters provided the original data encoded to a low bit rate. It is equipped with a machine learning unit that generates model data for generating improved data closer to the original data from bit rate encoded data by machine learning, and at least one of a single or a plurality of transmitters is also provided. , A transmitter that transmits low bit rate encoded data and model data to the outside of the device, and the receiving device improves the low bit rate encoded data from the received low bit rate encoded data and model data. It is a data transmission / reception system characterized by having an improved data generation unit for generating data. Further, it includes a configuration in which each configuration included in each embodiment of the moving image content distribution system 1 described above is adapted to another data format or a general-purpose data format instead of the moving image content target. You may do so.

When transmitting these various types of data, it is required to reduce the load on the transmission line, and when playing back on the receiving terminal, high playback quality is required. Similar to the problems in the system, the effect obtained by practicing the present invention is also the same as the effect described in each of the examples above.

[Fourth Embodiment-Direct distribution of model data to the client terminal]
Next, a fourth embodiment in which the details are different in each of the above-described embodiments will be described. It is possible to carry out the following configurations characteristic of the fourth embodiment in combination with the configurations of the respective embodiments described above, and these configurations are also included in the present invention. is there.

In each embodiment of the present invention described above, a request for distribution of a certain moving image content or data is made from a client terminal (corresponding to the first viewer terminal 11) to a server (moving image content distribution server 2-2). When it is sent to (equivalent), machine-learned model data suitable for improving this moving image content or data is selected, and another server (corresponding to model data creation server 2-1) sends a server (moving image content distribution server). 2-2 is equivalent), and the server (moving image content distribution server 2-2 is equivalent) is the low bit rate encoded data of the content or data requested to be distributed, and the selected machine-learned model. The data is distributed to the client terminal (corresponding to the first viewer terminal 11), and as a result, the client terminal is improved data from the received model data and the low bit rate encoded data. It was explained that it is possible to obtain moving image content with improved image quality.

Here, in implementing the present invention, the machine-learned model data is first sent from another server (corresponding to the model data creation server 2-1) to the server (corresponding to the moving image content distribution server 2-2), and then the server (corresponding to the moving image content distribution server 2-2). It is not essential or essential to distribute from the moving image content distribution server 2-2 (corresponding to the video content distribution server 2-2) to the client terminal (corresponding to the first viewer terminal 11). Instead, even if the machine-learned model data is directly distributed from another server (corresponding to the model data creation server 2-1) to the client terminal (corresponding to the first viewer terminal 11). Good.

When implemented in such a configuration, the server corresponding to the model data creation server 2-1 distributes from the client terminal corresponding to the first viewer terminal 11 to the server corresponding to the moving image content distribution server 2-2. Obtain information about the requested moving image content or data, select appropriate machine-learned model data to improve this content or data, and distribute it by the server corresponding to the moving image content distribution server 2-2. The machine-learned model data is sent to the client terminal corresponding to the first viewer terminal 11 at the delivery timing of the low bit rate encoded data (exemplified in the moving image content) or at a time before and after the distribution timing. It will be delivered directly.

That is, when the configuration of the fourth embodiment is realized in the field of moving image content distribution, a transmission system provided with one or more transmitting devices, that is, servers, produces low bit rate encoded moving image content. It has a component to be transmitted and a component to transmit machine-learned model data suitable for improving the low bit rate encoded moving image content into a high quality moving image content, while having a component to be transmitted. The receiving terminal has a component that generates high-quality moving image content from the received low bit rate-encoded moving image content and the similarly received machine-learned model data.

Further, when the configuration of the fourth embodiment is realized not only in the field of moving image content distribution but also in the general data distribution field, transmission provided with a server as a single or a plurality of transmission devices. A configuration in which the system sends low bit rate encoded data and a machine trained model data suitable for improving this low bit rate encoded data to data closer to the original data. On the other hand, the receiving terminal has a component that generates data improved so as to be closer to the original data from the received low bit rate encoded data and the similarly received machine-learned model data. ..

(Explanation of the effect of the invention)
The present invention provides efficient transmission band compression and closeness to the original image in a system that sends and receives video streaming to view moving image content via a transmission line such as an Internet communication network that has only a limited bandwidth. Image transmission / reception system, data transmission / reception system, transmission / reception method, computer program, image transmission system, image receiver, transmission that can efficiently perform image restoration with a sense of resolution while reducing the burden on the operator. A system and a receiving device can be provided.

Video content distribution system 1
Model data creation server 2-1
Video content distribution server 2-2
First viewer terminal 11
Original image 30
Low bitrate encoded image 31
Machine-learned model data 32
High-quality image 33

Claims (21)

Machine data for at least one of a single or a plurality of transmitters to generate an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate. Equipped with a machine learning unit generated by learning
At least one of the single or plural transmitters comprises a transmitter that transmits the low bit rate encoded image and the model data to the outside of the device.
An image transmission / reception system, wherein the receiving device has an improved image generation unit that generates the improved image of the low bit rate encoded image from the received low bit rate encoded image and the model data. The image transmission / reception system according to claim 1, wherein the data used for machine learning further includes meta information of the low bit rate encoded image. The meta information of the low bit rate encoded image is at least one of a coding block quantization parameter (QP), a prediction error coefficient, a prediction mode information, and a motion vector information in an image coding technique. The image transmission / reception system according to claim 2. At least one of the single or plural transmitters is further transmitted with the low bit rate encoded image based on information about any of the low bit rate encoded images transmitted from the transmitter. The image transmission / reception system according to any one of claims 1 to 3, further comprising a model data selection unit that selects model data from a plurality of models. At least one of a single or a plurality of transmitters provides model data for generating improved data closer to the original data from the low bit rate encoded data obtained by encoding the original data to a low bit rate. Equipped with a machine learning unit generated by learning
At least one of the single or plural transmitters comprises a transmitter that transmits the low bit rate encoded data and the model data to the outside of the device.
A data transmission / reception system, wherein the receiving device includes an improved data generation unit that generates the improved data of the low bit rate encoded data from the received low bit rate encoded data and the model data. How to send and receive images
A machine learning unit possessed by at least one of a single or a plurality of transmitters for generating an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate. Steps to generate model data by machine learning,
A step in which a transmitter of at least one of the single or plural transmitters transmits the low bit rate encoded image and the model data to the outside of the device.
Transmission / reception, wherein the improved image generation unit of the receiving device includes a step of generating the improved image of the low bit rate encoded image from the received low bit rate encoded image and the model data. Method. The transmission / reception method according to claim 6, wherein the data used for machine learning further includes meta information of the low bit rate encoded image. The meta information of the low bit rate encoded image is at least one of a coding block quantization parameter (QP), a prediction error coefficient, a prediction mode information, and a motion vector information in an image coding technique. The transmission / reception method according to claim 7. The model data transmitted by at least one of the single or plural transmitters together with the low bit rate encoded image based on information about the low bit rate encoded image transmitted from the transmitter. The transmission / reception method according to any one of claims 6 to 8, wherein the model data selection unit is provided to select the data from a plurality of the data. A machine learning unit possessed by at least one of a single or a plurality of transmitters for generating improved data closer to the original data from the low bit rate encoded data obtained by encoding the original data to a low bit rate. Steps to generate model data by machine learning,
A step in which a transmitter of at least one of the single or plural transmitters transmits the low bit rate encoded data and the model data to the outside of the device.
A transmission / reception method, wherein the improved data generation unit of the receiving device includes a step of generating the improved data of the low bit rate encoded data from the received low bit rate encoded data and the model data. .. A computer program for executing the transmission / reception method according to any one of claims 6 to 10. Model data for generating an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate provided in at least one of a single or a plurality of transmitters. With the machine learning unit generated by machine learning,
It is characterized by including a transmission unit for transmitting the low bit rate encoded image and the model data to the outside of the system, which are provided in at least one of the single or a plurality of transmission devices. Image transmission system. The image transmission system according to claim 12, wherein the data to be used for machine learning is meta information of the low bit rate encoded image. The meta information of the low bit rate converted image is at least one of a coding block quantization parameter (QP), a prediction error coefficient, a prediction mode information, and a motion vector information in an image coding technique. , The image transmission system according to claim 13. Further, it has a model data selection unit that selects from a plurality of the model data transmitted together with the low bit rate encoded image based on the information regarding the low bit rate encoded image transmitted from the transmission unit. The image transmission system according to any one of claims 12 to 14, wherein the image transmission system is characterized. Model data for generating improved data obtained by encoding low bit rate encoded data obtained by encoding the original data to a low bit rate, which is provided in at least one of a single or a plurality of transmitters, and closer to the original data. With the machine learning unit generated by machine learning,
A transmission including a transmission unit provided in at least one of the single or a plurality of transmission devices for transmitting the low bit rate encoded data and the model data to the outside of the device. system. Model data for generating an improved image closer to the original image from a low bit rate encoded image obtained by encoding the original image to a low bit rate, and the model data generated by machine learning and the low bit rate. A receiver that receives rate-encoded images from an image transmission system,
An image receiving device including an improved image generation unit that generates the improved image of the low bit rate encoded image from the received low bit rate encoded image and the model data. The image receiving device according to claim 17, wherein the data to be used for machine learning is meta information of the low bit rate encoded image. The meta information of the low bit rate encoded image is at least one of a coding block quantization parameter (QP), a prediction error coefficient, a prediction mode information, and a motion vector information in an image coding technique. The image receiving device according to claim 18. The image receiving device according to claim 17, wherein the model data received by the receiving unit is selected from a plurality of models based on the information regarding the low bit rate encoded image received together. Model data for generating improved data closer to the original data from the low bit rate encoded data obtained by encoding the original data to a low bit rate, and the model data generated by machine learning and the low bit. A receiver that receives rate-encoded data from the transmission system,
A receiving device including an improved data generation unit that generates the improved data of the low bit rate encoded data from the received low bit rate encoded data and the model data.

Images