KR102573511B1 - Image data processing appratus and method - Google Patents

Abstract

The present invention relates to an apparatus and method for processing image data based on artificial intelligence (AI) for real-time image communication. An image data transmission and processing device among image data processing devices according to an embodiment of the present invention uses pre-stored image data to train deep neural network models to perform optimal compression according to the characteristics of objects included in the image data. A storage unit for storing, a determination unit for determining whether a deep neural network model to be used for compressing the image data is stored in the storage unit based on information on the target image data to be transmitted, and a deep neural network to be used for the target image data to be transmitted based on the determination result of the determination unit. As the model is stored in the storage unit, a compression unit that selects a deep neural network model to compress image data from the storage unit and generates compressed image data by compressing target image data to be transmitted using the selected deep neural network model; It may include a transmitter for transmitting the compressed video data to the video data receiving and processing device.

Description

Image data processing device and method {IMAGE DATA PROCESSING APPRATUS AND METHOD}

The present invention relates to an apparatus and method for processing image data based on artificial intelligence (AI) for real-time image communication.

Traffic in the network communication between the transmitter and the receiver occupies a large proportion of large-sized video data compared to relatively small-sized text data or Nile images. In particular, in the case of high-capacity video data such as high-quality video over 4K or 8K, a high compression rate is required to support real-time streaming. Even the latest video compression technologies such as H.265/HEVC or VP9 have limitations in supporting real-time streaming of high-quality video. Although AI-based video compression techniques have been proposed to increase compression efficiency, general AI-based video compression techniques cannot be applied to real-time streaming due to the delay required for AI model training for each video content.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

Domestic Patent Publication No. 10-2019-0033541 (2019.03.29)

An object of the present invention is to propose a deep neural network-based image compression technique that removes redundancy and similarity of a target image to be transmitted through a deep neural network model that learns object characteristics from a pre-stored image or an image acquired through an initialization process. .

An object of the present invention is to support real-time streaming such as video communication by pre-learning characteristics of each user from a pre-stored image or an image acquired through an initialization process and storing the learned deep neural network model.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems and advantages of the present invention that are not mentioned can be understood by the following description and more clearly understood by the embodiments of the present invention. It will be. In addition, it will be appreciated that the problems and advantages to be solved by the present invention can be realized by the means and combinations indicated in the claims.

An image data transmission and processing device among image data processing devices according to an embodiment of the present invention uses pre-stored image data to train deep neural network models to perform optimal compression according to the characteristics of objects included in the image data. A storage unit for storing, a determination unit for determining whether a deep neural network model to be used for compressing the image data is stored in the storage unit based on information on the target image data to be transmitted, and a deep neural network to be used for the target image data to be transmitted based on the determination result of the determination unit. As the model is stored in the storage unit, a compression unit that selects a deep neural network model to compress image data from the storage unit and generates compressed image data by compressing target image data to be transmitted using the selected deep neural network model; It may include a transmitter for transmitting the compressed video data to the video data receiving and processing device.

An image data receiving and processing device among image data processing devices according to an embodiment of the present invention includes: a receiver for receiving compressed image data from the image data transmission and processing device; a storage unit for storing deep neural network models associated with a user account; According to the information of the user account that sent the image data, a deep neural network model to restore the compressed image data is selected among the deep neural network models stored in the storage unit, and the original image data is restored from the compressed image data using the selected deep neural network models. It may include a restoration unit that does.

Among the image data processing methods according to an embodiment of the present invention, the image data transmission processing method includes deep neural network models trained to perform optimal compression according to the characteristics of objects included in the image data using pre-stored image data. The step of constructing a storage unit to store, the step of determining whether a deep neural network model to be used for compression of the image data is stored in the storage unit based on the information of the target image data to be transmitted, the determination result, the step of determining whether or not the deep neural network model to be used for the image data to be transmitted is stored in the storage unit, and As the deep neural network model is stored in the storage unit, selecting a deep neural network model to compress image data from the storage unit, and generating compressed image data by compressing target image data to be transmitted using the selected deep neural network model; , transmitting the compressed video data to the video data receiving and processing device.

Among the image data processing methods according to an embodiment of the present invention, a method for transmitting and processing image data includes the steps of receiving compressed image data from an image data transmission processing device, and storing compressed image data in a storage unit for storing deep neural network models associated with a user account. Selecting a deep neural network model to restore compressed image data from among deep neural network models stored in a storage unit according to information of a user account that has transmitted data; and extracting original image data from compressed image data using the selected deep neural network model. Restoration may be included.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the present invention, redundancy and similarity of image data to be transmitted can be removed by compressing and transmitting a target image to be transmitted through a deep neural network model that has learned characteristics of an object from a pre-stored image or an image acquired through an initialization process.

In addition, unlike conventional deep neural network-based video compression techniques that require deep neural network model training for each video content, the characteristics of each user (video sender or video producer) are learned in advance from pre-stored images or images acquired through an initialization process, and learning Real-time streaming such as video communication can be supported by storing the deep neural network model.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary view of an image data processing environment according to an exemplary embodiment.
2 is an exemplary diagram of a video data processing environment for transmitting and receiving video content according to another embodiment.
FIG. 3 is a block diagram schematically illustrating configurations of an image data transmission device and an image data reception device included in the image data processing environment of FIG. 1 .
4 to 7 are examples for schematically explaining the operation of the compression unit in the video data transmission apparatus according to the present embodiment.
8 is a flowchart illustrating a method of processing image data according to an exemplary embodiment.

Advantages and features of the present invention, and methods for achieving them will become clear with reference to the detailed description of embodiments in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention and to fully inform those skilled in the art of the scope of the invention to which the present invention belongs. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Terms such as first and second may be used to describe various components, but components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

1 is an exemplary view of an image data processing environment according to an exemplary embodiment. Referring to FIG. 1 , an image data processing environment 1 may include an image data transmission processing device 100 , an image data reception processing device 200 and a network 300 .

The image data transmission and processing apparatus 100 may select a deep neural network model to be used for compressing image data based on information on image data to be transmitted from pre-stored deep neural network models. The image data transmission and processing apparatus 100 may transmit compressed image data compressed using a deep neural network model in which target image data to be transmitted is selected to the image data reception and processing apparatus 200 . Here, the information of the image data may include at least one of a user account generating the image data, a time at which the image data is generated, and a location where the image data is generated.

The image data transmitting and processing device 100 selects image data similar to target image data to be transmitted from pre-stored image data, learns characteristics of objects included in the selected similar image data and similar image data as training data, and builds a deep neural network model. can be created anew. The image data transmitting and processing apparatus 100 may transmit compressed image data obtained by compressing the target image data using the newly generated deep neural network model and the newly generated deep neural network model to the image data receiving and processing apparatus 200 .

The image data transmission and processing apparatus 100 may transmit compressed image data and a training manual used for training the selected deep neural network model to the image data reception and processing apparatus 200 .

When the image data receiving and processing device 200 receives compressed image data from the image data transmission and processing device 100, it selects a deep neural network model to restore the compressed image data from deep neural network models stored in association with a pre-stored user account. And, the original image data can be restored from the compressed image data using the selected deep neural network models.

When the image data receiving and processing device 200 receives compressed image data and a training manual used to train a deep neural network model from the image data transmission processing device 100, a deep neural network model is built using the received training manual. And, the original image data can be restored from the compressed image data using the built deep neural network model. Here, the time point at which the deep neural network model construction is completed using the training manual may include before restoring the original image data.

In this embodiment, the video data transmission and processing apparatus 100 may include a caller terminal (not shown), and the video data reception and processing apparatus 200 may include a called party terminal (not shown). Accordingly, when the video data transmission processing device 100 and the video data reception processing device 200 transmit and receive image data, video communication may be performed between the caller terminal and the called party terminal.

In this embodiment, the deep neural network model may include a compressed deep neural network model using an encoder and a reconstructed deep neural network model using a decoder. That is, one deep neural network model may be composed of a set of compressed deep neural network models and reconstructed deep neural network models. Depending on the type of data input to the deep neural network model, a compressed deep neural network model is selected or a reconstructed deep neural network model is selected, and output data may also vary. If data input to the deep neural network model is general raw image data, a compressed deep neural network model may be selected and compressed image data may be output. In addition, when data input to the deep neural network model is compressed image data, a reconstructed deep neural network model may be selected and reconstructed image data may be output.

The deep neural network model used or generated by the image data transmission and processing apparatus 100 may be a compressed deep neural network model. The compressed deep neural network model may be trained in advance to output compressed image data including objects using image data including objects. Here, video data including an object refers to user-specific characteristics according to situation and/or time (for example, in the case of a video call, the face of a user making a video call, background, user in the morning, user in the evening) , It may be image data including soccer images, basketball images, etc.) as types of images mainly taken by the user.

The deep neural network model used or constructed by the image data receiving and processing apparatus 200 may be a reconstructed deep neural network model. The reconstructed deep neural network model may be trained in advance to output original image data including objects using compressed image data including objects.

The network 300 may play a role of connecting the image data transmission and processing device 100 and the image data reception and processing device 200 . Such a network 300 may be, for example, a wired network such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), and integrated service digital networks (ISDNs), wireless LANs, CDMA, Bluetooth, and satellite communication. However, the scope of the present invention is not limited thereto. Also, the network 300 may transmit and receive information using short-range communication and/or long-distance communication. Here, the short-range communication may include Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, wireless fidelity (Wi-Fi) technology, Telecommunications include code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), and single carrier frequency division multiple access (SC-FDMA) technologies. can do.

The network 300 may include connections of network elements such as hubs, bridges, routers, and switches. Network 300 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to network 300 may be provided via one or more wired or wireless access networks. Furthermore, the network 300 may support an Internet of Things (IoT) network and/or 5G communication in which information is exchanged and processed between distributed components such as things.

2 is an exemplary diagram of a video data processing environment for transmitting and receiving video content according to another embodiment. Referring to FIG. 2 , the video data transmitting and processing apparatus 100 includes a server (not shown) that stores and/or provides video contents or a video contents provider terminal (not shown) that generates,/or stores, and provides video contents. can include In addition, the video data reception and processing apparatus 200 may include a video content receiver terminal (not shown) that accesses a server and/or a video content provider terminal to receive and reproduce video content.

From FIG. 2 , the video content currently transmitted by the video data transmission processing device 100 (server or video content sender terminal) is compared with the video content received 3 days ago by the video data reception processing device 200 (video content receiver terminal). Then, it can be seen that the background and the character are the same, and the food the character is eating is different.

In this case, the image data transmission and processing apparatus 100 uses a deep neural network model trained on the object characteristics (eg, background, person) of image content stored 3 days ago among a plurality of pre-stored deep neural network models. to compress the target video content (on air in FIG. 2) to be transmitted. Since the image data transmission and processing apparatus 100 compresses image content using a deep neural network model trained on object characteristics, it is possible to transmit image content in real time by removing redundancy and similarity when compressing image content.

The image data transmission and processing device 100 may transmit compressed target image content and a deep neural network model used for compressing the target image content to be transmitted (or a training manual used for deep neural network training) to the image data reception and processing device 200. .

Upon receiving the compressed target video content, the video data receiving and processing apparatus 200 may restore original video content from the compressed target video content using a deep neural network model stored in association with a user account.

Since the image data reception and processing apparatus 200 restores original image content using a deep neural network model trained on object characteristics, it can reproduce image content in real time by removing redundancy and similarity when reconstructing image content.

Alternatively, the image data receiving and processing apparatus 200 receiving the compressed target image content and the training manual used for training the deep neural network builds a deep neural network model using the training manual, and uses the built deep neural network model to compress the target image. Original video content may be restored from content.

FIG. 3 is a block diagram schematically illustrating configurations of an image data transmission device and an image data reception device included in the image data processing environment of FIG. 1 . In the following description, descriptions of portions overlapping those of FIGS. 1 and 2 will be omitted.

Referring to FIG. 3 , the image data transmission and processing apparatus 100 includes a first storage unit 110, a determination unit 120, a compression unit 130, a transmission unit 140, a database 150, and a generation unit 160. ) and a first control unit 170.

Also, referring to FIG. 3 , the image data receiving and processing apparatus 200 includes a receiving unit 210, a second storage unit 220, a search unit 230, a restoration unit 240, a construction unit 250, and a second storage unit 220. A controller 260 may be included. Also, in this specification, “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by the hardware component such as a processor.

First, the video data transmission and processing apparatus 100 will be described.

The first storage unit 110 may store deep neural network models trained to perform optimal compression according to characteristics of objects included in the image data using image data stored in the database 150 . Here, the deep neural network models are training models trained to perform compression using pre-stored image data according to the user account that generates the image data, and are created in the first user account to compress the image data of the first user account. It may include a first deep neural network model trained in advance by using existing image data as training data.

As an embodiment, the first storage unit 110 may store training manuals used when training a deep neural network model. Here, the training manual may store all information required to create a compressed deep neural network model, such as, for example, training data set information, number of times of training, random seed, and type of optimizer.

Here, the first storage unit 110 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. The first storage unit 110 may include a built-in memory and/or an external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, Non-volatile memory such as flash ROM, NAND flash memory, or NOR flash memory, SSD. It may include a compact flash (CF) card, a flash drive such as an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

The determination unit 120 may determine whether a deep neural network model to be used for compressing the image data is stored in the first storage unit 110 based on information on target image data to be transmitted to the image data receiving and processing apparatus 200 .

Here, the information of the target image data may include at least one of a user account for generating the image data, a time at which the image data is generated, and a location where the image data is generated, and the information of the target image data may include a first storage unit ( 110) may be tagged with the deep neural network model.

Accordingly, the determination unit 120 may determine whether there is a deep neural network model having the same tagging information based on the information of the target image data to be transmitted from the first storage unit 110 .

As another embodiment, the determination unit 120 determines the first deep neural network model to be used for compressing the image data based on the first similarity information calculated by extracting the characteristics of the target image data to be transmitted to the image data receiving and processing device 200. It may be determined whether or not it is stored in the storage unit 110 .

In this embodiment, image data having similar characteristics can be used as training data to train the deep neural network model. In this case, the second similarity information calculated by extracting the features of the image data used is provided along with the trained deep neural network model. 1 can be stored in the storage unit 110. Here, the calculation of the similarity information is similarity between images, such as image feature matching, context-based semantic matching, and context-based visual matching, which calculate similarity by extracting image features such as outlines and color characteristics included in the image. can be calculated using one of the techniques for calculating .

Accordingly, the determination unit 120 may determine whether there is a deep neural network model having second similarity information most similar to the first similarity information, from the first storage unit 110 .

The compression unit 130 extracts the image data from the first storage unit 110 according to the determination result of the determination unit 120 that the deep neural network model to be used for the image data to be transmitted is stored in the first storage unit 110. A deep neural network model to be compressed may be selected, and compressed image data obtained by compressing target image data to be transmitted may be generated using the selected deep neural network model. Hereinafter, a detailed description of the compression unit will be described with reference to FIGS. 4 to 7 .

The transmission unit 140 may transmit the compressed image data generated by the compression unit 130 to the image data receiving and processing device 200 .

As an optional embodiment, the transmission unit 140 may transmit the deep neural network model used in the compression unit 130 to the image data receiving and processing device 200 . In this way, when the video data transmission and processing apparatus 100 transmits the deep neural network model to the video data reception and processing apparatus 200, the video data transmission and processing apparatus 100 suffers a transmission cost loss, but receives the video data. From the viewpoint of the processing device 200, it is possible to create an effect that does not require a computing cost for training a deep neural network model.

As an optional embodiment, the transmission unit 140 may select a training manual capable of training a deep neural network model from the first storage unit 110 and transmit the compressed image data and the training manual to the image data receiving and processing device 200. there is. In this way, when the video data transmission and processing apparatus 100 transmits the training manual used for training the deep neural network model to the video data reception and processing apparatus 200, the video data transmission and processing apparatus 100 reduces the transmission cost. Savings can be achieved, but from the point of view of the image data receiving and processing device 200, an effect requiring computing cost for generating a deep neural network model can be created.

The database 150 stores image data. Here, the video data may include image, video, or video content combining text, image, video, and sound source.

Image data stored in the database 150 refers to previously transmitted image data, image data synchronized with the image data receiving and processing device 200, or acquired through an initialization process with the image data receiving and processing device 200. One or more of the image data may be included.

As a result of the determination of the determination unit 120, the generating unit 160 determines that the deep neural network model to be used for the target image data to be transmitted is not stored in the first storage unit 110, and thus the target image data to be transmitted and the target image data to be transmitted from the database 150. A deep neural network model may be created by selecting similar image data and learning characteristics of an object included in the selected similar image data and the similar image data as training data.

Here, the compression unit 130 may generate compressed image data obtained by compressing target image data to be transmitted using the deep neural network model generated by the generation unit 160, and the transmission unit 140 may generate the compressed image data and the generation unit ( 160) may transmit the generated deep neural network model to the image data receiving and processing device 200.

The first control unit 170 may control the entire operation of the video data transmission and processing apparatus 100 . The first controller 170 may include all types of devices capable of processing data, such as a processor. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit), field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

Next, the video data receiving and processing device 200 will be described.

The receiving unit 210 may receive compressed video data from the video data transmission and processing apparatus 100 .

The second storage unit 220 may store deep neural network models associated with a user account. Here, the deep neural network models are learning models trained to perform restoration using pre-stored image data according to the user account that transmits the image data, and to restore compressed image data of the image data of the first user account, the first First reconstructed deep neural network models previously trained using existing image data received from a user account as training data may be included.

The search unit 230 receives the compressed video data and the sender's user account information included in the compressed video data from the video data transmission processing apparatus 100 by the receiver 210, so that the second storage unit 220 A deep neural network model stored in association with a user account can be retrieved from .

The restoration unit 240 selects a deep neural network model to restore the compressed image data from among the deep neural network models stored in the second storage unit 220 according to the information of the user account that transmitted the compressed image data, and selects the selected deep neural network model. Original video data can be restored from compressed video data using .

As an example, the receiving unit 210 may receive compressed image data and a deep neural network model from the image data transmission and processing apparatus 100 . The restoration unit 220 may restore original image data from compressed image data using the received deep neural network model.

As an example, the receiving unit 210 may receive compressed image data and a training manual used for training a deep neural network model from the image data transmission and processing device 100 .

The builder 230 may build a deep neural network model by using the training manual as the receiver 210 receives the training manual used to train the deep neural network model. The deep neural network model built by the construction unit 230 may be stored in the second storage unit 240 . Here, when the deep neural network model is built using the training manual, the deep neural network to restore the compressed image data is not stored in the second storage unit 220, and the receiving unit 210 uses the deep neural network model for training. This may include a case where a training manual is received.

The restoration unit 220 may restore original image data from the compressed image data using the deep neural network model built by the construction unit 230 .

The second controller 260 can control the entire operation of the image data receiving and processing device 200, and since details are the same as those of the first controller 170, they will be omitted.

4 to 7 are examples for schematically explaining the operation of the compression unit in the video data transmission apparatus according to the present embodiment. In the following description, descriptions of portions overlapping those of FIGS. 1 to 3 will be omitted.

In general, image compression may be composed of spatial redundancy elimination and temporal redundancy elimination. In order to remove temporal redundancy, the frame of an image can be divided into three types (I frame, P frame, B frame).

In order to ensure random access of images, the I frame performs only spatial redundancy removal like JPEG without considering temporal redundancy removal, and may serve as a reference frame for other frames.

The P frame may increase compression efficiency by removing temporal redundancy between images by referring to a previously reproduced I frame or P frame.

The B frame increases compression efficiency through bidirectional prediction using a previously played I or P frame and a frame to be played later, and the B frame cannot be a reference frame for other frames.

As shown in FIG. 4, a video is encoded in a group of picture (GOP) unit, and generally has one or exceptionally more I-frames per GOP. Since I-frames that cannot perform temporal redundancy removal take up a large portion of the entire video, there is room for further compression of the video (the H.265 compression standard I-frame using 30 frames with GPO occupies about 40% of the total video size). occupied).

Until now, video compression technologies (e.g., VP8, VP9, AV1, H.264/AVC, H.265/HEVC, etc.) only use spatial and temporal redundancy elimination inside their video data. For the random access function, only spatial redundancy removal can be performed in the case of an I frame.

In the case of the present embodiment, it goes beyond the elimination of temporal/spatial redundancy in a single image of the existing image compression technique, and between image data through a deep neural network model that learns user-specific characteristics from image data that has been previously transmitted/synchronized/saved or acquired through an initialization process. That is, redundancy/similarity between video files can be used for video compression transmission.

As shown in FIG. 5, the existing video compression technology is a method of finding a matching pattern based on a macroblock (H.264) and its modified CTU (H.265) unit, generally in terms of time of one GOP unit. It is not easy to find an exact matching pattern except for extremely contiguous or spatially contiguous cases in one image.

In addition, when searching for macroblocks/CTUs that match between temporally adjacent frames, the computational complexity is too large to search all adjacent frames. ×64, or 128 × 128, etc.).

Therefore, in this embodiment, rather than finding redundancy in one GOP unit (short timescale) or limiting the search range in removing redundancy between frames, learning characteristics for each user from image data transmitted in advance or obtained through an initialization process An AI-based image compression technique that removes redundancy/similarity of target image data can be proposed through a deep neural network model.

In addition, since this embodiment compresses and restores the deep neural network model learned through data transmitted/synchronized/stored in advance, the I frame does not lose the existing random access function, and redundancy/similarity between files can be used.

Unlike existing AI-based video compression techniques that require AI model learning for each video content, this embodiment learns the characteristics of each user in advance from a pre-stored video or an image obtained through an initialization process and stores the learned deep neural network model to improve image communication and The same real-time streaming support is possible.

In this embodiment, the image compression method performed by the compression unit 130 can be largely classified into intra frame (I frame compression) compression and inter frame (P frame or B frame) compression.

The compression unit 130 may first convert target video data to be transmitted into group of pictures (GOP), which are image frame groups. Each image frame group may include an intra frame and an inter frame.

Here, an intra frame is also referred to as an I frame or a key frame, and every image frame group must have at least one I frame. In order to ensure random access of the image, the I frame is stored as the original input without considering temporal redundancy removal, or is formed by removing only spatial redundancy like, for example, JPEG, and does not refer to other frames. It is a frame that other frames can refer to.

Among the inter-frames, the P frame is a forward prediction frame, which is a frame in which only partial data that differs from the immediately previous frame is predicted and stored, and temporal redundancy between frames is removed to increase compression efficiency.

Among the inter-frames, the B frame is a bi-directional prediction frame, which improves compression efficiency by referring to both frames between the I frame and the P frame and storing motion between the two frames as guess data.

In the process of compressing target image data to be transmitted using a deep neural network model generated by reflecting user characteristics, the compression unit 130 compresses intra frames in each image frame group by applying the deep neural network model, and the inter frame is compressed by applying the deep neural network model. The frame may be compressed based on the intra frame to generate compressed image data.

Referring to FIG. 6, intra frame compression in the compression unit 130 is described. Using image data previously transmitted/synchronized/stored or obtained through an initialization process, user-specific characteristics are learned to compress and restore frames. AI-based deep neural network models can be created. An intra frame, that is, an I frame among frames of the GOP may be compressed through an encoder of the generated deep neural network model. The compressed intra frame may be restored through a decoder of the generated deep neural network model. Here, the characteristics of each user may be a background of an image produced according to the user, an object, a face appearing in the image, and the like.

Intra-frame compression technique As an example, learning of a deep neural network model may include a super resolution (SR) technique, which is one of frame restoration deep learning techniques. The SR technique is a deep neural network model that reconstructs low-quality frames into high-quality frames. In this embodiment, the SR model that learns object characteristics from similar images/videos can be used as an intra-frame compression technique deep neural network model.

In addition, as an example of learning a deep neural network model, an intra-frame compression technique may include an autoencoder (AE) technique that compresses and restores an original frame with an encoder and a decoder, which are deep neural networks. In this embodiment, the autoencoder model that has learned the characteristics of an object can be used as an intra-frame compression method deep neural network model, and the difference from the existing deep learning neural network is that the object image is intensively learned and shared in advance.

In this embodiment, the learning method of the deep neural network including SR or AE is largely 1) a lossless compression method that learns in the direction of minimizing the sum of the compressed image size and the offset bit, which is the difference between the original image and the reconstructed image. and 2) a lossy compression method that learns in the direction of maximizing the PSNR or SSIM values of the original image and the reconstructed image.

Referring to FIG. 7, inter-frame compression in the compression unit 130 is described. A deep neural network model learned based on user-specific characteristics can be created. That is, it is possible to search with low computational complexity by performing inter-frame redundancy search within the entire frame range through a trained deep neural network model. Through the encoder of the generated deep neural network model, it is possible to find a matching block from a selected reference frame among previous frames and compress inter-frames, that is, P-frames or B-frames, through motion vectors. The inter-frame may be reconstructed through the motion vector transmitted from the previously reconstructed reference frame through the decoder of the generated deep neural network model.

Inter-Frame Compression Technique As an example of deep neural network model learning, P frame uses unidirectional prediction to predict the motion vector of the current frame using two consecutive previous frames, and B frame uses the previous and next frames to predict the motion vector of the current frame. It may include a technique of performing bi-directional prediction for predicting through a deep neural network prediction model.

In addition, a motion estimation technique for finding a matched motion vector through a model learned from patterns of motion vectors matched between adjacent frames may be included.

In this embodiment, DNN models of techniques applying deep neural networks to motion estimation, such as the above two techniques, are learned using similar images/videos, and learning in a direction of minimizing the difference between the estimated motion vector and the motion vector that actually matches, , AI-based inter-frame compression can be applied to real-time streaming by pre-sharing this deep neural network model.

8 is a flowchart illustrating a method of processing image data according to an exemplary embodiment. In the following description, descriptions of portions overlapping those of FIGS. 1 to 7 will be omitted.

Referring to FIG. 8 , in step S801, the image data transmission processing apparatus 100 uses pre-stored image data to store deep neural network models trained to perform optimal compression according to the characteristics of objects included in the image data. A first storage unit to be built. Here, the deep neural network models are training models trained to perform compression using pre-stored image data according to the user account that generates the image data, and are created in the first user account to compress the image data of the first user account. It may include a first deep neural network model trained in advance by using existing image data as training data.

In step S803, the video data transmission processing apparatus 100 determines whether a deep neural network model to be used for compression of the video data is stored in the first storage unit based on the information of the target video data to be transmitted. Here, the information of the image data may include at least one of a user account for generating the image data, a time at which the image data is generated, and a location where the image data is generated, and may be tagged with a deep neural network model stored in the first storage unit. can

As an optional embodiment, the image data transmission and processing apparatus 100 determines whether a deep neural network model to be used for compression of the image data is stored in the first storage unit based on the first similarity information calculated by extracting the characteristics of the target image data to be transmitted. can judge In this embodiment, image data having similar characteristics can be used as training data to train the deep neural network model. In this case, the second similarity information calculated by extracting the features of the image data used is provided along with the trained deep neural network model. 1 can be stored in the storage unit. Accordingly, the image data transmission and processing apparatus 100 may determine whether there is a deep neural network model having second similarity information most similar to the first similarity information, from the first storage unit.

In step S805, when it is determined that the deep neural network model for compressing the image data to be transmitted is stored in the first storage unit, the image data transmission processing apparatus 100 selects a deep neural network model to be used for target image data to be transmitted from the first storage unit. do.

In step S807, the image data transmission processing apparatus 100 selects image data similar to the target image data to be transmitted from the database when it is determined that the deep neural network model for compressing the image data to be transmitted to the first storage unit is not stored.

In step S809, the image data transmitting and processing apparatus 100 learns the selected similar image data and the characteristics of objects included in the similar image data as training data to create a deep neural network model.

In step S811, the image data transmission processing apparatus 100 compresses the target image data to be transmitted using the deep neural network model selected from the first storage unit to generate compressed image data, or uses the generated deep neural network model to generate the target image to be transmitted. Compresses data to generate compressed video data. Here, the image data transmission and processing apparatus 100 converts the image data into image frame groups, each group including an intra frame and an inter frame, and transmits the object using a deep neural network model. In the process of compressing image data, intra frames in each image frame group are compressed by applying a deep neural network model, and inter frames are compressed based on the intra frames to generate compressed image data. Also, the deep neural network model may be one of a deep learning-based super-resolution model and an auto-encoder model.

In step S813, the image data transmitting and processing device 100 transfers the compressed image data, the compressed image data and the deep neural network model, or the training manual used when training the compressed image data and the deep neural network model to the image data receiving and processing device 200. can be sent to

In step S815, the image data receiving and processing apparatus 200 may receive compressed image data, compressed image data and deep neural network model, or compressed image data and training manual used to train the deep neural network model.

In step S817, the image data receiving and processing device 200 determines whether the deep neural network model to restore the compressed image data is a complete deep neural network model. In the case of the complete deep neural network model, if the deep neural network model corresponding to the user account included in the compressed image data is stored in the second storage unit, it may be determined that the deep neural network model is complete. In addition, in the case of a complete deep neural network model, when the deep neural network model is received together with compressed image data, it may be determined that the deep neural network model is complete. Also, in the case of an incomplete deep neural network model, the image data receiving and processing device 200 may receive a training manual used to train the deep neural network model from the image data transmitting and processing device 100.

In step S819, if the received deep neural network model is not a complete deep neural network model, the image data receiving and processing device 200 uses the same deep neural network model used for compression as the deep neural network model used for training the deep neural network model. Build a neural network model.

In step S821, the image data receiving and processing device 200 uses the deep neural network model selected from the second storage unit, the deep neural network model received from the image data transmission and processing device 100, or the deep neural network model built using the received training manual. Compressed video data is restored to original video data using a neural network model.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present invention, it includes an invention in which individual values belonging to the range are applied (unless there is a description to the contrary), and each individual value constituting the range is described in the detailed description of the invention Same as

The steps constituting the method according to the present invention may be performed in any suitable order unless an order is explicitly stated or stated to the contrary. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and all scopes equivalent to or equivalently changed from the claims as well as the claims described below are within the scope of the spirit of the present invention. will be said to belong to

100: video data transmission processing device
200: image data receiving and processing device
300: network

Claims (20)

An apparatus for transmitting and processing video data, comprising:
a database for storing the image data;
a storage unit for storing deep neural network models for performing compression according to object characteristics according to circumstances and time included in the image data stored in the database;
a determination unit that determines whether a deep neural network model to be used for compression of the image data is stored in the storage unit based on information on image data to be transmitted;
As a result of the determination of the determination unit, as the deep neural network model to be used for the target image data to be transmitted is stored in the storage unit, a deep neural network model to compress the image data is selected from the storage unit, and the selected deep neural network model is used. a compression unit which generates compressed image data obtained by compressing the target image data to be transmitted; and
A transmission unit for transmitting the compressed video data to a video data receiving and processing device;
As a result of the determination of the determination unit, when the deep neural network model to be used for the target image data to be transmitted is not stored in the storage unit, image data similar to the target image data to be transmitted is selected from the database, and the similar image data similar to the selected similar image data is selected. Further comprising a generation unit for generating a new deep neural network model by learning the characteristics of an object included in the image data as training data,
Video data transmission processing device. According to claim 1,
The deep neural network models are training models trained to perform compression using pre-stored image data according to a user account that generates image data, and are created in the first user account to compress image data of the first user account. Including a first deep neural network model pretrained using existing image data as training data,
Video data transmission processing device. According to claim 1,
The information of the image data includes at least one of a user account that generates the image data, a time when the image data is generated, and a location where the image data is generated.
Video data transmission processing device. According to claim 1,
the storage unit,
Some of the trained deep neural network models may use image data having similar features as training data when learning the deep neural network model, and similarity information calculated by extracting features of the training data and the trained deep neural network You can save the model together,
The judge,
configured to determine whether a deep neural network model having similarity information most similar to the similarity information calculated by extracting features of the target image data to be transmitted is stored in the storage unit,
Video data transmission processing device. According to claim 1,
the compression unit,
Converting the target image data to be transmitted into image frame groups, each group including an intra frame and an inter frame, and compressing the target image data to be transmitted using the deep neural network model The intra frames in each image frame group are compressed by applying the deep neural network model, and the inter frames are compressed based on the intra frames to generate compressed image data.
Video data transmission processing device. According to claim 5,
The deep neural network model is one of a deep learning-based super resolution model or an auto encoder model,
Video data transmission processing device. delete According to claim 1,
the compression unit,
configured to generate compressed image data obtained by compressing the target image data to be transmitted using a deep neural network model generated by the generating unit;
the transmission unit,
configured to transmit the compressed image data and the deep neural network model generated by the generating unit to the image data receiving and processing device;
Video data transmission processing device. According to claim 1,
the transmission unit,
And configured to transmit the compressed image data and a training manual used when training the selected deep neural network model to the image data receiving and processing device.
Video data transmission processing device. delete delete delete delete As a method of transmitting and processing video data,
constructing a database for storing the image data;
constructing a storage unit for storing deep neural network models for performing compression for each characteristic of an object according to a situation and time included in the image data stored in the database;
determining whether a deep neural network model to be used for compressing the image data is stored in the storage unit based on information on target image data to be transmitted;
As a result of the determination, as the deep neural network model to be used for the target image data to be transmitted is stored in the storage unit, a deep neural network model to compress the image data is selected from the storage unit, and the selected deep neural network model is used. generating compressed image data obtained by compressing the target image data to be transmitted; and
Transmitting the compressed video data to a video data receiving and processing device;
As a result of the determination, if the deep neural network model to be used for the target image data to be transmitted is not stored in the storage unit, image data similar to the target image data to be transmitted is selected from the database, and the similar image data similar to the selected similar image data is selected. generating a new deep neural network model by learning characteristics of an object included in the image data as training data;
generating compressed image data obtained by compressing the target image data to be transmitted using a deep neural network model generated using the selected similar image data as training data; and
Transmitting the compressed image data and the deep neural network model generated by using the selected similar image data as training data to the image data receiving and processing device,
Video data transmission processing method. 15. The method of claim 14,
The deep neural network models are training models trained to perform compression using pre-stored image data according to a user account that generates image data, and are created in the first user account to compress image data of the first user account. Including a first deep neural network model pretrained using existing image data as training data,
Video data transmission processing method. 15. The method of claim 14,
Generating the compressed video data,
converting the target image data to be transmitted into image frame groups, each group including an intra frame and an inter frame;
In the process of compressing the target image data to be transmitted using the deep neural network model, compressing an intra frame in each image frame group by applying the deep neural network model; and
Generating compressed image data by compressing the inter frame based on the intra frame.
Video data transmission processing method. delete delete delete delete

Images