CN112637609B - Image real-time transmission method, sending end and receiving end - Google Patents
Image real-time transmission method, sending end and receiving end Download PDFInfo
- Publication number
- CN112637609B CN112637609B CN202011506940.XA CN202011506940A CN112637609B CN 112637609 B CN112637609 B CN 112637609B CN 202011506940 A CN202011506940 A CN 202011506940A CN 112637609 B CN112637609 B CN 112637609B
- Authority
- CN
- China
- Prior art keywords
- image
- residual
- blocks
- reconstruction
- reconstructed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000000638 solvent extraction Methods 0.000 claims abstract description 14
- 238000013139 quantization Methods 0.000 claims description 21
- 238000012549 training Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 15
- 230000000903 blocking effect Effects 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 10
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 230000001575 pathological effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention provides a real-time image transmission method, a sending end and a receiving end, wherein the method comprises the following steps: carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image; carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image; partitioning the residual image to obtain a plurality of residual image blocks; and sending the down-sampled image and part or all of the residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all of the residual image blocks. The image real-time transmission method, the transmitting end and the receiving end greatly reduce the data volume to be transmitted, further reduce the requirement of transmission bandwidth, improve the image transmission efficiency and realize the real-time transmission of large-size images. Meanwhile, on the basis of ensuring the real-time property of image transmission, the precision of the image corresponding to the region of interest can be ensured, and the browsing experience of the user is improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an image real-time transmission method, a sending end and a receiving end.
Background
The image real-time transmission is an important application in the technical field of image processing, and the existing image real-time transmission method is to firstly carry out image compression on an image to be transmitted by a sending end, for example, a lossless LZW and lossy JPEG compression method is adopted, the compressed image is transmitted to a receiving end, and then the receiving end carries out image restoration on the compressed image. However, for large-size images such as pathological scanning slices or remote sensing images, the data size is large, the transmission efficiency is low, and in order to ensure the real-time performance of transmission, the accuracy is generally required to be reduced for transmission, so that the accuracy of the restored images is not high.
Disclosure of Invention
The invention provides an image real-time transmission method, a sending end and a receiving end, which are used for solving the defects of low real-time transmission efficiency and low precision of large-size images in the prior art.
The invention provides a real-time image transmission method, which comprises the following steps:
carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image;
carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image;
partitioning the residual image to obtain a plurality of residual image blocks;
and sending the down-sampled image and part or all residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all residual image blocks.
According to the image real-time transmission method provided by the invention, the image reconstruction is performed on the downsampled image of the image to be transmitted to obtain a first reconstructed image, and the method comprises the following steps:
inputting the down-sampled image into a first image reconstruction model to obtain a first reconstructed image of the down-sampled image output by the first image reconstruction model;
the first image reconstruction model is obtained based on sample images and corresponding sample downsampling image training.
According to the image real-time transmission method provided by the invention, the step of sending the down-sampled image and part or all residual image blocks to a receiving end comprises the following steps:
and sending the downsampled image and a residual image block corresponding to a preset area to the receiving end.
The image real-time transmission method provided by the invention further comprises the following steps:
receiving a region image block request including a target region returned by the receiving terminal;
and sending the residual image block corresponding to the target area to the receiving end so that the receiving end can restore the image block corresponding to the target area in the image to be transmitted based on the residual image block corresponding to the target area.
According to the image real-time transmission method provided by the invention, the residual image is partitioned to obtain a plurality of residual image blocks, and then the method further comprises the following steps:
and based on the pixel gray distribution of each residual image block or the numerical value distribution of each color channel, reducing the quantization bit number of each residual image block, and recoding the pixel numerical value of each residual image block by adopting a new quantization bit number.
The invention also provides a real-time image transmission method, which comprises the following steps:
receiving a downsampled image and a plurality of residual image blocks sent by a sending end;
performing image restoration based on the downsampled image and the residual image blocks;
the residual image blocks are part or all of a plurality of residual image blocks obtained by partitioning a residual image, the residual image is obtained by carrying out subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by carrying out image reconstruction on the downsampled image by the sending end.
According to the image real-time transmission method provided by the invention, the image restoration based on the downsampled image and the residual image blocks comprises the following steps:
carrying out image reconstruction on the down-sampling image to obtain a second reconstructed image;
partitioning the second reconstructed image based on the residual image blocks to obtain initial reconstructed image blocks corresponding to the residual image blocks;
and carrying out image reconstruction based on each residual image block and the corresponding initial reconstruction image block to obtain a plurality of final reconstruction image blocks corresponding to the initial reconstruction image blocks.
According to a real-time image transmission method provided by the present invention, the image reconstruction is performed based on each residual image block and the corresponding initial reconstruction image block to obtain a plurality of final reconstruction image blocks corresponding to the plurality of initial reconstruction image blocks, and the method comprises:
inputting each residual image block and the initial reconstruction image block corresponding to the residual image block into a second image reconstruction model to obtain a plurality of final reconstruction image blocks output by the second image reconstruction model;
the second image reconstruction model is obtained by training based on the sample image block, the initial reconstruction image block corresponding to the sample image block and the sample residual image block.
The invention provides a real-time image transmission method, which further comprises the following steps:
determining a target area;
sending a region image block request containing the target region to the sending end so that the sending end can select and send a residual image block corresponding to the target region based on the region image block request;
and receiving a residual image block corresponding to the target area sent by the sending end, and restoring a corresponding image block of the target area in the image to be transmitted based on the residual image block corresponding to the target area.
The invention also provides a transmitting end, comprising:
the image reconstruction unit is used for reconstructing a down-sampling image of an image to be transmitted to obtain a first reconstructed image;
the residual image determining unit is used for carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image;
the image blocking unit is used for blocking the residual image to obtain a plurality of residual image blocks;
and the image sending unit is used for sending the down-sampled image and part or all of the residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all of the residual image blocks.
The present invention also provides a receiving end, comprising:
the image receiving unit is used for receiving the downsampled image and a plurality of residual image blocks sent by the sending end;
an image restoration unit, configured to perform image restoration based on the downsampled image and the residual image blocks;
the residual image blocks are part or all of a plurality of residual image blocks obtained by partitioning a residual image, the residual image is obtained by carrying out subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by carrying out image reconstruction on the downsampled image by the sending end.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of any one of the image real-time transmission methods.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image real-time transmission method as described in any one of the above.
According to the image real-time transmission method, the transmitting end and the receiving end, the image to be transmitted is decomposed into the downsampled image and the residual image blocks, the downsampled image and part or all of the residual image blocks are transmitted to the receiving end, so that the receiving end can restore the image to be transmitted based on the received downsampled image and part or all of the residual image blocks, the data volume to be transmitted is greatly reduced, the requirement on transmission bandwidth is further reduced, the image transmission efficiency is improved, and real-time transmission of large-size images can be achieved. Meanwhile, on the basis of ensuring the real-time property of image transmission, the precision of the image corresponding to the region of interest can be ensured, and the browsing experience of the user is improved.
Drawings
In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow chart of a real-time image transmission method provided by the present invention;
FIG. 2 is a second schematic flow chart of the image real-time transmission method provided by the present invention;
fig. 3 is a schematic structural diagram of a transmitting end provided by the present invention;
FIG. 4 is a schematic structural diagram of a receiving end provided in the present invention;
FIG. 5 is a schematic structural diagram of an electronic device provided by the present invention;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow diagram of an image real-time transmission method provided in an embodiment of the present invention, an execution main body of the image real-time transmission method provided in an embodiment of the present invention is a sending end, and the image real-time transmission method provided in an embodiment of the present invention can be applied to a remote conference, remote sensing satellite image transmission, and other scenes, as shown in fig. 1, the method includes:
110, reconstructing a downsampled image of an image to be transmitted to obtain a first reconstructed image;
and 140, sending the downsampled image and part or all of residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received downsampled image and part or all of residual image blocks.
Specifically, the image to be transmitted is an image that needs to be transmitted in real time, and the image to be transmitted may be a medical pathological image or a remote sensing image, and the like. After the image to be transmitted is determined, the transmitting end performs down-sampling operation on the image to be transmitted to obtain a down-sampled image, performs image reconstruction on the down-sampled image to obtain a first reconstructed image, and performs super-resolution reconstruction on the down-sampled image based on interpolation or a learning mode.
After the first reconstructed image is obtained, the sending end performs subtraction on the image to be transmitted and the first reconstructed image, namely, pixels at corresponding positions in the image to be transmitted and the first reconstructed image are subtracted to obtain a residual image, and the residual image is partitioned to obtain a plurality of residual image blocks. Here, the blocking method of the residual image may be determined based on the type of the image to be transmitted, for example, when the image to be transmitted is a medical pathological image, the residual image may be sliced into residual blocks of 128 × 128 in size; when the image to be transmitted is a remote sensing image, the residual image can be cut into residual image blocks with the size of 512 × 512.
Then, the sending end can send the downsampled image and all residual image blocks, or send the downsampled image and part of the residual image blocks to the receiving end. The residual image block to be transmitted may be determined based on request information of the receiving end, and the method for determining the residual image block to be transmitted in the embodiment of the present invention is not particularly limited.
Here, the sending end may simultaneously or separately transmit the downsampled image and the residual image block to be transmitted, which is not specifically limited in the embodiment of the present invention. In addition, before step 140 is executed, the downsampled image and the residual image block to be transmitted may be compressed, for example, by using an RLE compression method, and the compressed data is sent to the receiving end.
After receiving the downsampled image and the residual image blocks sent by the sending end, the receiving end can restore the image to be transmitted based on the downsampled image and the residual image blocks. Here, because the size of the down-sampled image is consistent with that of the image to be transmitted, the receiving end can perform overall rough restoration on the image to be transmitted based on the down-sampled image, and on the basis, can also perform local detail restoration on the image to be transmitted based on a residual image block.
The image to be transmitted is decomposed into a downsampled image and a plurality of residual image blocks through the sending end, the downsampled image and part or all of the residual image blocks are sent to the receiving end, so that the receiving end can restore the image to be transmitted based on the received downsampled image and the part or all of the residual image blocks, and the data volume of the downsampled image and the residual image blocks is far smaller than that of the image to be transmitted, so that the data volume to be transmitted is greatly reduced, the requirement on transmission bandwidth is reduced, the image transmission efficiency is improved, and the real-time transmission of large-size images can be realized.
In addition, the receiving end can carry out overall rough restoration on the image to be transmitted based on the downsampled image and carry out local detail restoration on the image to be transmitted based on the residual image block, on the basis of guaranteeing the real-time performance of image transmission, the precision of the image corresponding to the region of interest can be guaranteed, and the browsing experience of a user is improved.
According to the method provided by the embodiment of the invention, the image to be transmitted is decomposed into the downsampled image and the residual image blocks, and the downsampled image and part or all of the residual image blocks are sent to the receiving end, so that the receiving end can restore the image to be transmitted based on the received downsampled image and part or all of the residual image blocks, the data volume to be transmitted is greatly reduced, the requirement on transmission bandwidth is further reduced, the image transmission efficiency is improved, and the real-time transmission of large-size images can be realized. Meanwhile, on the basis of ensuring the real-time property of image transmission, the precision of the image corresponding to the region of interest can be ensured, and the browsing experience of the user is improved.
Based on the above embodiment, step 110 includes:
inputting the downsampled image into a first image reconstruction model to obtain a first reconstructed image of the downsampled image output by the first image reconstruction model;
the first image reconstruction model is obtained based on sample images and corresponding sample downsampling image training.
Specifically, after obtaining the down-sampled image, image reconstruction for the down-sampled image may be specifically implemented by a first image reconstruction model obtained through pre-training. The first image reconstruction model is used for performing super-resolution reconstruction on the down-sampled image and outputting a first reconstructed image of the down-sampled image.
Before step 110 is executed, the first image reconstruction model may be obtained through pre-training, and specifically, the first image reconstruction model may be obtained through training as follows: firstly, a large number of sample images are collected, and the down-sampling operation is carried out on the sample images to obtain sample down-sampled images corresponding to the sample images. And then, inputting the sample image and the corresponding sample downsampling image into the initial model for training, thereby obtaining a first image reconstruction model. The initial model may be constructed based on a generation countermeasure network (GAN).
Based on any of the above embodiments, step 130 includes:
and sending the downsampled image and a residual image block corresponding to the preset area to a receiving end.
Specifically, after obtaining a plurality of residual image blocks, the sending end may select a residual image block corresponding to the preset area from the preset area, and send the downsampled image and the residual image block corresponding to the preset area to the receiving end. The preset region may be a region where the occurrence probability of the target in which the user is interested is high, and the preset region may be determined through practical experience, for example, a central region of the image to be transmitted may be used as the preset region.
By determining the preset area in advance and simultaneously sending the downsampled image and the residual image blocks corresponding to the preset area to the receiving end, the receiving end can locally restore the image blocks corresponding to the preset area in the image to be transmitted, the message transmission time is saved, and the real-time performance of image restoration of the receiving end is further ensured.
Based on any of the above embodiments, the method further comprises:
receiving a region image block request including a target region returned by a receiving terminal;
and sending the residual image blocks corresponding to the target area to a receiving end so that the receiving end restores the image blocks corresponding to the target area in the image to be transmitted based on the residual image blocks corresponding to the target area.
Specifically, after receiving the down-sampled image sent by the sending end, the receiving end can perform overall rough restoration on the image to be transmitted based on the down-sampled image, and on the basis, a user can select an interested target area, for example, click through a mouse, and after determining the target area, the receiving end sends an area image block request containing the target area to the sending end.
After receiving a region image block request including a target region returned by a receiving terminal, a sending terminal selects a residual image block corresponding to the target region from a plurality of residual image blocks obtained by residual image blocking, and sends the residual image block corresponding to the target region to the receiving terminal.
After receiving the residual image block corresponding to the target area, the receiving end can restore the image block corresponding to the target area in the image to be transmitted based on the residual image block of the target area, so that the restoration of the local details of the user interested area can be realized. On the basis of guaranteeing the real-time property of image transmission, the precision of the image corresponding to the region of interest is guaranteed, and the browsing experience of a user is improved.
Based on any of the above embodiments, step 130 further includes:
and based on the pixel gray distribution of each residual image block or the numerical value distribution of each color channel, reducing the quantization bit number of each residual image block, and recoding the pixel numerical value of each residual image block by adopting a new quantization bit number.
Specifically, in order to further reduce the amount of data to be transmitted, after a plurality of residual image blocks are obtained, the quantization bits of the residual image blocks may be reduced based on the pixel gray scale distribution of any residual image block or the numerical value distribution of each color channel, and the pixel values of each residual image block are re-encoded by using a new quantization bit, so as to send the re-encoded residual image blocks to the receiving end. When the image to be transmitted is a gray image, the quantization bits of the residual image block can be determined again based on the pixel gray distribution of any residual image block; when the image to be transmitted is a color image, the quantization bit number of the residual image block can be determined again based on the numerical value distribution of each color channel of any residual image block, for example, three color channels of RGB. The pixel gray distribution of any residual image block is used for representing the distribution condition of the gray values of the pixels in the residual image block on values, and the pixel gray distribution of any residual image block or the value distribution of any residual image block on each color channel can be represented as a histogram. On the basis, if the variance of the histogram curve of the residual image block is larger, the more the quantization bits of the residual image block are determined again; if the variance of the histogram curve of the residual image block is smaller, the lower the quantization bit number of the residual image block is determined again.
Because the image level richness of the residual image is generally lower than that of the image to be transmitted, the new quantization bit number of each residual image block is smaller than the original quantization bit number, and the reduction of the quantization bit number can result in the reduction of the data size, for example, the original quantization bit number 7 of a residual image block with the size of 128 × 128, based on the pixel gray scale distribution of the residual image block, the newly determined quantization bit number of the residual image block is 5, the original data size of the residual image block is 128 × 128 × 7, and the data size of the residual image block after the re-quantization is 128 × 128 × 5.
In the embodiment of the invention, each residual image block is subjected to quantization again in a self-adaptive manner based on the pixel gray level distribution of each residual image block or the numerical value distribution of each color channel, each residual image block is further compressed, the data volume to be transmitted is further reduced, the requirement on transmission bandwidth is further reduced, and the image transmission efficiency is improved.
Based on any of the above embodiments, fig. 2 is a schematic flow chart of a real-time image transmission method provided by an embodiment of the present invention, where an execution subject of the real-time image transmission method provided by the embodiment of the present invention is a receiving end, as shown in fig. 2, the method includes:
220, restoring the image based on the down-sampled image and a plurality of residual image blocks;
the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on a downsampled image by a sending end.
Specifically, an image to be transmitted is an image to be transmitted in real time, after the image to be transmitted is determined, a sending end performs down-sampling operation on the image to be transmitted to obtain a down-sampled image, performs image reconstruction on the down-sampled image to obtain a first reconstructed image, performs subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image, and further performs blocking on the residual image to obtain a plurality of residual image blocks.
Then, the sending end can send the down-sampled image and all residual image blocks, or send the down-sampled image and part of the residual image blocks to the receiving end. The residual image block to be transmitted may be determined based on request information of the receiving end, and the method for determining the residual image block to be transmitted in the embodiment of the present invention is not particularly limited.
Here, the sending end may simultaneously or separately transmit the downsampled image and the residual image block to be transmitted, which is not specifically limited in the embodiment of the present invention.
After receiving the downsampled image and the residual image blocks sent by the sending end, the receiving end can restore the image to be transmitted based on the downsampled image and the residual image blocks. Here, because the size of the down-sampled image is consistent with that of the image to be transmitted, the receiving end can perform overall rough restoration on the image to be transmitted based on the down-sampled image, and on the basis, can also perform local detail restoration on the image to be transmitted based on a residual image block.
According to the method provided by the embodiment of the invention, the down-sampled image and the residual image blocks sent by the sending end are received, and the image restoration is carried out based on the down-sampled image and the residual image blocks, so that the data volume to be transmitted is greatly reduced, the requirement on transmission bandwidth is further reduced, the image transmission efficiency is improved, and the real-time transmission of large-size images can be realized. Meanwhile, on the basis of ensuring the real-time property of image transmission, the precision of the image corresponding to the region of interest can be ensured, and the browsing experience of the user is improved.
Based on any of the above embodiments, step 220 includes:
carrying out image reconstruction on the down-sampled image to obtain a second reconstructed image;
partitioning the second reconstructed image based on the residual image blocks to obtain a plurality of initial reconstructed image blocks corresponding to the residual image blocks;
and carrying out image reconstruction based on each residual image block and the corresponding initial reconstruction image block to obtain a plurality of final reconstruction image blocks corresponding to the initial reconstruction image blocks.
Specifically, after receiving the down-sampled image, the receiving end performs image reconstruction on the down-sampled image to obtain a second reconstructed image, where super-resolution reconstruction may be performed on the down-sampled image based on interpolation or a learning-based manner, and image reconstruction may also be performed based on a first image reconstruction model, and the operation method of the first image reconstruction model is referred to the above embodiments, which is not described herein again.
Then, the receiving end takes the image block at the position corresponding to each residual image block in the second reconstructed image as an initial reconstructed image block, so as to obtain a plurality of initial reconstructed image blocks corresponding to the residual image blocks one by one, and carries out image reconstruction based on each residual image block and the corresponding initial reconstructed image block, so as to obtain a plurality of final reconstructed image blocks corresponding to the initial reconstructed image blocks, so as to realize local detail restoration, wherein the image reconstruction can be carried out based on a deep learning algorithm.
In the embodiment of the invention, the image reconstruction is respectively carried out on the downsampled image and the initial reconstruction image block, so that a user can carry out overall rough browsing on the image to be transmitted based on the second reconstruction image block and carry out local detail browsing on the image to be transmitted based on the plurality of final reconstruction image blocks, and further the browsing experience of the user is improved.
Based on any of the above embodiments, the performing image reconstruction based on each residual image block and the initial reconstruction image block corresponding thereto to obtain a plurality of final reconstruction image blocks corresponding to a plurality of initial reconstruction image blocks includes:
inputting each residual image block and the initial reconstruction image block corresponding to the residual image block into a second image reconstruction model to obtain a plurality of final reconstruction image blocks output by the second image reconstruction model;
the second image reconstruction model is obtained by training based on the sample image block, the corresponding sample initial reconstruction image block and the sample residual image block.
Specifically, after each residual image block and the initial reconstruction image block corresponding to the residual image block are obtained, image reconstruction for the initial reconstruction image block can be specifically realized through a second image reconstruction model obtained through pre-training. Here, the second image reconstruction model is configured to perform super-resolution reconstruction based on each residual image block and the corresponding initial reconstruction image block, and output a final reconstruction image block corresponding to each initial reconstruction image block.
Here, the second image reconstruction model may also be obtained through pre-selection training, and specifically, the second image reconstruction model may be obtained through training in the following manner: firstly, a large number of sample images are collected, down-sampling operation is carried out on the sample images to obtain sample down-sampled images corresponding to the sample images, and subtraction operation is carried out on the sample images and the sample down-sampled images to obtain sample residual images. And performing image reconstruction on the sample downsampled image to obtain a sample initial reconstructed image, wherein the image reconstruction can be performed based on the first image reconstruction model. And partitioning the sample image, the sample initial reconstruction image and the sample residual image according to a unified method to obtain a sample image block, a corresponding sample initial reconstruction image block and a corresponding sample residual image block.
And then, inputting the sample image block, the corresponding sample initial reconstruction image block and the sample residual image block into the initial model for training, thereby obtaining a second image reconstruction model. The initial model may be constructed based on a generation countermeasure network (GAN).
According to the method provided by the embodiment of the invention, each residual image block and the corresponding initial reconstruction image block are input into the second image reconstruction model to obtain a plurality of final reconstruction image blocks output by the second image reconstruction model, so that the precision of the final reconstruction image blocks is improved, the real-time performance of large-size image transmission is ensured, the precision of the image is not lost, and the browsing experience of a user is improved.
According to any of the above embodiments, the method further comprises;
determining a target area;
sending the regional image block request containing the target region to a sending end so that the sending end can select and send residual image blocks corresponding to the target region based on the regional image block request;
and residual image blocks corresponding to the target area sent by the sending end are received, and the corresponding image blocks of the target area in the image to be transmitted are restored based on the residual image blocks corresponding to the target area.
Specifically, after receiving the downsampled image sent by the sending end, the receiving end may perform overall rough restoration of the image to be transmitted based on the downsampled image, and the user may select an interested target area on the basis, for example, click with a mouse, and after determining the target area, the receiving end sends an area pattern block request including the target area to the sending end.
After receiving a region image block request including a target region returned by a receiving terminal, a sending terminal selects a residual image block corresponding to the target region from a plurality of residual image blocks obtained by residual image blocking, and sends the residual image block corresponding to the target region to the receiving terminal.
After a receiving end receives a residual image block corresponding to a target area sent by a sending end, based on the residual image block of the target area, the corresponding image block of the target area in an image to be transmitted can be restored, and restoration of local details of a user interested area can be achieved. On the basis of guaranteeing the real-time property of image transmission, the precision of the image corresponding to the region of interest can be guaranteed, and the browsing experience of the user is improved.
Based on any of the above embodiments, the transmission process of the transmitting end and the receiving end includes the following steps:
after a downsampled image and a plurality of residual image blocks are obtained, a sending end sends the downsampled image to a receiving end, the receiving end inputs the received downsampled image to a first image reconstruction model to obtain a second reconstructed image, a user roughly browses the image to be transmitted by using the second reconstructed image and selects a target region from the image, the receiving end returns a region image block request containing the target region to the sending end, the sending end sends residual image blocks corresponding to the target region to the receiving end based on the region image block request, the receiving end inputs the received residual image blocks corresponding to the target region and initial reconstructed image blocks corresponding to the target region in the second reconstructed image to the second image reconstruction model to obtain final reconstructed image blocks corresponding to the target region, and the user can browse local detail information of the target region through the final reconstructed image blocks corresponding to the target region.
Based on any of the above embodiments, the transmission process of the transmitting end and the receiving end includes the following steps:
after a downsampled image and a plurality of residual image blocks are obtained, a sending end sends the downsampled image and the residual image blocks corresponding to a preset area to a receiving end, the receiving end inputs the received downsampled image to a first image reconstruction model to obtain a second reconstructed image, a user roughly browses the image to be transmitted by using the second reconstructed image and selects a target area from the first reconstructed image, and if the target area is within the range of the preset area, the receiving end locally calls the residual image blocks corresponding to the preset area to restore the target area.
If the target area is out of the range of the preset area, the receiving end returns an area image block request containing the target area to the sending end, the sending end sends a residual image block corresponding to the target area to the receiving end based on the area image block request, the receiving end inputs the received residual image block corresponding to the target area and an initial reconstruction image block corresponding to the target area in the second reconstructed image into the second image reconstruction model to obtain a final reconstruction image block corresponding to the target area, and a user can browse local detail information of the target area through the final reconstruction image block corresponding to the target area.
Based on any of the foregoing embodiments, fig. 3 is a schematic structural diagram of a sending end according to an embodiment of the present invention, and as shown in fig. 3, the sending end includes:
the image reconstruction unit 310 is configured to perform image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image;
a residual image determining unit 320, configured to perform subtraction on the image to be transmitted and the first reconstructed image to obtain a residual image;
an image blocking unit 330, configured to block the residual image to obtain a plurality of residual image blocks;
an image sending unit 340, configured to send the downsampled image and part or all of the residual image blocks to a receiving end, so that the receiving end restores the image to be transmitted based on the downsampled image and the part or all of the residual image blocks.
Based on any of the above embodiments, the image reconstruction unit 310 is configured to:
inputting the down-sampled image into a first image reconstruction model to obtain a first reconstructed image of the down-sampled image output by the first image reconstruction model;
the first image reconstruction model is obtained by training based on the sample image and the corresponding sample downsampling image.
Based on any of the above embodiments, the image sending unit 340 is configured to:
and sending the downsampled image and a residual image block corresponding to a preset area to the receiving end.
Based on any of the above embodiments, the sending end further includes:
a target area residual image block sending unit, configured to receive an area image block request including a target area, returned by the receiving end;
and sending the residual image block corresponding to the target area to the receiving end so that the receiving end can restore the image block corresponding to the target area in the image to be transmitted based on the residual image block corresponding to the target area.
Based on any of the above embodiments, the sending end further includes:
and the residual image block compression unit is used for reducing the quantization bit number of each residual image block based on the pixel gray distribution of each residual image block or the numerical value distribution of each color channel, and recoding the pixel numerical value of each residual image block by adopting a new quantization bit number.
Based on any of the above embodiments, fig. 4 is a schematic structural diagram of a receiving end provided in an embodiment of the present invention, and as shown in fig. 4, the receiving end includes:
an image receiving unit 410, configured to receive a downsampled image and a plurality of residual image blocks sent by a sending end;
an image restoration unit 420, configured to perform image restoration based on the downsampled image and the residual image blocks;
the residual image blocks are part or all of a plurality of residual image blocks obtained by partitioning a residual image, the residual image is obtained by carrying out subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by carrying out image reconstruction on the downsampled image by the sending end.
Based on any of the above embodiments, the image restoration unit 420 includes:
the down-sampling image reconstruction module is used for carrying out image reconstruction on the down-sampling image to obtain a second reconstructed image;
an initial reconstruction block determining module, configured to block the second reconstructed image based on the residual blocks to obtain initial reconstruction blocks corresponding to the residual blocks;
and the final reconstruction image block determining module is used for carrying out image reconstruction on the basis of each residual image block and the corresponding initial reconstruction image block to obtain a plurality of final reconstruction image blocks corresponding to the plurality of initial reconstruction image blocks.
Based on any of the above embodiments, the final reconstructed tile determination module is to:
inputting each residual image block and the initial reconstruction image block corresponding to the residual image block into a second image reconstruction model to obtain a plurality of final reconstruction image blocks output by the second image reconstruction model;
the second image reconstruction model is obtained by training based on the sample image block, the initial reconstruction image block corresponding to the sample image block and the sample residual image block.
Based on any of the above embodiments, the receiving end further includes:
a target area restoration unit for determining a target area;
sending a region image block request containing the target region to the sending end so that the sending end can select and send a residual image block corresponding to the target region based on the region image block request;
and receiving a residual image block corresponding to the target area sent by the sending end, and restoring a corresponding image block of the target area in the image to be transmitted based on the residual image block corresponding to the target area.
Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor) 510, a communication Interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may call logic instructions in memory 530 to perform the following method: carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image; carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image; partitioning the residual image to obtain a plurality of residual image blocks; and sending the down-sampled image and part or all of the residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all of the residual image blocks.
Further, processor 510 may call logic instructions in memory 530 to perform the following method: receiving a downsampled image and a plurality of residual image blocks sent by a sending end; based on the down-sampled image and a plurality of residual image blocks, restoring the image; the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on a downsampled image by a sending end.
In addition, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the methods provided by the methods described above, for example, including: carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image; carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image; partitioning the residual image to obtain a plurality of residual image blocks; and sending the down-sampled image and part or all of the residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all of the residual image blocks.
Embodiments of the present invention also provide a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, enable the computer to perform the methods provided by the above-mentioned method embodiments, for example, including: receiving a downsampled image and a plurality of residual image blocks sent by a sending end; based on the down-sampled image and a plurality of residual image blocks, restoring the image; the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on a down-sampling image by a sending end.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program that, when executed by a processor, is implemented to perform the methods provided above, for example, comprising: carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image; carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image; partitioning the residual image to obtain a plurality of residual image blocks; and sending the down-sampled image and part or all of the residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all of the residual image blocks.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided in the foregoing embodiments when executed by a processor, and the method includes: receiving a downsampled image and a plurality of residual image blocks sent by a sending end; based on the down-sampled image and a plurality of residual image blocks, restoring the image; the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on a downsampled image by a sending end.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (11)
1. A method for real-time image transmission, comprising:
carrying out image reconstruction on a downsampled image of an image to be transmitted to obtain a first reconstructed image;
carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image;
partitioning the residual image to obtain a plurality of residual image blocks;
sending the down-sampled image and part or all residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received down-sampled image and part or all residual image blocks;
further comprising:
receiving a region image block request including a target region returned by the receiving terminal; the target area is selected by a user from a second reconstructed image, and the second reconstructed image is obtained by image reconstruction of the downsampled image by the receiving end;
and sending the residual image block corresponding to the target area to the receiving end so that the receiving end can restore the image block corresponding to the target area in the image to be transmitted based on the residual image block corresponding to the target area.
2. The method for transmitting the image in real time according to claim 1, wherein the image reconstruction of the downsampled image of the image to be transmitted to obtain a first reconstructed image comprises:
inputting the down-sampled image into a first image reconstruction model to obtain a first reconstructed image of the down-sampled image output by the first image reconstruction model;
the first image reconstruction model is obtained by training based on the sample image and the corresponding sample downsampling image.
3. The method for transmitting the image in real time according to claim 1, wherein the sending the downsampled image and part or all of the residual image blocks to a receiving end comprises:
and sending the downsampled image and a residual image block corresponding to a preset area to the receiving end.
4. The method for real-time image transmission according to any one of claims 1 to 3, wherein the blocking the residual image to obtain a plurality of residual blocks further comprises:
and based on the pixel gray distribution of each residual image block or the numerical value distribution of each color channel, reducing the quantization bit number of each residual image block, and recoding the pixel numerical value of each residual image block by adopting a new quantization bit number.
5. A method for real-time image transmission, comprising:
receiving a downsampled image and a plurality of residual image blocks sent by a sending end;
performing image restoration based on the downsampled image and the residual image blocks;
the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on the downsampled image by the sending end;
further comprising:
determining a target area; the target area is selected by a user from a second reconstructed image, and the second reconstructed image is obtained by image reconstruction of the downsampled image by the receiving end;
sending a region image block request containing the target region to the sending end so that the sending end can select and send a residual image block corresponding to the target region based on the region image block request;
and receiving a residual image block corresponding to the target area sent by the sending end, and restoring a corresponding image block of the target area in the image to be transmitted based on the residual image block corresponding to the target area.
6. The method for real-time image transmission according to claim 5, wherein the image restoration based on the downsampled image and the residual image blocks comprises:
carrying out image reconstruction on the down-sampling image to obtain a second reconstructed image;
partitioning the second reconstructed image based on the residual image blocks to obtain a plurality of initial reconstructed image blocks corresponding to the residual image blocks;
and carrying out image reconstruction based on each residual image block and the corresponding initial reconstruction image block to obtain a plurality of final reconstruction image blocks corresponding to the initial reconstruction image blocks.
7. The method of claim 6, wherein the reconstructing the image based on each residual block and the corresponding initial reconstructed block to obtain a plurality of final reconstructed blocks corresponding to the initial reconstructed blocks comprises:
inputting each residual image block and the initial reconstruction image block corresponding to the residual image block into a second image reconstruction model to obtain a plurality of final reconstruction image blocks output by the second image reconstruction model;
the second image reconstruction model is obtained by training based on the sample image block, the initial reconstruction image block corresponding to the sample image block and the sample residual image block.
8. A transmitting end, comprising:
the image reconstruction unit is used for reconstructing a downsampled image of an image to be transmitted to obtain a first reconstructed image;
the residual image determining unit is used for carrying out subtraction operation on the image to be transmitted and the first reconstructed image to obtain a residual image;
the image blocking unit is used for blocking the residual image to obtain a plurality of residual image blocks;
the image sending unit is used for sending the downsampled image and part or all of residual image blocks to a receiving end so that the receiving end can restore the image to be transmitted based on the received downsampled image and the part or all of residual image blocks;
further comprising a request receiving unit for:
receiving a region image block request including a target region returned by the receiving end; the target area is selected by a user from a second reconstructed image, and the second reconstructed image is obtained by image reconstruction of the downsampled image by the receiving end;
and sending the residual image blocks corresponding to the target area to the receiving end so that the receiving end can restore the image blocks corresponding to the target area in the image to be transmitted based on the residual image blocks corresponding to the target area.
9. A receiving end, comprising:
the image receiving unit is used for receiving the downsampled image and a plurality of residual image blocks sent by the sending end;
the image restoration unit is used for restoring an image based on the downsampled image and the residual image blocks;
the residual image is obtained by performing subtraction operation on an image to be transmitted and a first reconstructed image, and the first reconstructed image is obtained by performing image reconstruction on the downsampled image by the sending end;
further comprising a target tile restoration unit for:
determining a target area; the target area is selected from a second reconstructed image by a user, and the second reconstructed image is obtained by image reconstruction of the downsampled image by the receiving end;
sending the region image block request containing the target region to the sending end so that the sending end can select and send a residual image block corresponding to the target region based on the region image block request;
and receiving a residual image block corresponding to the target area sent by the sending end, and restoring a corresponding image block of the target area in the image to be transmitted based on the residual image block corresponding to the target area.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the image real-time transmission method according to any one of claims 1 to 7 when executing the program.
11. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for real-time transmission of images according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011506940.XA CN112637609B (en) | 2020-12-18 | 2020-12-18 | Image real-time transmission method, sending end and receiving end |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011506940.XA CN112637609B (en) | 2020-12-18 | 2020-12-18 | Image real-time transmission method, sending end and receiving end |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112637609A CN112637609A (en) | 2021-04-09 |
| CN112637609B true CN112637609B (en) | 2023-03-17 |
Family
ID=75317311
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011506940.XA Active CN112637609B (en) | 2020-12-18 | 2020-12-18 | Image real-time transmission method, sending end and receiving end |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112637609B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115797750B (en) * | 2023-02-02 | 2023-04-25 | 天津滨海迅腾科技集团有限公司 | Large-size image rapid transmission method based on deep learning algorithm |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6937772B2 (en) * | 2000-12-20 | 2005-08-30 | Eastman Kodak Company | Multiresolution based method for removing noise from digital images |
| CN105184759B (en) * | 2015-09-22 | 2018-02-27 | 中国科学院西安光学精密机械研究所 | Image self-adaptive enhancement method based on histogram compactness transformation |
| CN106204617B (en) * | 2016-07-21 | 2018-10-02 | 大连海事大学 | Adapting to image binarization method based on residual image histogram cyclic shift |
| CN107454412B (en) * | 2017-08-23 | 2020-03-06 | 合肥美菱物联科技有限公司 | Video image processing method, device and system |
| CN110636289B (en) * | 2019-09-27 | 2021-11-05 | 北京金山云网络技术有限公司 | Image data transmission method, system, device, electronic equipment and storage medium |
-
2020
- 2020-12-18 CN CN202011506940.XA patent/CN112637609B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112637609A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2916543A1 (en) | Method for coding/decoding depth image and coding/decoding device | |
| WO2019012363A1 (en) | Visual quality preserving quantization parameter prediction with deep neural network | |
| CN109146813B (en) | Multitask image reconstruction method, device, equipment and medium | |
| WO2017190691A1 (en) | Picture compression method and apparatus | |
| KR101805550B1 (en) | Image data encoding method for presentation virtualization and server therefor | |
| CN113421312B (en) | Coloring method and device for black-and-white video, storage medium and terminal | |
| CN109191411B (en) | Multitask image reconstruction method, device, equipment and medium | |
| CN110852980A (en) | Interactive image filling method and system, server, device and medium | |
| EP4300958A1 (en) | Video image encoding method, video image decoding method and related devices | |
| CN113888410A (en) | Image super-resolution method, apparatus, device, storage medium, and program product | |
| CN112637609B (en) | Image real-time transmission method, sending end and receiving end | |
| CN111145202B (en) | Model generation method, image processing method, device, equipment and storage medium | |
| CN111083494A (en) | Video coding method and device and terminal equipment | |
| CN115209150B (en) | Video coding parameter acquisition method and device and electronic equipment | |
| CN115941966A (en) | Video compression method and electronic equipment | |
| CN114663536B (en) | Image compression method and device | |
| CN113096019B (en) | Image reconstruction method, image reconstruction device, image processing equipment and storage medium | |
| CN111553961B (en) | Method and device for acquiring line manuscript corresponding color map, storage medium and electronic device | |
| WO2022226850A1 (en) | Point cloud quality enhancement method, encoding and decoding methods, apparatuses, and storage medium | |
| CN113766319A (en) | Image information processing method and device, and storage medium | |
| CN114792284A (en) | Image switching method and device, storage medium and electronic equipment | |
| CN113554719A (en) | Image encoding method, decoding method, storage medium and terminal equipment | |
| CN112887726B (en) | Image processing method and apparatus, storage medium, and electronic device | |
| CN114286113B (en) | Image compression recovery method and system based on multi-head heterogeneous convolution self-encoder | |
| CN116708793B (en) | Video transmission method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 201114 room 1302, 13 / F, building 16, 2388 Chenhang Road, Minhang District, Shanghai Patentee after: Shanghai Bi Ren Technology Co.,Ltd. Country or region after: China Address before: 201114 room 1302, 13 / F, building 16, 2388 Chenhang Road, Minhang District, Shanghai Patentee before: Shanghai Bilin Intelligent Technology Co.,Ltd. Country or region before: China |