CN113573059A - Image display method, image display apparatus, storage medium, and electronic apparatus - Google Patents
Image display method, image display apparatus, storage medium, and electronic apparatus Download PDFInfo
- Publication number
- CN113573059A CN113573059A CN202111111256.6A CN202111111256A CN113573059A CN 113573059 A CN113573059 A CN 113573059A CN 202111111256 A CN202111111256 A CN 202111111256A CN 113573059 A CN113573059 A CN 113573059A
- Authority
- CN
- China
- Prior art keywords
- target
- image
- target data
- data
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/127—Prioritisation of hardware or computational resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The embodiment of the invention provides an image display method, an image display device, a storage medium and an electronic device, wherein the method comprises the following steps: under the condition that a target image is received, extracting a target area contained in the target image, wherein the target area comprises an area where information of a target type is located; carrying out hardware coding processing on a target image to obtain first target data; extracting information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data; packaging the first target data and the second target data to obtain a target data packet; and sending the target data packet to the terminal to instruct the terminal to display the target image based on the target data packet. The invention solves the problem of high cost caused by the need of increasing hardware resources in the related technology, and achieves the effect of further reducing the data volume and the transmission bandwidth while displaying in high fidelity.
Description
Technical Field
The invention relates to the technical field of computers, in particular to an image display method, an image display device, a storage medium and an electronic device.
Background
In a cloud computer scene, in order to achieve user experience consistent with that of a common computer, a higher requirement is put forward on color display, namely a high-fidelity lossless display effect. Hardware coding and decoding are widely applied to various commercial streaming media systems, and the characteristics of high speed, high decoding efficiency, no occupation of a CPU and the like are widely accepted. At present, a mainstream coding and decoding chip only supports a YUV420 format, the format performs down-sampling on original data, data distortion is caused, the high-definition display requirement of a cloud computer scene is not met, and for the current situation that the mainstream coding and decoding chip only supports a YUV420 lossy format in the industry, a method for using two paths of YUV420 format data to bear lossless data code streams is mentioned in the related art. Therefore, the high-efficiency and high-fidelity nondestructive imaging is urgently required by design.
Aiming at the problem that the high-fidelity display in the related technology needs to depend on the increase of hardware resources to cause high cost, an effective solution is not provided at present.
Disclosure of Invention
The embodiment of the invention provides an image display method, an image display device, a storage medium and an electronic device, which are used for at least solving the problem of high cost caused by the fact that high-fidelity display needs to depend on increased hardware resources in the related technology.
According to an aspect of an embodiment of the present invention, there is provided an image display method including: under the condition that a target image is received, extracting a target area contained in the target image, wherein the target area comprises an area where information of a target type is located; performing hardware coding processing on the target image to obtain first target data; extracting information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data; packaging the first target data and the second target data to obtain a target data packet; and sending the target data packet to a terminal to instruct the terminal to display the target image based on the target data packet.
In one exemplary embodiment, extracting the target region included in the target image includes: partitioning the region of the target image according to a classification algorithm to obtain a plurality of block regions; and determining the target area from a plurality of block areas.
In an exemplary embodiment, the encoding the compressed data to obtain the second target data includes: judging whether the terminal feeds back a frame rate message, wherein the frame rate message is used for indicating the frequency of a frame image sent by a display server of the terminal; under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is larger than or equal to a preset frame rate threshold value, performing hardware coding processing on the compressed data to obtain second target data; under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is smaller than the preset frame rate threshold, performing software coding processing on the compressed data to obtain second target data; and under the condition that the frame rate message is not fed back by the terminal, performing the hardware coding processing on the compressed data to obtain second target data.
In an exemplary embodiment, the hardware encoding processing the target image to obtain the first target data includes: carrying out down-sampling processing on the target image to obtain a sampled image; and carrying out the hardware coding processing on the sampling image to obtain the first target data.
According to another aspect of the embodiments of the present invention, there is also provided an image display method including: receiving a target data packet sent by a server, wherein the target data packet is obtained by encapsulating first target data and second target data by the server, the first target data is obtained by hardware coding processing of a target image by the server, the second target data is obtained by compressing information in a target area by the server and coding the compressed data obtained after compression, and the target area is an area which includes information of a target type and is included in the target image; decapsulating the target data packet to obtain the first target data and the second target data; performing hardware decoding processing on the first target data to obtain the target image; decompressing the second target data to obtain information in the target area; recombining the target image and the information in the target area to obtain a recombined image; and displaying the recombined image.
In one exemplary embodiment, displaying the recombined image includes: and overhauling the recombined image, and displaying the overhauled recombined image.
In one exemplary embodiment, the method further comprises: and counting the frequency of the frame image sent by the display server, and feeding back frame rate information to the server.
In an exemplary embodiment, the reconstructing the target image and the information in the target region to obtain a reconstructed image includes: restoring the image data in the target area based on the information in the target area to obtain an image of the target area; and recombining the target image and the image in the target area to obtain the recombined image.
In an exemplary embodiment, decompressing the second target data to obtain the information in the target area includes: decoding the second target data to obtain compressed data; and performing the decompression processing on the compressed data to obtain the information in the target area.
According to still another aspect of an embodiment of the present invention, there is also provided an image display apparatus including: the device comprises a first extraction module, a second extraction module and a third extraction module, wherein the first extraction module is used for extracting a target area contained in a target image under the condition that the target image is received, and the target area comprises an area where information of a target type is located; the first coding module is used for carrying out hardware coding processing on the target image to obtain first target data; the second extraction module is used for extracting the information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data; the encapsulation module is used for encapsulating the first target data and the second target data to obtain a target data packet; and the sending module is used for sending the target data packet to a terminal so as to instruct the terminal to display the target image based on the target data packet.
According to still another aspect of an embodiment of the present invention, there is also provided an image display apparatus including: a receiving module, configured to receive a target data packet sent by a server, where the target data packet is obtained by encapsulating, by the server, first target data and second target data, the first target data is obtained by performing hardware coding processing on a target image by the server, the second target data is obtained by compressing, by the server, information in a target area, and encoding compressed data obtained after compression, and the target area is an area that includes information of a target type and is included in the target image; a decapsulation module, configured to decapsulate the target data packet to obtain the first target data and the second target data; the first decoding module is used for performing hardware decoding processing on the first target data to obtain the target image; the decompression module is used for decompressing the second target data to obtain the information in the target area; the recombination module is used for recombining the target image and the information in the target area to obtain a recombined image; and the display module is used for displaying the recombined image.
According to a further aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the steps in any one of the method embodiments through the computer program.
According to the invention, the first target data is obtained by hardware coding processing of the target image, the information in the target area in the target image is extracted and compressed and coded to obtain the second target data, then the first target data and the second target data are packaged to obtain the target data packet, and the target data packet is sent to the terminal to instruct the terminal to display the target image based on the target data packet. By adopting the technical scheme, the information in the target area is compressed and coded, and the flexible coding processing method is adopted for different areas of the target image, so that the problems that only a hardware coding mode is adopted and hardware resources need to be increased in the related technology are solved, the problems that the coding mode is single and the cost is high due to the fact that the hardware resources need to be increased in the related technology are solved, and the effects of further reducing the data volume and the transmission bandwidth while having high-fidelity display are achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a terminal hardware configuration of an image display method of an embodiment of the present invention;
FIG. 2 is a flowchart of a first image display method according to an embodiment of the present invention;
FIG. 3 is a flowchart of an image display method according to an embodiment of the present invention;
FIG. 4 is a flow chart of an image display method according to an embodiment of the invention;
FIG. 5 is a schematic diagram of a frame rate feedback process according to an embodiment of the present invention;
FIG. 6 is a block diagram of an image display system in accordance with an embodiment of the present invention;
FIG. 7 is a block diagram showing the first configuration of an image display apparatus according to an embodiment of the present invention;
fig. 8 is a block diagram of the structure of an image display apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The method embodiments provided in the embodiments of the present application may be executed in a terminal, a server, for example, a mobile terminal, a computer terminal, a server, or a similar computing device. Taking the operation on the terminal as an example, fig. 1 is a block diagram of a hardware structure of the terminal of the image display method according to the embodiment of the present invention. As shown in fig. 1, the terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and in an exemplary embodiment, may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the terminal. For example, the terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store a computer program, for example, a software program and a module of application software, such as a computer program corresponding to the image display method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In the present embodiment, an image display method is provided, and fig. 2 is a flowchart of an image display method according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, under the condition that a target image is received, extracting a target area contained in the target image, wherein the target area comprises an area where information of a target type is located;
step S204, carrying out hardware coding processing on the target image to obtain first target data;
step S206, extracting the information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data;
step S208, packaging the first target data and the second target data to obtain a target data packet;
step S210, sending the target data packet to a terminal to instruct the terminal to display the target image based on the target data packet.
Through the steps, the first target data is obtained by carrying out hardware coding processing on the target image, the information in the target area in the target image is extracted and compressed and coded to obtain the second target data, then the first target data and the second target data are packaged to obtain the target data packet, and the target data packet is sent to the terminal to indicate the terminal to display the target image based on the target data packet. By adopting the technical scheme, the information in the target area is compressed and coded, and the flexible coding processing method is adopted for different areas of the target image, so that the problems that only a hardware coding mode is adopted and hardware resources need to be increased in the related technology are solved, the problems that the coding mode is single and the cost is high due to the fact that the hardware resources need to be increased in the related technology are solved, and the effects of further reducing the data volume and the transmission bandwidth while having high-fidelity display are achieved.
The execution main body of the above steps may be a server, or a cloud, for example, a server in a hi-fi display system, and the execution main body of the above steps may also be other processing devices or processing units with similar processing capabilities, but is not limited thereto. The following description takes the server side to perform the above operation as an example (which is only an exemplary description, and in actual operation, other devices or modules may also perform the above operation):
in the above embodiment, the server extracts the target area included in the target image when receiving the target image, in practical applications, the target area may be a display area with higher sensitivity, for example, a text area with higher requirement for definition, a software drawing area, etc., the server performs a hardware encoding process on the target image to obtain first target data, for example, frame data of the target image may be encoded by using H264, H265, VP9 standards to obtain first target data, and extracts information in the target area and performs a compression and encoding process on the information to obtain second target data, in practical applications, the compression algorithm selected includes, but is not limited to Zlib/Oodle/ZSTD, etc., extracts auxiliary enhanced image quality in the target area according to practical situations such as definition, bandwidth, CPU hardware resources, etc., or called as an auxiliary stream, and then performs encoding processing on the auxiliary stream, for example, hardware encoding or software encoding may be performed on the auxiliary stream according to actual conditions to obtain second target data; then, packaging and encapsulating the first target data and the second target data to obtain a target data packet, wherein in practical application, the packaging mode can be selected, and if an H264 protocol is adopted, an H264 nal packet is used for packaging; if H265 is adopted, the nal packet of H265 is used; and then sending the target data packet to the terminal to instruct the terminal to display the target image based on the target data packet, for example, in practical application, the target data packet may be sent to the terminal through a network transport protocol such as TCP/UDP. By adopting the technical scheme, the information in the target area is compressed and coded, and the flexible coding processing method is adopted for different areas of the target image, so that the problems that only a hardware coding mode is adopted and hardware resources need to be increased in the related technology are solved, the problems that the coding mode is single and the cost is high due to the fact that the hardware resources need to be increased in the related technology are solved, and the effects of further reducing the data volume and the transmission bandwidth while having high-fidelity display are achieved.
In an optional embodiment, extracting the target region included in the target image includes: partitioning the region of the target image according to a classification algorithm to obtain a plurality of block regions; and determining the target area from a plurality of block areas. In this embodiment, the region of the entire target image is partitioned according to a classification algorithm, and the target region is determined from the multiple block regions, for example, the available classification algorithm includes image edge gradient, RGB histogram statistics, and the like, so as to obtain a visual high-sensitivity region, and the visual high-sensitivity region may be determined as the target region, for example, in practical applications, the visual high-sensitivity region may include: a color text area, a tool software drawing area, such as AutoCAD, model sim, etc., or a background color area closer to the color text area or the tool software drawing area, etc. According to the embodiment, the purpose of blocking the region of the target image is achieved, the hardware coding or compression and coding processing mode is flexibly adopted according to the display requirements of different regions, the purpose of flexibly coding the target image is achieved, the effect of high-fidelity display is achieved, and meanwhile the effects of reducing the data volume and the transmission bandwidth are achieved.
In an optional embodiment, the encoding the compressed data to obtain the second target data includes: judging whether the terminal feeds back a frame rate message, wherein the frame rate message is used for indicating the frequency of a frame image sent by a display server of the terminal; under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is larger than or equal to a preset frame rate threshold value, performing hardware coding processing on the compressed data to obtain second target data; under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is smaller than the preset frame rate threshold, performing software coding processing on the compressed data to obtain second target data; and under the condition that the frame rate message is not fed back by the terminal, performing the hardware coding processing on the compressed data to obtain second target data. In this embodiment, according to different situations of the terminal feedback frame rate, different encoding processing manners are adopted for the compressed data to obtain second target data, which specifically includes: when the terminal has a feedback frame rate and the frame rate is greater than or equal to a preset threshold (for example, 30fps, or 50 fps, or other frame rate thresholds), performing hardware coding processing on the compressed data to obtain second target data; when the terminal has a feedback frame rate and the frame rate is smaller than a preset threshold (for example, 30fps, or 50 fps, or other frame rate thresholds), performing software coding processing on the compressed data to obtain second target data; and when the terminal does not feed back the frame rate, the hardware coding processing is also carried out on the compressed data to obtain second target data. That is, the frame rate determines whether to adopt software coding processing or hardware coding processing, and when the frame rate is low, it indicates that the hardware resources are not enough, software coding is used, and when the frame rate is high, it indicates that the hardware capability is enough, hardware coding is performed.
In an optional embodiment, the hardware encoding processing on the target image to obtain the first target data includes: carrying out down-sampling processing on the target image to obtain a sampled image; and carrying out the hardware coding processing on the sampling image to obtain the first target data. In this embodiment, a target image is downsampled to obtain a sampled image, and then the sampled image is subjected to hardware encoding to obtain first target data, in practical applications, in a cloud desktop scene, for an image in a region with low visual sensitivity, hardware encoding processing may be performed, for example, the frame data of the target image may be encoded according to H264, H265, and VP9 standards to obtain the first target data. By the embodiment, the purpose of performing hardware coding processing on the target image is achieved.
In this embodiment, an image display method is further provided, and fig. 3 is a flowchart of a second image display method according to an embodiment of the present invention, as shown in fig. 3, the flowchart includes the following steps:
step S302, receiving a target data packet sent by a server, where the target data packet is obtained by encapsulating, by the server, first target data and second target data, the first target data is obtained by performing hardware coding processing on a target image by the server, the second target data is obtained by compressing, by the server, information in a target area, and coding compressed data obtained after compression, and the target area is an area including information of a target type included in the target image;
step S304, decapsulating the target data packet to obtain the first target data and the second target data;
step S306, carrying out hardware decoding processing on the first target data to obtain the target image;
step S308, decompressing the second target data to obtain the information in the target area;
step S310, recombining the target image and the information in the target area to obtain a recombined image;
and step S312, displaying the recombined image.
Through the steps, a target data packet sent by the server is received, the target data packet is obtained by encapsulating first target data and second target data by the server, wherein the first target data is obtained by performing hardware coding processing on a target image by the server, the second target data is obtained by performing compression and coding processing on information in a target area contained in the target image by the server, the target data packet is unpacked to obtain the first target data and the second target data, the first target data and the second target data are respectively subjected to hardware decoding processing and uncompressing processing to obtain the target image and the information in the target area, then the target image and the information in the target area are recombined to obtain a recombined image, and the recombined image is displayed. The method achieves the purpose of restoring the image data in the target area and recombining the target image and the image data in the target area, thereby achieving the effect of high-fidelity display. The problem of high cost caused by the fact that hardware resources need to be increased in high-fidelity display in the related technology is solved.
The main body of the above steps may be a terminal or a client, for example, a terminal in a hi-fi display system, or other processing devices or processing units with similar processing capabilities, but is not limited thereto. The following description is given by taking the terminal as an example (which is only an exemplary description, and in actual operation, other devices or modules may also be used to perform the above operations):
in the above embodiment, the terminal performs decapsulation processing on the received target data packet, performs hardware decoding and decompression processing on the first target data and the second target data obtained through the decapsulation processing, to obtain the target image and the information in the target area, reassembles the target image and the information in the target area to obtain a reassembled image, and displays the reassembled image by the terminal. The problem of high cost caused by the fact that hardware resources need to be increased in high-fidelity display in the related technology is solved.
In an alternative embodiment, displaying the recombined image includes: and overhauling the recombined image, and displaying the overhauled recombined image. In this embodiment, after the target image and the information in the target area are reconstructed to obtain a reconstructed image, the reconstructed image is self-checked and modified, and then the reconstructed image after maintenance is displayed. Through this embodiment, realized the purpose that redisplays after overhauing the reorganization image.
In an optional embodiment, the method further comprises: and counting the frequency of the frame image sent by the display server, and feeding back frame rate information to the server. In this embodiment, the terminal counts the frequency of the frame image sent by the display server, and feeds back the frame rate information to the server. By the embodiment, the purpose of feeding back the frame rate to the server side by the terminal is achieved.
In an optional embodiment, the reconstructing the target image and the information in the target region to obtain a reconstructed image includes: restoring the image data in the target area based on the information in the target area to obtain an image of the target area; and recombining the target image and the image in the target area to obtain the recombined image. In this embodiment, the image data of the target region is restored according to the information in the target region to obtain an image of the target region, and then the target image and the image in the target region are recombined to obtain a recombined image. Through the embodiment, the aim of recombining the target image and the information in the target area is fulfilled.
In an optional embodiment, the decompressing the second target data to obtain the information in the target area includes: decoding the second target data to obtain compressed data; and performing the decompression processing on the compressed data to obtain the information in the target area. In this embodiment, the decompressing the second target data includes: and decoding the second target data to obtain compressed data, decompressing the compressed data, and finally obtaining the information in the target area.
It is to be understood that the above-described embodiments are only a few, but not all, embodiments of the present invention. The present invention will be described in detail with reference to the following specific examples:
fig. 4 is a flowchart of an image display method according to an embodiment of the present invention, as shown in fig. 4, the flowchart includes the following steps:
step S402, inputting RGB data of an original image frame into a system server of the application, firstly partitioning the whole image (corresponding to the target image) through a region identification module of the application, and distinguishing a high-sensitivity region and a low-sensitivity region visually, wherein a specific classification algorithm includes but is not limited to image edge gradient, RGB histogram statistics and the like to obtain the high-sensitivity region visually, and in practical application, the high-sensitivity region includes but is not limited to: 1) a color character area; 2) tool software mapping areas, such as AutoCAD, modelSIM, etc.; 3) and 1) and 2) the background color area close to the area;
step S404, performing hardware encoding on the original image frame data, inputting the original image frame data into a hardware encoder for encoding, wherein the encoder comprises but is not limited to H.264, H.265, VP9 and the like;
step S406, extracting enhancement information, and extracting information (or called auxiliary stream) for assisting in enhancing the image quality for the high-sensitivity area;
it should be noted that, in step S406, compression and encoding are included, and different encoding processing manners are adopted for the compressed data according to different situations of the terminal feedback frame rates, for example, (1) if there is a feedback frame rate, and if the frame rate is greater than a threshold (corresponding to the preset threshold, or frame rate threshold), hardware encoding is continuously performed on the auxiliary stream; if the frame rate is less than the threshold value, the auxiliary stream is subjected to software coding; (2) if no frame rate feedback message exists, hardware coding is carried out on the auxiliary stream;
step S408, packaging and encapsulating the data (corresponding to the first target data) of step S404 and the auxiliary stream data (corresponding to the second target data) of step S406 to form a data packet;
step S410, the data packet generated in step S408 is sent through a network transmission protocol such as TCP/UDP;
step S412, the client (corresponding to the terminal) receives the network data, and decapsulates the network data according to the packaging format (i.e. corresponding to the decapsulation in fig. 4);
step S414, according to the corresponding decoder used by the encoder, the hardware decoding is performed on the whole frame data unpacked in step S412;
step S416, decompressing the auxiliary image quality enhancement information, and restoring the image data of the high-sensitivity area according to the auxiliary information;
step S418, the high-sensitivity area and the original frame image are recombined, and a display image frame is output, wherein it needs to be explained that self-checking modification can be carried out after recombination, and then the recombined image after maintenance is displayed;
step S420, the client feeds back the frame rate to the server.
It should be noted that, in the step S406, different encoding processing manners may be adopted for the auxiliary stream according to different situations of the frame rate fed back by the terminal, as shown in fig. 5, fig. 5 is a schematic diagram of a frame rate feedback process according to an embodiment of the present invention, where the process includes the following steps:
s502, judging whether the client feeds back frame rate information or not;
s504, under the condition that the client is judged to have the frame rate feedback information, further judging whether the frame rate is larger than a threshold value;
s506, under the condition that the frame rate is judged to be smaller than the threshold value, software coding is carried out on the auxiliary stream;
s508, if the frame rate information is not fed back as a result of the determination in the step S502, or if the frame rate is greater than the threshold value as a result of the determination in the step S504, performing hardware encoding on the auxiliary stream.
Through the embodiment, the original image frame is subjected to blocking processing according to definition requirements, the yuv420 format transcoding is carried out on the low-sensitivity area, and the structured data compression is carried out on the high-sensitivity area (including but not limited to the text area) to form the auxiliary enhancement information. The embodiment has the characteristics of high-definition lossless display and high frame rate, and simultaneously further reduces the data volume, reduces the transmission bandwidth and improves the frame rate through a certain compression algorithm. The following beneficial effects can be achieved: the method has the advantages that the method adopts the enhanced auxiliary information to compress the image for the first time in the industry, so that the method has the characteristic of high-definition display; hardware coding and decoding are combined with auxiliary enhancement information, so that the method has the characteristic of high frame rate; the data volume is reduced through a certain compression algorithm, and the transmission bandwidth requirement is met in a self-adaptive mode.
There is also provided in this embodiment an image display system, as shown in fig. 6, fig. 6 is a block diagram of an image display system according to an embodiment of the present invention, the system including: a server 602 and a client 604. Wherein the content of the first and second substances,
the server 602 includes: a region identification module 60202 for blocking the entire image (corresponding to the aforementioned target image); an encoding module 60204 for hardware encoding of the original image frame data; an auxiliary stream extraction module 60206, configured to extract information (auxiliary stream) for assisting image quality enhancement, for example, extract an auxiliary stream for the high-sensitivity region; the compression module 60208 is used for performing structured data compression on the high-sensitivity region to form auxiliary enhancement information; an encapsulating module 60210, configured to perform packet encapsulation on the first target data and the second target data to form a data packet;
the client 604 includes: a decapsulation module 60402, configured to decapsulate the received data packet; a decompression module 60404, configured to decompress the information for assisting in enhancing the image quality; a frame rate statistic module 60406, configured to count a frame rate of frame images displayed by the client; a decoding module 60408, configured to perform hardware-by-hardware decoding on the decapsulated whole frame data; a region reorganizing module 60410, configured to reorganize the high-sensitivity region and the original frame image.
There is also provided in this embodiment an image display apparatus, and fig. 7 is a block diagram showing a configuration of the image display apparatus according to the embodiment of the present invention, as shown in fig. 7, the apparatus includes:
a first extraction module 702, configured to, in a case that a target image is received, extract a target area included in the target image, where the target area includes an area where information of a target type is located;
a first encoding module 704, configured to perform hardware encoding processing on the target image to obtain first target data;
a second extracting module 706, configured to extract information in the target region, compress the information in the target region to obtain compressed data, and encode the compressed data to obtain second target data;
an encapsulating module 708, configured to encapsulate the first target data and the second target data to obtain a target data packet;
a sending module 710, configured to send the target data packet to a terminal, so as to instruct the terminal to display the target image based on the target data packet.
In an alternative embodiment, the first extracting module 702 includes: the blocking unit is used for blocking the region of the target image according to a classification algorithm to obtain a plurality of block regions; a determining unit, configured to determine the target area from the plurality of block areas.
In an optional embodiment, the second extracting module 706 includes: the judging unit is used for judging whether the terminal feeds back a frame rate message, wherein the frame rate message is used for indicating the frequency of a frame image sent by the terminal display server; a first encoding unit, configured to perform the hardware encoding processing on the compressed data to obtain second target data when it is determined that the frame rate message is fed back by the terminal and a frame rate indicated by the frame rate message is greater than or equal to a preset frame rate threshold; a second encoding unit, configured to perform software encoding processing on the compressed data to obtain second target data when it is determined that the frame rate message is fed back by the terminal and a frame rate indicated by the frame rate message is smaller than the preset frame rate threshold; and a third encoding unit, configured to perform the hardware encoding processing on the compressed data to obtain the second target data when it is determined that the terminal does not feed back the frame rate message.
In an alternative embodiment, the first encoding module 704 includes: the sampling unit is used for carrying out downsampling processing on the target image to obtain a sampled image; and the fourth encoding unit is used for carrying out the hardware encoding processing on the sampling image so as to obtain the first target data.
In this embodiment, there is further provided an image display apparatus, and fig. 8 is a block diagram of a structure of the image display apparatus according to the embodiment of the present invention, as shown in fig. 8, the apparatus includes:
a receiving module 802, configured to receive a target data packet sent by a server, where the target data packet is obtained by encapsulating, by the server, first target data and second target data, the first target data is obtained by performing hardware coding processing on a target image by the server, the second target data is obtained by compressing, by the server, information in a target area, and encoding compressed data obtained after compression, and the target area is an area that includes information of a target type and is included in the target image;
a decapsulation module 804, configured to decapsulate the target data packet to obtain the first target data and the second target data;
a first decoding module 806, configured to perform hardware decoding processing on the first target data to obtain the target image;
a decompression module 808, configured to decompress the second target data to obtain information in the target area;
a restructuring module 810, configured to restructure the target image and the information in the target region to obtain a restructured image;
and a display module 812 for displaying the recombined image.
In an alternative embodiment, the display module 812 includes: and the processing unit is used for overhauling the recombined image and displaying the overhauled recombined image.
In an optional embodiment, the apparatus further comprises: and the counting module is used for counting the frequency of the frame image sent by the display server and feeding back frame rate information to the server side.
In an alternative embodiment, the restructuring module 810 includes: the restoring unit is used for restoring the image data in the target area based on the information in the target area to obtain an image of the target area; and the recombination unit is used for recombining the target image and the image in the target area to obtain the recombined image.
In an alternative embodiment, the decompression module 808 comprises: a decoding unit, configured to perform decoding processing on the second target data to obtain compressed data; and the decompression unit is used for carrying out decompression processing on the compressed data to obtain the information in the target area.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the invention described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into various integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (13)
1. An image display method, comprising:
under the condition that a target image is received, extracting a target area contained in the target image, wherein the target area comprises an area where information of a target type is located;
performing hardware coding processing on the target image to obtain first target data;
extracting information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data;
packaging the first target data and the second target data to obtain a target data packet;
and sending the target data packet to a terminal to instruct the terminal to display the target image based on the target data packet.
2. The method of claim 1, wherein extracting the target region contained in the target image comprises:
partitioning the region of the target image according to a classification algorithm to obtain a plurality of block regions;
and determining the target area from a plurality of block areas.
3. The method of claim 1, wherein encoding the compressed data to obtain the second target data comprises:
judging whether the terminal feeds back a frame rate message, wherein the frame rate message is used for indicating the frequency of a frame image sent by a display server of the terminal;
under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is larger than or equal to a preset frame rate threshold value, performing hardware coding processing on the compressed data to obtain second target data;
under the condition that the frame rate message is judged to be fed back by the terminal and the frame rate indicated by the frame rate message is smaller than the preset frame rate threshold, performing software coding processing on the compressed data to obtain second target data;
and under the condition that the frame rate message is not fed back by the terminal, performing the hardware coding processing on the compressed data to obtain second target data.
4. The method of claim 1, wherein performing hardware encoding processing on the target image to obtain first target data comprises:
carrying out down-sampling processing on the target image to obtain a sampled image;
and carrying out the hardware coding processing on the sampling image to obtain the first target data.
5. An image display method, comprising:
receiving a target data packet sent by a server, wherein the target data packet is obtained by encapsulating first target data and second target data by the server, the first target data is obtained by hardware coding processing of a target image by the server, the second target data is obtained by compressing information in a target area by the server and coding the compressed data obtained after compression, and the target area is an area which includes information of a target type and is included in the target image;
decapsulating the target data packet to obtain the first target data and the second target data;
performing hardware decoding processing on the first target data to obtain the target image;
decompressing the second target data to obtain information in the target area;
recombining the target image and the information in the target area to obtain a recombined image;
and displaying the recombined image.
6. The method of claim 5, wherein displaying the recombined image comprises:
and overhauling the recombined image, and displaying the overhauled recombined image.
7. The method of claim 5, further comprising:
and counting the frequency of the frame image sent by the display server, and feeding back frame rate information to the server.
8. The method of claim 5, wherein the recombining the target image and the information within the target region to obtain a recombined image comprises:
restoring the image data in the target area based on the information in the target area to obtain an image of the target area;
and recombining the target image and the image in the target area to obtain the recombined image.
9. The method of claim 5, wherein decompressing the second target data to obtain the information in the target area comprises:
decoding the second target data to obtain compressed data;
and performing the decompression processing on the compressed data to obtain the information in the target area.
10. An image display apparatus, comprising:
the device comprises a first extraction module, a second extraction module and a third extraction module, wherein the first extraction module is used for extracting a target area contained in a target image under the condition that the target image is received, and the target area comprises an area where information of a target type is located;
the first coding module is used for carrying out hardware coding processing on the target image to obtain first target data;
the second extraction module is used for extracting the information in the target area, compressing the information in the target area to obtain compressed data, and encoding the compressed data to obtain second target data;
the encapsulation module is used for encapsulating the first target data and the second target data to obtain a target data packet;
and the sending module is used for sending the target data packet to a terminal so as to instruct the terminal to display the target image based on the target data packet.
11. An image display apparatus, comprising:
a receiving module, configured to receive a target data packet sent by a server, where the target data packet is obtained by encapsulating, by the server, first target data and second target data, the first target data is obtained by performing hardware coding processing on a target image by the server, the second target data is obtained by compressing, by the server, information in a target area, and encoding compressed data obtained after compression, and the target area is an area that includes information of a target type and is included in the target image;
a decapsulation module, configured to decapsulate the target data packet to obtain the first target data and the second target data;
the first decoding module is used for performing hardware decoding processing on the first target data to obtain the target image;
the decompression module is used for decompressing the second target data to obtain the information in the target area;
the recombination module is used for recombining the target image and the information in the target area to obtain a recombined image;
and the display module is used for displaying the recombined image.
12. A computer-readable storage medium, comprising a stored program, wherein the program is operable to perform the method of any one of claims 1 to 4 or 5 to 9.
13. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program and the processor is arranged to execute the method of any of claims 1 to 4 or 5 to 9 by means of the computer program.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111111256.6A CN113573059B (en) | 2021-09-23 | 2021-09-23 | Image display method, image display apparatus, storage medium, and electronic apparatus |
| PCT/CN2022/093980 WO2023045364A1 (en) | 2021-09-23 | 2022-05-19 | Image display method and apparatus, and storage medium and electronic apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111111256.6A CN113573059B (en) | 2021-09-23 | 2021-09-23 | Image display method, image display apparatus, storage medium, and electronic apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113573059A true CN113573059A (en) | 2021-10-29 |
| CN113573059B CN113573059B (en) | 2022-03-01 |
Family
ID=78174070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111111256.6A Active CN113573059B (en) | 2021-09-23 | 2021-09-23 | Image display method, image display apparatus, storage medium, and electronic apparatus |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113573059B (en) |
| WO (1) | WO2023045364A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023045364A1 (en) * | 2021-09-23 | 2023-03-30 | 中兴通讯股份有限公司 | Image display method and apparatus, and storage medium and electronic apparatus |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106161991A (en) * | 2016-07-29 | 2016-11-23 | 青岛海信移动通信技术股份有限公司 | A kind of camera video processing method and terminal |
| CN108184118A (en) * | 2016-12-08 | 2018-06-19 | 中兴通讯股份有限公司 | Cloud desktop contents encode and coding/decoding method and device, system |
| CN108833932A (en) * | 2018-07-19 | 2018-11-16 | 湖南君瀚信息技术有限公司 | A kind of method and system for realizing the ultralow delay encoding and decoding of HD video and transmission |
| CN110351564A (en) * | 2019-08-08 | 2019-10-18 | 上海纽菲斯信息科技有限公司 | A kind of text clearly video compress transmission method and system |
| CN111935483A (en) * | 2020-09-29 | 2020-11-13 | 北京泽塔云科技股份有限公司 | Video image lossless coding and decoding method and system |
| CN112509146A (en) * | 2020-11-23 | 2021-03-16 | 歌尔光学科技有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4656912B2 (en) * | 2004-10-29 | 2011-03-23 | 三洋電機株式会社 | Image encoding device |
| US20070113242A1 (en) * | 2005-11-16 | 2007-05-17 | Fetkovich John E | Selective post-processing of compressed digital video |
| CN104065964B (en) * | 2014-06-19 | 2017-06-16 | 上海交通大学 | The decoding method and video encoding/decoding apparatus of interested area information |
| CN110933461B (en) * | 2018-09-19 | 2022-12-30 | 中兴通讯股份有限公司 | Image processing method, device, system, network equipment, terminal and storage medium |
| CN111193928B (en) * | 2019-10-28 | 2023-08-11 | 腾讯科技（深圳）有限公司 | Method and apparatus for delivering region of interest information in video |
| CN113573059B (en) * | 2021-09-23 | 2022-03-01 | 中兴通讯股份有限公司 | Image display method, image display apparatus, storage medium, and electronic apparatus |
-
2021
- 2021-09-23 CN CN202111111256.6A patent/CN113573059B/en active Active
-
2022
- 2022-05-19 WO PCT/CN2022/093980 patent/WO2023045364A1/en unknown
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106161991A (en) * | 2016-07-29 | 2016-11-23 | 青岛海信移动通信技术股份有限公司 | A kind of camera video processing method and terminal |
| CN108184118A (en) * | 2016-12-08 | 2018-06-19 | 中兴通讯股份有限公司 | Cloud desktop contents encode and coding/decoding method and device, system |
| CN108833932A (en) * | 2018-07-19 | 2018-11-16 | 湖南君瀚信息技术有限公司 | A kind of method and system for realizing the ultralow delay encoding and decoding of HD video and transmission |
| CN110351564A (en) * | 2019-08-08 | 2019-10-18 | 上海纽菲斯信息科技有限公司 | A kind of text clearly video compress transmission method and system |
| CN111935483A (en) * | 2020-09-29 | 2020-11-13 | 北京泽塔云科技股份有限公司 | Video image lossless coding and decoding method and system |
| CN112509146A (en) * | 2020-11-23 | 2021-03-16 | 歌尔光学科技有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023045364A1 (en) * | 2021-09-23 | 2023-03-30 | 中兴通讯股份有限公司 | Image display method and apparatus, and storage medium and electronic apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113573059B (en) | 2022-03-01 |
| WO2023045364A1 (en) | 2023-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11012489B2 (en) | Picture file processing method, picture file processing device, and storage medium | |
| CN111193948B (en) | Picture transmission and display method and system of display terminal | |
| CN107483934B (en) | Coding and decoding method, device and system | |
| US8873625B2 (en) | Enhanced compression in representing non-frame-edge blocks of image frames | |
| US20110194616A1 (en) | Embedded video compression for hybrid contents | |
| CN110868426A (en) | Data transmission method, system and equipment | |
| US9686560B2 (en) | Lossless data compression and decompression apparatus, system, and method | |
| CN111131828B (en) | Image compression method and device, electronic equipment and storage medium | |
| CN108347602B (en) | Method and apparatus for lossless compression of video data | |
| CN113573059B (en) | Image display method, image display apparatus, storage medium, and electronic apparatus | |
| CN111193928B (en) | Method and apparatus for delivering region of interest information in video | |
| CN110519607B (en) | Video decoding method and device, and video encoding method and device | |
| CN110650357B (en) | Video decoding method and device | |
| CN109982091B (en) | Image processing method and device | |
| US9083954B2 (en) | Video processing method and system and related device | |
| US20170201759A1 (en) | Method and device for image encoding and image decoding | |
| CN113127637A (en) | Character restoration method and device, storage medium and electronic device | |
| WO2022127565A1 (en) | Video processing method and apparatus, and device | |
| CN110545446A (en) | desktop image encoding and decoding methods, related devices and storage medium | |
| WO2022057746A1 (en) | Image processing method and apparatus, device, and computer readable storage medium | |
| WO2021057676A1 (en) | Video coding method and apparatus, video decoding method and apparatus, electronic device and readable storage medium | |
| CN108933945B (en) | GIF picture compression method, device and storage medium | |
| CN110868391A (en) | Remote transmission method, system and equipment | |
| CN115278264B (en) | Image amplification and OSD superposition coding method and system | |
| CN209787335U (en) | Data coding and transcoding equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |