CN110809152A - Information processing method, encoding device, decoding device, system, and storage medium - Google Patents
Information processing method, encoding device, decoding device, system, and storage medium Download PDFInfo
- Publication number
- CN110809152A CN110809152A CN201911076277.1A CN201911076277A CN110809152A CN 110809152 A CN110809152 A CN 110809152A CN 201911076277 A CN201911076277 A CN 201911076277A CN 110809152 A CN110809152 A CN 110809152A
- Authority
- CN
- China
- Prior art keywords
- depth information
- image
- code stream
- image data
- decoder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
The embodiment of the application discloses an information processing method, an encoding device, a decoding device, a system and a storage medium, wherein the method comprises the following steps: acquiring image data and depth information of a target object; carrying out joint coding or independent coding on the depth information and the image data to obtain a code stream; and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
Description
Technical Field
Embodiments of the present disclosure relate to electronic application technologies, and in particular, to an information processing method, an encoding device, a decoding device, a system, and a storage medium.
Background
At present, when an image signal is transmitted, in order to improve the transmission speed, an encoder is firstly utilized to perform image coding on a two-dimensional image collected by an image sensor and a depth image collected by a depth camera to form a code stream, the code stream is sent to a decoder, and the decoder analyzes the code stream to obtain the two-dimensional image and the depth image.
Disclosure of Invention
The application provides an information processing method, an encoding device, a decoding device, a system and a storage medium, which can improve the utilization rate of information.
The technical scheme of the application is realized as follows:
the embodiment of the application provides an information processing method, which is applied to a coding device and comprises the following steps:
acquiring image data and depth information of a target object;
carrying out joint coding or independent coding on the depth information and the image data to obtain a code stream;
and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
In the above method, the acquiring depth information of the target object includes:
acquiring initial depth information corresponding to the target object;
and determining the initial depth information as the depth information corresponding to the target object.
In the above method, after acquiring the initial depth information corresponding to the target object, the method further includes:
processing the initial depth information to obtain processed initial depth information;
and determining the processed initial depth information as the depth information corresponding to the target object.
In the above method, the jointly encoding or independently encoding the depth information and the image data to obtain a code stream includes:
determining target depth information at a specified position in the image data from the depth information;
and carrying out joint coding or independent coding on the image data and the target depth information to obtain the code stream.
In the above method, the jointly encoding or independently encoding the image data and the target depth information to obtain a code stream includes:
when the image coding standard is judged to be expandable, joint coding is carried out on the image data and the target depth information by utilizing the correlation between the image data and the target depth information to obtain the code stream;
and when the image coding standard is judged to be not expandable, independently coding the image data and the target depth information to obtain the code stream.
In the above method, the independently encoding the depth information to obtain a code stream includes:
performing redundancy removal processing on the depth information to obtain the depth information after redundancy removal, wherein the redundancy removal processing at least comprises one of frame prediction, frequency domain transformation, quantization and sampling;
and entropy coding is carried out on the depth information after redundancy removal, so as to obtain the code stream.
The embodiment of the application provides an information processing method, which is applied to a decoding device and comprises the following steps:
when a code stream is received, performing joint decoding or independent decoding on the code stream to obtain depth information and image data;
and processing the image data by using the depth information to obtain a target image.
In the above method, after the code stream is jointly decoded or independently decoded to obtain depth information and image data, the method further includes:
and restoring the depth information to generate a depth image corresponding to the depth information.
In the above method, the processing the image data by using the depth information to obtain a target image includes:
when the depth information is phase information, performing virtualization foreground or background processing on the image data by using the phase information to obtain a virtualized image;
and taking the blurred image as the target image.
An embodiment of the present application provides an encoding apparatus, including: the device comprises a depth information module, an image sensor and an encoder;
the image sensor is used for acquiring image data;
the depth information module is used for acquiring the depth information of the target object;
the encoder is used for carrying out joint encoding or independent encoding on the depth information and the image data to obtain a code stream; and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
In the above apparatus, the depth information module includes a depth information sensor;
the depth information sensor is used for acquiring initial depth information corresponding to the target object; and determining the initial depth information as the depth information corresponding to the target object.
In the above apparatus, the depth information module is further configured to process the initial depth information to obtain processed initial depth information; and determining the processed initial depth information as the depth information corresponding to the target object.
In the above apparatus, the encoder is further configured to determine, from the depth information, target depth information at a specified position in the image data; and carrying out joint coding or independent coding on the image data and the target depth information to obtain the code stream.
In the above apparatus, the encoder comprises: a depth information encoder and an image encoder;
the encoder is further configured to perform joint encoding on the image data and the target depth information by using correlation between the image data and the target depth information when it is determined that the image encoding standard is extensible, so as to obtain the code stream;
the depth information encoder is used for independently encoding the target depth information to obtain the code stream when the image coding standard is judged to be not expandable;
and the image encoder is used for independently encoding the image data to obtain the code stream when the image encoding standard is judged not to be expandable.
In the above apparatus, the encoder is further configured to perform a redundancy removal process on the depth information to obtain the depth information after the redundancy removal, where the redundancy removal process at least includes one of frame prediction, frequency domain transformation, quantization and sampling; and entropy coding is carried out on the depth information after redundancy removal, so as to obtain the code stream.
An embodiment of the present application provides a decoding apparatus, including: an image processor and a decoder;
the decoder is used for carrying out joint decoding or independent decoding on the code stream when the code stream is received to obtain depth information and image data;
and the image processor is used for processing the image data by using the depth information to obtain a target image.
In the above apparatus, the decoder further comprises a depth image generator;
and the depth image generator is used for recovering the depth information and generating a depth image corresponding to the depth information.
In the above apparatus, the image processor is further configured to perform blurring foreground or background processing on the image data by using the phase information when the depth information is the phase information, so as to obtain a blurred image; and taking the blurred image as the target image.
In the above apparatus, the depth image generator and the image processor may both be integrated in the decoder; the depth image generator and the image processor may both be independent of the decoder and connected to the decoder;
the depth image generator may be integrated in the decoder, and the image processor may be connected to the decoder independently of the decoder;
the image processor may be integrated in the decoder, and the depth image generator may be connected to the decoder independently of the decoder.
An embodiment of the present application provides an information processing system, including: the device comprises an encoding device and a decoding device, wherein the encoding device comprises a depth information module, an image sensor and an encoder, and the decoding device comprises an image processor and a decoder;
the image sensor is used for acquiring image data;
the depth information module is used for acquiring the depth information of the target object;
the encoder is used for carrying out joint encoding or independent encoding on the depth information and the image data to obtain a code stream; transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream;
the decoder is used for carrying out joint decoding or independent decoding on the code stream when the code stream is received to obtain depth information and image data;
and the image processor is used for processing the image data by using the depth information to obtain a target image.
The embodiment of the application provides a computer readable storage medium applied to a coding device, and the computer readable storage medium stores one or more programs which can be executed by one or more first processors to realize the method according to any one of the above items.
The embodiment of the application provides a computer readable storage medium applied to a decoding device, and the computer readable storage medium stores one or more programs, and the one or more programs can be executed by one or more second processors to realize the method according to any one of the above items.
The embodiment of the application provides an information processing method, an encoding device, a decoding device, a system and a storage medium, wherein the information processing method applied to the encoding device comprises the following steps: acquiring image data and depth information of a target object; carrying out joint coding or independent coding on the depth information and the image data to obtain a code stream; and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream. By adopting the method, the depth information of the target object is directly adopted for coding to obtain the code stream carrying the depth information, and the code stream is transmitted to the decoding device, so that the decoding device can decode the depth information and the image data from the code stream, and further, the decoding device can recover to obtain the depth image by utilizing the depth information and can also utilize the depth information to carry out image processing on the image data, thereby improving the information utilization rate.
Drawings
Fig. 1 is a schematic flowchart of an information processing method applied to an encoding apparatus according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of an information processing method applied to a decoding apparatus according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating an information processing method applied to an information processing system according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an encoding apparatus according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 6(a) is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 6(b) is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 6(c) is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 6(d) is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 7(a) is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application;
fig. 7(b) is a schematic structural diagram six of a decoding apparatus according to an embodiment of the present application;
fig. 7(c) is a schematic structural diagram seven of a decoding device according to an embodiment of the present application;
fig. 7(d) is a schematic structural diagram eight of a decoding apparatus according to an embodiment of the present application;
fig. 8(a) is a first schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 8(b) is a schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 8(c) is a schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 8(d) is a schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 9(a) is a schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 9(b) is a sixth schematic structural diagram of an information processing system according to an embodiment of the present application;
fig. 9(c) is a schematic structural diagram seven of an information processing system according to an embodiment of the present application;
fig. 9(d) is a schematic structural diagram eight of an information processing system according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant application and are not limiting of the application. It should be noted that, for the convenience of description, only the parts related to the related applications are shown in the drawings.
An embodiment of the present application provides an information processing method, which is applied to an encoding device, and as shown in fig. 1, the information processing method includes:
s101, collecting image data and depth information of a target object;
in the embodiment of the application, the encoding device comprises an image sensor and a depth information module, the encoding device simultaneously utilizes the image sensor to acquire the image data of the target object, and the depth information module is utilized to acquire the depth information of the target object.
In some embodiments, the depth information module collects initial depth information corresponding to the target object and determines the initial depth information as depth information corresponding to the target object.
Optionally, the depth information module includes a TOF camera or a binocular camera, and is specifically selected according to actual conditions, and the embodiment of the present application is not specifically limited.
Optionally, when the TOF camera is used to obtain initial depth information corresponding to the target object, the depth information module determines the original charge image and the sensor attribute parameters, such as temperature, as the initial depth information corresponding to the target object, where the obtaining process of the original charge image may be: under two different emission signal frequencies, 8 groups of signals with different phases are obtained by controlling integration time and sampling of a depth information module, and after photoelectric conversion, 10-bit quantization is carried out on the eight groups of signals to generate 8 original charge images.
Optionally, when initial depth information corresponding to the target object is acquired by using the binocular camera, the depth information module uses two images obtained by shooting by using the binocular camera, and calculates the positions of the two images to obtain information such as parallax, and the depth information module uses the parallax information and camera parameters as the initial depth information.
In other embodiments, the depth information module processes the initial depth information to obtain processed initial depth information, and determines the processed initial depth information as depth information corresponding to the target object.
Optionally, the depth information module may perform phase calibration on the initial depth information to obtain phase information; or, the depth information module may perform other processing on the initial depth information to generate other information, such as an intermediate image of the depth image, and the like, and the specific processing procedure and the processing degree are selected according to the actual situation, which is not specifically limited in the embodiment of the present application,
illustratively, after the raw charge image is acquired by the TOF camera, the depth information module generates 2 pieces of process depth data and one piece of background data from the raw circuit image, and uses the 2 pieces of process depth data and the one piece of background data as the depth information of the target object.
Illustratively, the phase information may be speckles, laser stripes, gray codes, sinusoidal stripes, and the like acquired by the depth information module, and the specific phase information may be determined according to an actual situation, which is not limited in the embodiment of the present application.
S102, carrying out combined coding or independent coding on the depth information and the image data to obtain a code stream;
an encoder in the encoding device performs combined encoding on the depth information and the image data to obtain a code stream representing that the code stream simultaneously carries the depth information and the image data, namely a mixed code stream; or independently coding the depth information and the image data to obtain a depth information code stream and an image data code stream, namely 2 independent code streams.
When encoding depth information, the encoding device may encode depth information corresponding to all image data, or encode depth information corresponding to specified image data or specified positions in the image data, and depth information corresponding to other non-specified images or non-specified positions is not encoded, and is specifically selected according to actual situations, and embodiments of the present invention are not specifically limited.
In the embodiment of the application, the encoding device determines the target depth information at the appointed position in the image data from the depth information; and then, the encoder performs joint encoding or independent encoding on the image data and the target depth information to obtain a code stream.
In the embodiment of the application, the encoding device acquires the expandability of the image encoding standard, and when the image encoding standard is judged to be expandable, the encoder performs joint encoding on the image data and the target depth information by using the correlation between the image data and the target depth information to obtain a code stream; when the image coding standard is judged to be not expandable, the encoder independently encodes the image data and the target depth information to obtain a code stream, wherein the code stream comprises a depth information code stream and an image data code stream, the target depth information is independently encoded by utilizing the spatial correlation of the depth information to obtain a depth information code stream, and the position of the depth information stored in the depth information code stream is an additional parameter set or other arbitrary positions.
In some embodiments, the preset position may be a picture header, a sequence header, an additional parameter set, or any other position.
It should be noted that, for joint coding, because the code stream has decoupleability or independence, after receiving the code stream, a decoding device adopting an image standard coding and decoding protocol can extract only image data from the code stream, and does not extract depth information; only depth information can be extracted from the code stream, and image data is not extracted; the specific selection is performed according to the actual situation, and the embodiment of the application is not limited.
The encoder in the encoding device comprises a depth information encoder and an image encoder, and for joint encoding, the depth information and the image data are subjected to joint encoding by directly utilizing the image encoder; for independent coding, a depth information encoder encodes depth information by using spatial correlation or temporal correlation of the depth information and the like to obtain a depth information code stream; and the image encoder encodes the image data to obtain an image data code stream.
Specifically, the image data code stream still follows an image coding and decoding protocol, such as JPEG, JPEG XR, HEIC, and the like, and the depth information code stream follows a unique standard or an industry standard of a manufacturer and other organizations.
In some embodiments, the encoding apparatus performs a redundancy removal process on the depth information to obtain the depth information after the redundancy removal, where the redundancy removal process at least includes one of frame prediction, frequency domain transformation, quantization and sampling; and entropy coding is carried out on the depth information after the redundancy is removed, so as to obtain a depth information code stream.
In order to compress the size of the coded information, the coding device executes the operation of eliminating redundancy in the process of coding the depth information, and then codes the depth information after removing the redundancy to obtain a depth information code stream.
Illustratively, when the depth information module in the encoding apparatus determines that the depth information is at least two pieces of phase information, redundancy elimination is performed on the at least two pieces of phase information by using phase correlation between the at least two pieces of phase information, so as to obtain the depth information after redundancy elimination.
Illustratively, when the depth information module in the encoding device determines that the depth information is not at least two pieces of phase information, the depth information module performs redundancy elimination on the depth information by using spatial correlation of the depth information to obtain the depth information after redundancy removal.
Illustratively, the depth information module in the encoding apparatus performs redundancy elimination on the depth information by using the time correlation of the depth information, so as to obtain the depth information after redundancy removal.
Illustratively, the depth information module in the encoding apparatus performs redundancy elimination on the depth information by using a preset depth range, so as to obtain the depth information after redundancy removal.
Illustratively, a depth information module in the encoding apparatus performs frequency domain conversion on the depth information to obtain frequency domain information; and (4) redundancy elimination is carried out on the frequency domain information by utilizing the frequency domain correlation, so as to obtain the depth information after redundancy elimination.
It should be noted that the preset depth range is a range in which the depth information sensor can acquire depth information.
S103, transmitting the code stream to a decoding device so that the decoding device can perform image processing based on the code stream.
In the embodiment of the application, the coding device transmits the code stream to the decoding device, and the decoding device can perform image processing such as image enhancement on the image data based on the depth information in the code stream.
It can be understood that, because the depth information of the target object is directly adopted for encoding to obtain the code stream carrying the depth information, and the code stream is transmitted to the decoding device, the decoding device can decode the depth information and the image data from the code stream, and then the decoding device can recover to obtain the depth image by using the depth information, and can also perform image processing on the image data by using the depth information, so that the information utilization rate is improved.
An embodiment of the present application further provides an information processing method, which is applied to a decoding device, and as shown in fig. 2, the information processing method includes:
s201, when a code stream is received, performing combined decoding or independent decoding on the code stream to obtain depth information and image data;
and after a decoder in the decoding device receives the code stream, performing joint decoding or independent decoding according to the code stream to obtain depth information and image data.
In some embodiments, the codestream is a mixed codestream; the decoding device comprises a decoder which decodes the mixed code stream to obtain image data and depth information.
In some embodiments, the codestream comprises an image data codestream and a depth information codestream; the decoder comprises an image decoder and a depth information decoder, wherein the image decoder decodes the respective image data code streams to obtain image data, and the depth information decoder decodes the depth information code streams to obtain depth information.
Furthermore, the decoding device further comprises a depth image generator, wherein the depth image generator is used for recovering the depth information after the decoding device performs combined decoding or independent decoding on the code stream to obtain the depth information and the image data, and generating a depth image corresponding to the depth information.
S202, processing the image data by using the depth information to obtain a target image.
In the embodiment of the application, the decoding device further includes an image processor, and after the decoding device acquires the depth information and the image data, the image processor processes the image data by using the depth information to obtain the target image.
Optionally, the method for processing the image data by using the depth information includes: denoising, white balance adjustment, background blurring, portrait cutout and the like, and the selection is specifically carried out according to the actual situation, and the embodiment of the application is not specifically limited.
In one embodiment, when the depth information is phase information, the image processor performs blurring foreground or background processing on the image data by using the phase information to obtain a blurring image; and then taking the blurred image as a target image.
In another embodiment, when the depth information is charge information, the image processor determines noise and external visible light in the shooting scene using the charge information, thereby facilitating image denoising and white balance adjustment.
In the embodiment of the present application, for independent decoding, at least one of the depth image generator and the image processor may be integrated in the decoder, so that the decoder at least implements functions of depth image generation and image generation; or the depth image generator and the image processor are respectively connected with the output end of the decoder, and the three realize respective functions.
In the embodiment of the present application, for joint coding, the depth image generator may be integrated in the depth information decoder, or the depth image generator is connected to an output end of the depth information decoder; the image processor may be integrated in the image decoder, or the image processor is connected to an output terminal of the image decoder, which is specifically selected according to the actual situation, and the embodiment of the present application is not particularly limited.
In some embodiments, the information processing system includes an encoding device and a decoding device, and an information processing method applied to the information processing system, as shown in a flowchart of an information processing method shown in fig. 3, the information processing method includes:
s301, the encoding device collects image data and depth information of a target object;
s302, the coding device performs combined coding or independent coding on the depth information and the image data to obtain a code stream;
s303, the coding device transmits the code stream to the decoding device;
s304, the decoding device performs joint decoding or independent decoding on the code stream to obtain depth information and image data;
s305, the decoding device processes the image data using the depth information to obtain a target image.
It can be understood that, because the depth information of the target object is directly adopted for encoding to obtain the code stream carrying the depth information, and the code stream is transmitted to the decoding device, the decoding device can decode the depth information and the image data from the code stream, and then the decoding device can recover to obtain the depth image by using the depth information, and can also perform image processing on the image data by using the depth information, so that the information utilization rate is improved.
An embodiment of the present application further provides an encoding apparatus, as shown in fig. 4, where the encoding apparatus 4 includes: a depth information module 41, an image sensor 42, and an encoder 43;
an image sensor 42 for acquiring image data;
a depth information module 41, configured to collect depth information of the target object;
an encoder 43, configured to perform joint encoding or independent encoding on the depth information and the image data to obtain a code stream; and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
In some embodiments, the depth information module 41 includes a depth information sensor 411;
the depth information sensor 411 is configured to acquire initial depth information corresponding to the target object; and determining the initial depth information as the depth information corresponding to the target object.
In some embodiments, the depth information module 41 is further configured to process the initial depth information to obtain processed initial depth information; and determining the processed initial depth information as the depth information corresponding to the target object.
In some embodiments, the encoder 43 is further configured to determine, from the depth information, target depth information at a specified position in the image data; and carrying out joint coding or independent coding on the image data and the target depth information to obtain the code stream.
In some embodiments, the encoder 43 includes: a depth information encoder 431 and an image encoder 432;
the encoder 43 is further configured to, when it is determined that the image coding standard is extensible, perform joint coding on the image data and the target depth information by using correlation between the image data and the target depth information to obtain the code stream;
the depth information encoder 431 is configured to, when it is determined that the image coding standard is not extensible, independently encode the target depth information to obtain the code stream;
the image encoder 432 is configured to, when it is determined that the image encoding standard is not extensible, independently encode the image data to obtain the code stream.
In some embodiments, the encoder 43 is further configured to perform a de-redundancy process on the depth information to obtain de-redundant depth information, where the de-redundancy process at least includes one of frame prediction, frequency domain transformation, quantization and sampling; and entropy coding is carried out on the depth information after redundancy removal, so as to obtain the code stream.
The embodiment of the application provides a computer readable storage medium applied to a coding device, wherein one or more programs are stored in the computer readable storage medium, the one or more programs can be executed by one or more first processors, and when the program is executed by the first processors, the information processing method applied to the coding device is realized.
An embodiment of the present application further provides a decoding apparatus 5, as shown in fig. 5, where the decoding apparatus 5 includes: an image processor 51 and a decoder 52;
the decoder 52 is configured to perform joint decoding or independent decoding on the code stream when the code stream is received, so as to obtain depth information and image data;
the image processor 51 is configured to process the image data by using the depth information to obtain a target image.
In some embodiments, the decoder 52 further comprises a depth image generator 53;
the depth image generator 53 is configured to restore the depth information and generate a depth image corresponding to the depth information.
In some embodiments, the image processor 51 is further configured to perform blurring foreground or background processing on the image data by using the phase information when the depth information is the phase information, so as to obtain a blurred image; and taking the blurred image as the target image.
In some embodiments, the depth image generator 53 and the image processor 51 may both be integrated in the decoder 52; the depth image generator 53 and the image processor 51 may be both independent of the decoder 52 and connected to the decoder 52;
the depth image generator 53 may be integrated in the decoder 52, and the image processor 51 may be connected to the decoder 52 independently of the decoder 52;
the image processor 51 may be integrated in the decoder 52, and the depth image generator 53 may be connected to the decoder 52 independently of the decoder 52.
Illustratively, as a schematic structural diagram of a decoding apparatus shown in fig. 6(a), the decoding apparatus 18 includes an image processor 181, an image decoder 182, and a depth image generator 183; both the depth image processor 183 and the image processor 181 are independent of the image decoder 182, the image decoder 182 connecting the depth image generator 183 and the image processor 181; wherein, the image decoder 182 processes the code stream and outputs depth information and image data, the image decoder 182 transmits the depth information to the image processor 181, and the image processor 181 processes the image data by using the depth information and outputs the target image; the image decoder 182 inputs the image data and the depth information to the image processor 181, and the image processor 181 processes the image data using the depth information to output a target image.
Illustratively, as shown in fig. 6(b) which is a schematic structural diagram of a decoding apparatus, the decoding apparatus 28 includes an image processor 281, an image decoder 282 and a depth image generator 283; both the depth image processor 283 and the image processor 281 are integrated in the image decoder 282; the image decoder 282 processes the code stream to directly output the depth image and/or the target image.
Illustratively, as shown in fig. 6(c), the decoding apparatus 38 includes an image processor 381, an image decoder 382, and a depth image generator 383; the depth image processor 383 is integrated in the image decoder 382, the image processor 381 is independent of the image decoder 382, and the image decoder 382 is connected to the image processor 381; the image decoder 382 processes the coded information to output a depth image frame and image data, and the image decoder 382 sends the image data and the depth information to the image processor 381; the image processor 381 processes the image data using the depth information, and outputs a target image frame.
Illustratively, as shown in fig. 6(d), the decoding apparatus 48 includes an image processor 481, an image decoder 482 and a depth image generator 483; the image processor 481 is integrated in the image decoder 482, the depth image processor 483 is separate from the image decoder 482, the image decoder 482 is connected to a depth image generator 483; wherein, the image decoder 482 processes the coded information and outputs depth information and a target image frame, and the image decoder 482 sends the depth information to the depth image generator 483; the depth image generator 483 restores the depth information and outputs a depth image frame.
In some embodiments, the decoder includes a depth information decoder and an image decoder, the decoding apparatus further includes a depth image generator;
the depth image generator is independent of the depth information decoder, the image processor is independent of the image decoder, the depth information decoder is connected with the depth image generator and the image processor, and the image decoder is connected with the image processor; or the depth image generator is integrated in a depth information decoder, the image processor is independent of the image decoder, and the depth information decoder and the image decoder are connected with the image processor; or, the depth image generator is independent of the depth information decoder, the image processor is integrated in the image decoder, and the depth information decoder is connected with the depth image generator and the image decoder; alternatively, the depth image generator is integrated in an image decoder, the image processor is integrated in a depth information decoder, and the depth information decoder is connected to the image decoder.
Illustratively, as shown in fig. 7(a), the decoding apparatus 19 includes an image processor 191, and further includes a depth information decoder 192, an image decoder 193, and a depth image generator 194; the depth image generator 194 is independent of the depth information decoder 192, the image processor 191 is independent of the image decoder 193, the depth information decoder 192 is connected with the depth image generator 194 and the image processor 191, and the image decoder 193 is connected with the image processor 191; wherein the image decoder 193 outputs image data, and the depth information decoder 192 outputs depth information; the image decoder 193 transmits the image data to the image processor 191, the depth information decoder 192 transmits the depth information to the depth image generator 194 and the image processor 191, the depth image generator 194 outputs a depth image frame, and the image processor 191 outputs a target image frame.
Illustratively, as shown in fig. 7(b), the decoding apparatus 29 includes an image processor 291, and further includes a depth information decoder 292, an image decoder 293, and a depth image generator 294; the depth image generator 294 is integrated in the depth information decoder 292, the image processor 291 is independent of the image decoder 293, and the depth information decoder 292 and the image decoder 293 are connected to the image processor 291; the image decoder 293 outputs image data, and the depth information decoder 292 outputs depth information and a depth image frame; the image decoder 293 transmits the image data to the image processor 291, the depth information decoder 292 transmits the depth information to the image processor 291, and the image processor 291 outputs a target image frame.
Illustratively, as shown in fig. 7(c), the decoding apparatus 39 comprises an image processor 391, and further comprises a depth information decoder 392, an image decoder 393, and a depth image generator 394; the depth image generator 394 is independent of the depth information decoder 392, the image processor 391 is integrated in the image decoder 393, the depth information decoder 392 connects the depth image generator 394 and the image decoder 393; wherein the image decoder 393 outputs image data and the depth information decoder 392 outputs depth information; the depth information decoder 392 transmits the depth information to the depth image generator 394 and the image decoder 393, the depth image generator 394 outputs a depth image frame, and the image decoder 393 outputs a target image frame.
Illustratively, as shown in fig. 7(d), the decoding apparatus 49 includes an image processor 491, and further includes a depth information decoder 492, an image decoder 493, and a depth image generator 494; the depth image generator 494 is integrated in the depth information decoder 492, the image processor 491 is integrated in the image decoder 493, and the depth information decoder 492 is connected to the image decoder 493; among them, the image decoder 493 outputs image data, and the depth information decoder 492 outputs depth information and a depth image frame; the depth information decoder 492 transmits the depth information to the image decoder 493, and the image decoder 493 outputs a target image frame.
The embodiment of the application provides a computer readable storage medium applied to a decoding device, wherein one or more programs are stored in the computer readable storage medium, the one or more programs can be executed by one or more second processors, and when the programs are executed by the second processors, the information processing method applied to the decoding device is realized.
An embodiment of the present application further provides an information processing system, including: the device comprises an encoding device and a decoding device, wherein the encoding device comprises a depth information module, an image sensor and an encoder, and the decoding device comprises an image processor and a decoder;
the image sensor is used for acquiring image data;
the depth information module is used for acquiring the depth information of the target object;
the encoder is used for carrying out joint encoding or independent encoding on the depth information and the image data to obtain a code stream; transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream;
the decoder is used for carrying out joint decoding or independent decoding on the code stream when the code stream is received to obtain depth information and image data;
and the image processor is used for processing the image data by using the depth information to obtain a target image.
Illustratively, as a schematic configuration diagram of an information processing system shown in fig. 8(a), the information processing system 101 includes an encoding device 7 and a decoding device 18.
Illustratively, as shown in fig. 8(b) which is a schematic diagram of the structure of an information processing system, the information processing system 102 includes an encoding device 7 and a decoding device 28.
Illustratively, as shown in fig. 8(c) which is a schematic diagram of the structure of an information processing system, the information processing system 103 includes an encoding device 7 and a decoding device 38.
Illustratively, as a schematic configuration diagram of an information processing system shown in fig. 8(d), the information processing system 104 includes the encoding device 7 and the decoding device 48.
Illustratively, as a schematic diagram of a structure of an information processing system shown in fig. 9(a), the information processing system 201 includes an encoding device 6 and a decoding device 19.
Illustratively, as a schematic configuration diagram of an information processing system shown in fig. 9(b), the information processing system 202 includes an encoding device 6 and a decoding device 29.
Illustratively, as a schematic configuration diagram of an information processing system shown in fig. 9(c), the information processing system 301 includes an encoding device 6 and a decoding device 39.
Illustratively, as a schematic configuration diagram of an information processing system shown in fig. 9(d), the information processing system 401 includes an encoding device 6 and a decoding device 49.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, systems and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.
Claims (22)
1. An information processing method applied to an encoding device, the method comprising:
acquiring image data and depth information of a target object;
carrying out joint coding or independent coding on the depth information and the image data to obtain a code stream;
and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
2. The method of claim 1, wherein the acquiring depth information of the target object comprises:
acquiring initial depth information corresponding to the target object;
and determining the initial depth information as the depth information corresponding to the target object.
3. The method of claim 2, wherein after acquiring the initial depth information corresponding to the target object, the method further comprises:
processing the initial depth information to obtain processed initial depth information;
and determining the processed initial depth information as the depth information corresponding to the target object.
4. The method of claim 1, wherein the jointly or independently encoding the depth information and the image data to obtain a code stream comprises:
determining target depth information at a specified position in the image data from the depth information;
and carrying out joint coding or independent coding on the image data and the target depth information to obtain the code stream.
5. The method of claim 4, wherein the jointly or independently encoding the image data and the target depth information to obtain a code stream comprises:
when the image coding standard is judged to be expandable, joint coding is carried out on the image data and the target depth information by utilizing the correlation between the image data and the target depth information to obtain the code stream;
and when the image coding standard is judged to be not expandable, independently coding the image data and the target depth information to obtain the code stream.
6. The method of claim 1, wherein the independently encoding the depth information to obtain a code stream comprises:
performing redundancy removal processing on the depth information to obtain the depth information after redundancy removal, wherein the redundancy removal processing at least comprises one of frame prediction, frequency domain transformation, quantization and sampling;
and entropy coding is carried out on the depth information after redundancy removal, so as to obtain the code stream.
7. An information processing method applied to a decoding device, the method comprising:
when a code stream is received, performing joint decoding or independent decoding on the code stream to obtain depth information and image data;
and processing the image data by using the depth information to obtain a target image.
8. The method of claim 7, wherein after jointly decoding or independently decoding the code stream to obtain depth information and image data, the method further comprises:
and restoring the depth information to generate a depth image corresponding to the depth information.
9. The method of claim 8, wherein processing the image data using the depth information to obtain a target image comprises:
when the depth information is phase information, performing virtualization foreground or background processing on the image data by using the phase information to obtain a virtualized image;
and taking the blurred image as the target image.
10. An encoding apparatus, characterized in that the encoding apparatus comprises: the device comprises a depth information module, an image sensor and an encoder;
the image sensor is used for acquiring image data;
the depth information module is used for acquiring the depth information of the target object;
the encoder is used for carrying out joint encoding or independent encoding on the depth information and the image data to obtain a code stream; and transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream.
11. The apparatus of claim 10, wherein the depth information module comprises a depth information sensor;
the depth information sensor is used for acquiring initial depth information corresponding to the target object; and determining the initial depth information as the depth information corresponding to the target object.
12. The apparatus of claim 11,
the depth information module is also used for processing the initial depth information to obtain the processed initial depth information; and determining the processed initial depth information as the depth information corresponding to the target object.
13. The apparatus of claim 10,
the encoder is further configured to determine, from the depth information, target depth information at a specified position in the image data; and carrying out joint coding or independent coding on the image data and the target depth information to obtain the code stream.
14. The apparatus of claim 13, wherein the encoder comprises: a depth information encoder and an image encoder;
the encoder is further configured to perform joint encoding on the image data and the target depth information by using correlation between the image data and the target depth information when it is determined that the image encoding standard is extensible, so as to obtain the code stream;
the depth information encoder is used for independently encoding the target depth information to obtain the code stream when the image coding standard is judged to be not expandable;
and the image encoder is used for independently encoding the image data to obtain the code stream when the image encoding standard is judged not to be expandable.
15. The apparatus of claim 10, wherein the apparatus is a portable electronic device
The encoder is further configured to perform redundancy removal processing on the depth information to obtain the depth information after redundancy removal, where the redundancy removal processing at least includes one of frame prediction, frequency domain transformation, quantization, and sampling; and entropy coding is carried out on the depth information after redundancy removal, so as to obtain the code stream.
16. A decoding apparatus, characterized in that the decoding apparatus comprises: an image processor and a decoder;
the decoder is used for carrying out joint decoding or independent decoding on the code stream when the code stream is received to obtain depth information and image data;
and the image processor is used for processing the image data by using the depth information to obtain a target image.
17. The apparatus of claim 16, wherein the decoder further comprises a depth image generator;
and the depth image generator is used for recovering the depth information and generating a depth image corresponding to the depth information.
18. The apparatus of claim 17,
the image processor is further configured to perform blurring foreground or background processing on the image data by using the phase information when the depth information is the phase information, so as to obtain a blurred image; and taking the blurred image as the target image.
19. The apparatus of claim 16,
the depth image generator and the image processor may both be integrated in the decoder; the depth image generator and the image processor may both be independent of the decoder and connected to the decoder;
the depth image generator may be integrated in the decoder, and the image processor may be connected to the decoder independently of the decoder;
the image processor may be integrated in the decoder, and the depth image generator may be connected to the decoder independently of the decoder.
20. An information processing system, the system comprising: the device comprises an encoding device and a decoding device, wherein the encoding device comprises a depth information module, an image sensor and an encoder, and the decoding device comprises an image processor and a decoder;
the image sensor is used for acquiring image data;
the depth information module is used for acquiring the depth information of the target object;
the encoder is used for carrying out joint encoding or independent encoding on the depth information and the image data to obtain a code stream; transmitting the code stream to a decoding device so that the decoding device performs image processing based on the code stream;
the decoder is used for carrying out joint decoding or independent decoding on the code stream when the code stream is received to obtain depth information and image data;
and the image processor is used for processing the image data by using the depth information to obtain a target image.
21. A computer-readable storage medium applied to an encoding apparatus, the computer-readable storage medium storing one or more programs executable by one or more first processors to implement the method according to any one of claims 1 to 6.
22. A computer-readable storage medium for use in a decoding apparatus, the computer-readable storage medium storing one or more programs, the one or more programs being executable by one or more second processors to implement the method of any one of claims 7-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911076277.1A CN110809152A (en) | 2019-11-06 | 2019-11-06 | Information processing method, encoding device, decoding device, system, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911076277.1A CN110809152A (en) | 2019-11-06 | 2019-11-06 | Information processing method, encoding device, decoding device, system, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110809152A true CN110809152A (en) | 2020-02-18 |
Family
ID=69501388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911076277.1A Pending CN110809152A (en) | 2019-11-06 | 2019-11-06 | Information processing method, encoding device, decoding device, system, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110809152A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112037295A (en) * | 2020-09-04 | 2020-12-04 | 上海数迹智能科技有限公司 | Event type ToF camera encoding and decoding method, device, medium and equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101616322A (en) * | 2008-06-24 | 2009-12-30 | 深圳华为通信技术有限公司 | Stereo video coding-decoding method, Apparatus and system |
| CN102792699A (en) * | 2009-11-23 | 2012-11-21 | 通用仪表公司 | Depth coding as an additional channel to video sequence |
| CN106796656A (en) * | 2014-10-14 | 2017-05-31 | 微软技术许可有限责任公司 | Away from the depth of time-of-flight camera |
| CN106993112A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Background-blurring method and device and electronic installation based on the depth of field |
| US20170230637A1 (en) * | 2016-02-07 | 2017-08-10 | Google Inc. | Multiple camera computing system having camera-to-camera communications link |
| CN107590828A (en) * | 2017-08-09 | 2018-01-16 | 广东欧珀移动通信有限公司 | The virtualization treating method and apparatus of shooting image |
| US20180031137A1 (en) * | 2015-12-21 | 2018-02-01 | Intel Corporation | Auto range control for active illumination depth camera |
-
2019
- 2019-11-06 CN CN201911076277.1A patent/CN110809152A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101616322A (en) * | 2008-06-24 | 2009-12-30 | 深圳华为通信技术有限公司 | Stereo video coding-decoding method, Apparatus and system |
| CN102792699A (en) * | 2009-11-23 | 2012-11-21 | 通用仪表公司 | Depth coding as an additional channel to video sequence |
| CN106796656A (en) * | 2014-10-14 | 2017-05-31 | 微软技术许可有限责任公司 | Away from the depth of time-of-flight camera |
| US20180031137A1 (en) * | 2015-12-21 | 2018-02-01 | Intel Corporation | Auto range control for active illumination depth camera |
| US20170230637A1 (en) * | 2016-02-07 | 2017-08-10 | Google Inc. | Multiple camera computing system having camera-to-camera communications link |
| CN106993112A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Background-blurring method and device and electronic installation based on the depth of field |
| CN107590828A (en) * | 2017-08-09 | 2018-01-16 | 广东欧珀移动通信有限公司 | The virtualization treating method and apparatus of shooting image |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112037295A (en) * | 2020-09-04 | 2020-12-04 | 上海数迹智能科技有限公司 | Event type ToF camera encoding and decoding method, device, medium and equipment |
| CN112037295B (en) * | 2020-09-04 | 2024-03-08 | 上海数迹智能科技有限公司 | Event type ToF camera coding and decoding method, device, medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5241500B2 (en) | Multi-view video encoding and decoding apparatus and method using camera parameters, and recording medium on which a program for performing the method is recorded | |
| US8374444B2 (en) | Method and apparatus for providing higher resolution images in an embedded device | |
| WO2014005297A1 (en) | Panorama based 3d video coding | |
| RU2014152105A (en) | DECODING DEVICE AND METHOD FOR DECODING | |
| EP2629520B1 (en) | Apparatus and method for shooting moving picture in camera device | |
| US8798356B2 (en) | Apparatus and method for encoding and decoding multi-view image | |
| WO2019147816A1 (en) | Hybrid projection-based point cloud texture coding | |
| CN113630609B (en) | Video encoding method, decoding method, storage medium and terminal equipment | |
| CN110809152A (en) | Information processing method, encoding device, decoding device, system, and storage medium | |
| WO2010113086A1 (en) | System and format for encoding data and three-dimensional rendering | |
| CN114391259A (en) | Information processing method, terminal device and storage medium | |
| US20150358643A1 (en) | Method of Depth Coding Compatible with Arbitrary Bit-Depth | |
| CN114175626B (en) | Information processing method, encoding device, decoding device, system, and storage medium | |
| CN110784722B (en) | Encoding and decoding method, encoding and decoding device, encoding and decoding system and storage medium | |
| KR20130084227A (en) | Image processing device and image processing method | |
| CN110784706B (en) | Information processing method, encoding device, decoding device, system, and storage medium | |
| CN112788325B (en) | Image processing method, encoding device, decoding device and storage medium | |
| CN111988621A (en) | Video processor training method and device, video processing device and video processing method | |
| WO2021087810A1 (en) | Information processing methods and systems, and encoding apparatus, decoding apparatus and storage medium | |
| CN110771167A (en) | Video compression method, device, computer system and movable equipment | |
| CN110855997B (en) | Image processing method and device and storage medium | |
| CN110661975B (en) | Image encoding and decoding method and device, electronic equipment and storage medium | |
| CN111225218A (en) | Information processing method, encoding device, decoding device, system, and storage medium | |
| JP2010212800A (en) | Image processing apparatus and method | |
| JP6311141B2 (en) | Filter selection device, filter device and program thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200218 |