CN114391259B

CN114391259B - Information processing method, terminal device and storage medium

Info

Publication number: CN114391259B
Application number: CN201980100362.9A
Authority: CN
Inventors: 贾玉虎
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-11-06
Filing date: 2019-11-06
Publication date: 2024-05-31
Anticipated expiration: 2039-11-06
Also published as: WO2021087819A1; CN114391259A

Abstract

The invention discloses an information processing method, which comprises the following steps: acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; acquiring video image data of the target object through an image sensor; and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream. The invention also discloses another information processing method, terminal equipment and a storage medium.

Description

Information processing method, terminal device and storage medium

Technical Field

The present invention relates to computer technologies, and in particular, to an information processing method, a terminal device, and a storage medium.

Background

In the current society, more and more terminals are provided with camera devices, so that users can take pictures or take videos at any time and any place conveniently. In practical application, the encoding end obtains depth information Of the target object through a depth information sensor such as a Time Of Flight (TOF) camera and a binocular camera by using an existing image pickup device, and the decoding end recovers a depth image Of the target object through the depth information. But the depth image provides only depth information of the target object and does not improve image quality of a video image of the target object.

Disclosure of Invention

The embodiment of the invention provides an information processing method, terminal equipment and a storage medium, which can improve the image quality of a video image of a target object.

In a first aspect, an embodiment of the present invention provides an information processing method, including:

acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

acquiring video image data of the target object through an image sensor;

and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream.

In a second aspect, an embodiment of the present invention provides an information processing method, including:

receiving a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained under the condition that a depth information sensor obtains depth information of a target object, the video image data is obtained by an image sensor, and the original depth information represents an acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

decoding the video image code stream to obtain the original depth information and a video image corresponding to the video image data;

and carrying out image processing on the original depth information and the video image to obtain a target video image.

In a third aspect, an embodiment of the present invention provides a terminal device, including:

The first acquisition unit is configured to acquire original depth information corresponding to the depth information under the condition that the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information;

A second acquisition unit configured to acquire video image data of the target object through an image sensing unit;

the coding unit is configured to perform combined coding on the original depth information and the video image data to obtain a video image code stream;

and the output unit is configured to output the video image code stream.

In a fourth aspect, an embodiment of the present invention provides a terminal device, including:

The receiving unit is configured to receive a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained when the depth information of a target object is obtained through the depth information sensing unit, and the video image data is obtained through the image sensing unit; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensing unit or information beyond the acquired depth information;

The decoding unit is configured to decode the video image code stream to obtain the original depth information and a video image corresponding to the video image data;

And the image processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.

In a fifth aspect, an embodiment of the present invention provides a terminal device, including a processor and a memory configured to store a computer program capable of being executed on the processor, where the processor is configured to execute steps of an information processing method executed by the terminal device when the computer program is executed.

In a sixth aspect, an embodiment of the present invention provides a storage medium storing an executable program, where the executable program, when executed by a processor, implements the information processing method executed by the terminal device.

The information processing method provided by the embodiment of the invention comprises the following steps: acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired by a depth information sensor at an encoding end; acquiring video image data of the target object through an image sensor; and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream. Receiving a video image code stream at a decoding end; decoding the video image code stream to obtain the original depth information and a video image corresponding to the video image data; and carrying out image processing on the original depth information and the video image to obtain a target video image. The original depth information obtained by the depth sensor is directly written into a video image code stream at the encoding end, and is analyzed at the decoding end, so that a video image obtained by the original depth information on image data collected by the image sensor is obtained through analysis, a target video image is obtained, the quality of the video image is improved, and more real image video experience is brought to a user.

Drawings

FIG. 1A is a schematic diagram of an alternative information handling system according to an embodiment of the present invention;

FIG. 1B is a schematic diagram of an alternative structure of an encoding end according to an embodiment of the present invention;

FIG. 1C is a schematic diagram of an alternative decoding end according to an embodiment of the present invention

FIG. 2 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an alternative processing flow of an information processing method according to an embodiment of the present invention;

FIG. 8A is a schematic diagram of an alternative architecture of an information handling system according to an embodiment of the present invention;

FIG. 8B is a schematic diagram of an alternative architecture of an information handling system according to an embodiment of the present invention;

FIG. 9A is a schematic diagram of an alternative decoding side according to an embodiment of the present invention;

FIG. 9B is a schematic diagram of an alternative decoding side according to an embodiment of the present invention;

FIG. 9C is a schematic diagram of an alternative decoding side according to an embodiment of the present invention;

FIG. 9D is a schematic diagram of an alternative decoding side according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of sampling original depth information according to an embodiment of the present invention;

FIG. 11 is a schematic diagram of an alternative configuration of a terminal device embodying the present invention;

FIG. 12 is a schematic diagram of an alternative configuration of a terminal device according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of an alternative electronic device according to an embodiment of the present invention.

Detailed Description

So that the manner in which the features and techniques of the embodiments of the present invention can be understood in more detail, a more particular description of the invention, briefly summarized below, may be had by reference to the appended drawings, which are not intended to be limiting of the embodiments of the present invention.

Before describing the information processing method provided by the embodiment of the invention in detail, a depth image process is described.

The depth image (DEPTH IMAGE) is also called a distance image (RANGE IMAGE), and is an image in which the distance (depth) from the image sensor to each point in the scene is taken as a pixel value, and can directly reflect the geometry of the visible surface of the target object. The depth image can be calculated as point cloud data through coordinate conversion, and the point cloud data with regular and necessary information can also be reversely calculated as depth image data.

Here, the encoding end performs video encoding on the depth image captured and formed by the depth information sensor to obtain encoded depth image information, and the decoder end can only recover the depth image according to the encoded depth image information. But the depth information sensor receives an information amount far beyond the information amount of the ultra-deep image. This massive information is discarded as redundancy after the depth image is generated. Therefore, in the above-described scheme, other effects of the redundant information, such as image enhancement at the decoding end, etc., are not considered.

In view of the above, embodiments of the present invention provide an information processing method, which can be applied to an information processing system,

By way of example, the information handling system 100 to which embodiments of the present invention may be applied may be as shown in FIG. 1A. The information processing 100 may include an encoding side 101 and a decoding side 102. The encoding end 101 is configured to collect video image data and original depth information, and encode the video image data and the original depth information to form a video image code stream. The decoding end 120 is configured to decode the image video code stream to obtain video image data and original depth information, and perform image processing on the video image data and the original depth information to obtain a target video image.

Encoding end 101 and decoding end 102 may include handsets including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, smart phones, etc., televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, or the like.

As shown in fig. 1A, the decoding end 102 may receive the coded video image stream from the coding end 101 via the link 103. Link 103 may comprise one or more media and/or devices capable of moving a video image bitstream from encoding end 101 to decoding end 102.

In an example, link 103 may include one or more communication media that enable encoding end 101 to send encoded video data directly to decoding end 102 in real-time. In this example, the encoding end 101 may modulate the video image bitstream according to a communication standard (e.g., a wireless communication protocol), and may send the modulated video image bitstream to the decoding end 102.

In one example, link 103 may comprise a storage medium storing a video image bitstream formed by encoding end 101. In this example, the decoding side 102 may access the storage medium via disk access or card access. The storage medium may include a variety of locally accessed data storage media such as blu-ray discs, DVDs, CD-ROMs, flash memory, or other suitable digital storage media for storing video image code streams.

In yet another example, the link 103 may comprise a file server or another intermediate storage device storing the video image code stream formed by the encoding end 101. In this example, the decoding end 102 may access the video image bitstream stored at a file server or other intermediate storage device via streaming or download. The file server may be a type of server capable of storing and transmitting video image streams to the decoding end 102. File servers include web servers (e.g., for websites), file transfer protocol servers, network attached storage, and local disk drives, among others.

The decoding side 102 may access the video image bitstream via a standard data connection (e.g., an internet connection). Example types of data connections include wireless links (e.g., wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both, suitable for accessing video image code streams stored on a file server.

As shown in fig. 1B, the encoding end 101 includes: a depth information sensor 1011, an image sensor 1012, and a video image encoder 1013, the depth information sensor 1011 for acquiring original depth information, the image sensor 1012 for acquiring video image data, the video image encoder 1013 for encoding the original depth information and the video image data to form a video image code stream.

As shown in fig. 1C, the decoding end 102 includes: the video image decoder 1021 and the image processor 1022, the video image decoder 1021 is used for decoding the video image code stream to obtain the video image corresponding to the original depth information and the video image data, and the image processor 1022 is used for processing the original depth information and the video image to obtain the target video image. Here, the original depth information is applied to the video image, and a high-quality video image with high sharpness, low noise, and the like can be obtained.

In an example, as shown in fig. 1C, the decoding end 102 further includes: a depth image generator 1023 for generating a depth image based on the original depth information.

An optional processing flow of the information processing method provided by the embodiment of the invention is applied to an encoding end, as shown in fig. 2, and includes the following steps:

S201, when depth information of a target object is acquired by a depth information sensor, original depth information corresponding to the depth information is acquired.

The original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information.

The depth information sensor is a sensor capable of acquiring depth information of a target object. In one example, the depth information sensor is a TOF module employing TOF ranging methods. In one example, the depth information sensor is a binocular camera.

In the embodiment of the invention, under the condition that the depth information sensor collects the depth information, the encoding end acquires the original depth information through the depth information sensor, wherein the original depth information comprises at least one of the following components: charge information, phase information, and attribute parameters of the depth information sensor. The charge information and the phase information are information except the depth information acquired by the depth information sensor, and the attribute parameters of the depth information sensor represent the acquisition state of the depth information sensor for acquiring the depth information.

Taking the original depth information as charge information as an example, the charge information at one time point can be represented as a charge image. Here, the optical signal received when the depth information sensor collects the depth information is acquired, and the optical signal is converted into an electrical signal through photoelectric conversion, and the electrical signal is quantized to generate a charge image.

Taking the original depth information as the phase information as an example, the phase information at a time point may be embodied as a phase image.

Taking the original depth information as an example of the attribute parameter of the depth information sensor, the original depth information may include: temperature, pose and other attribute parameters.

S202, acquiring video image data of the target object through an image sensor.

The encoding end acquires video image data of a target object through an image sensor in an image preview or video shooting scene, wherein the video image data comprises at least one frame of image frame.

In the embodiment of the invention, the original depth information corresponds to the video frames one by one. In one example, different charge images or phase images correspond to different image frames, respectively.

S203, carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream.

The encoding end performs combined encoding on the original depth information and the video image data through the video image encoder, the video image encoder outputs a video image code stream, and the video image code stream output by the video image encoder is output to the decoding end, so that the decoding end performs image processing on a video image corresponding to the video image data based on the original depth information.

Optionally, the video image encoder encodes the video image frame or the original depth information by adopting a video image encoding and decoding protocol to obtain video image code stream information; the video codec protocol may be h.264, h.265, h.266, VP9, AV1, or the like.

Alternatively, the original depth information and the image video data are encoded using a video image codec protocol, where the data carried by the video image information does not include depth information.

In the embodiment of the invention, under the condition that the data carried by the video image information does not comprise the depth information, the coding end can only acquire the original depth information of the depth information sensor when acquiring the depth information, does not acquire the depth information acquired by the depth information sensor, or discards the acquired depth information.

Optionally, the original depth information and the image video data are encoded by using a video image encoding and decoding protocol, and the depth information collected by the depth information sensor is encoded by using the video image encoding and decoding protocol, where the data carried by the video image information includes: original depth information, and video image data.

In the embodiment of the invention, the processing of the depth information acquired by the depth information sensor is not limited.

Optionally, the video image encoder encodes the video image frames or the original depth information using industry standards or specific standards of a specific organization to obtain a video image code stream.

The encoding end may input all original depth information into the video image encoder to encode all original depth information, or may input only part of original depth information into the video image encoder to encode part of original depth information. Optionally, the partial original depth information is original depth information corresponding to the designated image frame. Optionally, the part of the original depth information is original depth information corresponding to the designated image position.

Taking partial original depth information as original depth information corresponding to a specified image video as an example, the merging and encoding of the original depth information and the video image data to obtain a video image code stream includes: and merging and encoding the original depth information corresponding to the appointed image frame in the image frames corresponding to the video image data and the video image data to obtain a video image code stream.

Optionally, the designated image frame is one of the image frames corresponding to the video image data. Optionally, the specified image frame includes a plurality of image frames among the image frames corresponding to the video image data.

The embodiment of the invention does not limit the number of the designated image frames.

The encoding end only carries out merging encoding on the original depth information corresponding to the appointed image frame and the video image data, and the original depth information corresponding to the non-appointed video frame except the appointed image frame in the image frames corresponding to the video image data is not encoded.

Taking partial original depth information as original depth information corresponding to a designated image position as an example, the merging and encoding of the original depth information and the video image data to obtain a video image code stream includes: and merging and encoding the original depth information corresponding to the designated image position and the video image data to obtain a video image code stream.

The designated image position is the position of the designated point in the image acquisition range. Optionally, the designated image position is a position of the designated area in the image acquisition range. The embodiment of the invention does not limit the size of the range of the designated image position or the position of the designated image position.

The encoding end only carries out merging encoding on the original depth information corresponding to the appointed image position and the video image data, and does not encode the original depth information corresponding to the non-appointed video position except the appointed image position in the image frame.

In the embodiment of the present invention, the encoding mode of the encoding end for performing the merging encoding on the original depth information and the video image data includes one of the following:

According to the first coding mode, the original depth information and the video image data are subjected to mixed coding according to the correlation of the original depth information and the video image data, and a video image code stream is obtained;

And the second coding mode is used for independently coding the original depth information and the video image data to obtain an image video code stream comprising a first code stream and a second code stream, wherein the first code stream is the code stream obtained after the original depth information is coded, and the second code stream is the code stream obtained after the image video data is coded.

In the first encoding mode, the encoding protocol used for encoding the original depth information and the video image data is the same.

Optionally, in the first encoding mode, the encoding information in the video image code stream is mixed encoding information obtained by jointly encoding the original depth information and the video image data. The video image encoder may use spatial correlation or temporal correlation of the original depth information and the image video data to jointly encode the original depth information and the video image data.

Optionally, in the first encoding mode, the first encoding information corresponding to the original depth information is written in a designated position of the second encoding information corresponding to the video image data. Alternatively, the specified location may be an image information header, a sequence information header, an additional parameter set, or any other location.

Optionally, in the first encoding mode, the original depth information is encoded by using spatial correlation or temporal correlation of the original depth information and the image video data, so as to obtain first encoded information, encoding the video image data, so as to obtain second encoded information, and writing the first encoded information into a designated position of the first encoded information, so as to obtain the video image code stream.

In the second encoding mode, the codec protocol used for encoding the original depth information is independent of the codec protocol used for encoding the video image data. Optionally, the original depth information is encoded using the same codec protocol as the video image data. Optionally, the original depth information is encoded using a different codec than the video image data.

In one embodiment, as shown in fig. 3, before S203, the method includes:

204A, preprocessing the original depth information.

The original depth information and the video image data are subjected to merging encoding in S203, which may be performed as S203A: and merging and encoding the preprocessed original depth information and the video image data to obtain a video image code stream.

In the embodiment of the invention, the preprocessing can be one or two of modes such as filtering, denoising, signal amplification and the like, and can also be other processing modes, and the specific preprocessing can be determined according to actual conditions, so that the embodiment of the invention is not limited.

Optionally, the encoding end preprocesses the original depth information through a depth information sensor.

In one embodiment, as shown in fig. 4, before S203, the method includes:

204B, performing redundancy elimination processing on the original depth information to eliminate redundancy information in the original depth information.

The original depth information and the video image data are subjected to merging encoding in S203, which may be performed as S203B: and combining and encoding the original depth information subjected to redundancy elimination processing and the video image data to obtain a video image code stream.

The encoding end can eliminate redundant information in the original depth information by carrying out redundancy elimination processing on the original depth information, so that the information quantity of the original depth information is compressed, and the size of a video data code stream is reduced.

In an embodiment of the present invention, the performing redundancy elimination processing on the original depth information includes at least one of the following:

redundancy elimination processing is carried out on the original depth information based on phase correlation;

performing redundancy elimination processing on the original depth information based on spatial correlation;

Performing redundancy elimination processing on the original depth information based on time correlation;

Performing redundancy elimination processing on the original depth information based on a specified depth;

performing redundancy elimination processing on the original depth information based on the frequency domain correlation;

redundancy elimination processing is performed on the coded bits of the original depth information based on correlation between the coded binary data.

Optionally, the original depth information is converted into a frequency domain, and redundancy elimination processing is performed on the original depth information converted into the frequency domain based on the frequency domain correlation.

Optionally, the designated depth is the range of the depth sensitive to the scene where the target object is located, redundancy elimination processing is performed on the original depth information based on the designated depth, and the original depth information corresponding to the depth outside the range of the depth sensitive to the scene is eliminated as redundancy.

Optionally, the original depth information is entropy-encoded, and the redundancy elimination processing is performed on the encoded bits of the entropy-encoded result of the original depth information based on the correlation between the encoded binary data.

Taking the redundancy elimination processing of the original depth information based on the spatial correlation as an example, the original depth information of the coding end corresponds to at least one view; determining an interval viewpoint from at least one viewpoint, and taking original depth information corresponding to the interval viewpoint as interval original depth information; and taking the original depth information except the original depth in the original depth information as redundancy elimination, and carrying out merging coding on the interval original depth information and the video image data to obtain a video data code stream.

Taking the time correlation-based redundancy elimination processing for the original depth information as an example, the encoding end obtains the original depth information in one end time, samples the obtained original depth information based on a sampling interval, reserves the sampled original depth information, takes the original depth information except the sampled original depth information in the obtained original depth information as redundancy elimination, and performs merging encoding for the sampled original depth information and video image data to obtain a video data code stream.

An optional processing flow of the information processing method provided by the embodiment of the present invention is applied to a decoding end, as shown in fig. 5, and includes the following steps:

s501, receiving a video image code stream.

The decoding end receives the video image code stream sent by the encoding end through a link. The video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained under the condition that the depth information of a target object is obtained through a depth information sensor, and the video image data is obtained through an image sensor; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information.

S502, decoding the video image code stream to obtain the video image corresponding to the original depth information and the video image data.

Here, the video image code stream is decoded by a video image decoder, and the original depth information and the video image corresponding to the video image data are obtained.

The decoding end sends the received video image code stream to a video image decoder, and the video image decoder decodes the video image code stream.

Optionally, the video image codec and the video encoder at the encoding end support the same video image codec protocol.

Alternatively, in the case where the video image encoder performs hybrid encoding of the original depth information and the video image data,

And the video image decoder performs mixed decoding on the video image code stream to obtain original depth information and a video image corresponding to the video image data.

Optionally, when the video image encoder independently encodes the original depth information and the video image data, the video image decoder independently decodes a first code stream and a second code stream in the video image code stream, decodes the first code stream of the video image data to obtain the original depth information, and decodes the second code stream to obtain the video image corresponding to the video image data. Here, the video image corresponding to the video image data may also be referred to as an original video image. The original video image decoded from the image video stream may comprise one or more frames of the original video image.

And S503, performing image processing on the original depth information and the video image to obtain a target video image.

And carrying out image processing on the original depth information and the video image by an image processor to obtain a target video image.

After the decoding end decodes the original depth information and the video image data, the image processor acts the original depth information on the video image, and the image processing is carried out on the video image to obtain a target video image. The image quality of the target video image is higher than the original video image.

Optionally, the decoding end may perform redundancy recovery on the original depth information obtained by decoding based on the phase correlation, the spatial correlation, the time correlation, the specified depth, the frequency domain correlation, and the correlation between the encoded binary data, to obtain the original depth information after redundancy recovery, and perform image processing on the video image based on the original depth information after redundancy recovery, to obtain the target video image.

Taking the original depth information obtained by decoding based on spatial correlation for redundancy recovery, and obtaining the original depth information after redundancy recovery as an example, the decoding end independently decodes or mixes and decodes the video image code stream to obtain the original depth information of the interval view point and the video image of at least one view point; the original depth information of the interval view points is subjected to difference value to obtain the original depth information of other view points except the interval view points in at least one view point; and performing image processing on the video image by using the original depth information of the interval view point and the original depth information of other view points to obtain a target video image.

Taking the original depth information obtained by decoding based on time correlation for redundancy recovery, obtaining the original depth information after redundancy recovery as an example, the decoding end independently decodes or mixes and decodes the video image code stream to obtain the original depth information after sampling, and recovers the original depth information between the adjacent original depth information after sampling based on the original depth information after sampling adjacent in time, and performs image processing on the video image by utilizing the original depth information obtained by decoding and the original depth information recovered to obtain the target video image.

Optionally, the video image decoder and the image processor are independent of each other. Optionally, the image processor is integrated within the video image decoder.

In an example, taking the original depth information as charge information as an example, the performing image processing on the original depth information and the video image to obtain a target video image includes: and denoising or adjusting white balance of the video image according to the original depth information to obtain the target video image.

In an example, taking the original depth information as phase information as an example, the performing image processing on the original depth information and the video image to obtain a target video image includes: and performing deblurring processing on the video image according to the original depth information to obtain the target video image.

An image processor in the decoding end analyzes each phase information to obtain an analysis result, and deblurs the video frames corresponding to the analysis result to obtain a target video image.

In an example, in a high dynamic range (HDR, high Dynamic Range) video, each frame of HDR image is obtained by fusing 1 long-exposure image and 1 short-exposure image, and at the current moment, for the same scene, the image sensor is controlled to shoot the long-exposure image and the short-exposure image, and the depth information sensor is controlled to shoot the phase image, wherein the phase image is used as original depth information; performing mixed coding or independent coding on the phase image and the long exposure image, and performing mixed coding or independent coding on the phase image and the short exposure image to obtain a video image code stream; outputting the video image code stream to a decoding end; decoding the long exposure image, the short exposure image and the phase image from the video image code stream by a decoding end; respectively deblurring the long-exposure image and the short-exposure image by using the phase image to obtain a deblurred long-exposure image and a deblurred short-exposure image; and fusing the deblurred long-exposure image and the deblurred short-exposure image to obtain a frame of clearer HDR image.

As shown in fig. 6, after S502, further includes:

s504, recovering the original depth information to obtain a depth image.

Optionally, recovering the original depth information by a depth image generator to obtain the depth image.

In the embodiment of the present invention, in fig. 6, taking S504 located after S503 as an example, the order of obtaining the target video image and obtaining the depth image is illustrated as an example, and in practical application, the execution of S504 and S503 is not performed in the order of precedence.

Optionally, the depth image generator is independent of the video image decoder. Optionally, the depth image generator is integrated within the video image decoder.

In an example, the video image decoder, the depth image generator, and the image processor are independent of each other, at which time the video image code stream is input to the video image decoder, the video image decoder outputs the original depth information and the video image, and the original depth information and the video image are input to the image processor, the original depth information is input to the depth image generator, the image processor outputs the target video image, and the depth image generator outputs the depth image.

In one example, the depth image generator and the image processor are integrated in a video image decoder, at which time the video image bitstream is input to the video image decoder, which outputs the target video image and the depth image.

In an example, the depth image generator is integrated within the video image decoder, the image processor and the video image decoder are independent of each other, at this time, the video image code stream is input to the video image decoder, the video image decoder outputs the original depth information and the target video image, and the original depth information is input to the depth image generator, which outputs the depth image.

In one example, the image processor is integrated within the video image decoder, the depth image generator and the video image decoder are independent of each other, at which time the video image code stream is input to the video image decoder, the video image decoder outputs the original depth information, the video image and the depth image, and the original depth information and the video image are input to the image processor, which outputs the target video image.

The embodiment of the invention also provides an information processing method, which is applied to an information processing system comprising an encoding end and a decoding end, as shown in fig. 7, and comprises the following steps:

S701, under the condition that the depth information sensor acquires the depth information of the target object, the encoding end acquires original depth information corresponding to the depth information.

The original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information;

S702, the encoding end obtains video image data of a target object through an image sensor;

S703, the coding end performs merging coding on the original depth information and the video image data to obtain a video image code stream, and outputs the video image code stream.

S704, the decoding end receives the video image code stream.

S705, the decoding end decodes the video image code stream to obtain the video image corresponding to the original depth information and the video image data.

S706, the decoding end performs image processing on the original depth information and the video image to obtain a target video image.

In the embodiment of the invention, the decoding end receives the video image code stream comprising the coding information of the original depth information and the coding information of the image video information, so that the decoding end can decode the original depth information and the video image from the video image code stream, and further, the decoding end not only can recover to obtain the depth image by utilizing the original depth information, but also can perform optimization processing such as denoising, white balance adjustment, deblurring and the like on the video image by utilizing the original depth information, thereby improving the information utilization rate, and compared with the original video image, the target video image obtained after the optimization processing has higher image quality.

The information processing method provided by the embodiment of the present invention is illustrated by way of an example of a scene.

The framework of the information system of the present invention is shown in fig. 8A and 8B. The video image encoder 1013 performs merging encoding on the original depth information 801 acquired by the depth information sensor 1011 and the video image data 802 acquired by the image sensor 1012 to form a video image code stream 803; after the video image decoder 1021 acquires the video image code stream 803, the video image code stream 803 is parsed to obtain the original depth information 804 and the video image 805, the depth image generator 1023 restores the original depth information 804 to obtain the depth image 806, and the image processor 1022 processes the video image 805 obtained by the video image decoder 1021 through the original depth information 804 to obtain the target video image 807. Wherein the depth image generator 1023, the image processor 1022 and the video image decoder 1021 may be independent, respectively, as shown in fig. 8A; the depth image generator 1023 and the image processor 1022 may also be included as part of the video image decoder 1021, as shown in fig. 8B.

The original depth information output by the depth information sensor can be original data information which is obtained by the depth information sensor and is not preprocessed, or intermediate data information which is obtained by preprocessing the original data information and is preprocessed; when the output information is the initial data information, the output information can be an electric signal subjected to photoelectric conversion such as charge information or phase information; when the output information is intermediate data information, the output information may be intermediate video image data capable of generating a depth image after performing phase calibration or other processing on the initial data signal.

The video image encoder encodes the input original depth information to form a video image code stream. The coding mode comprises the following steps:

The encoding method 1, utilizing the relativity of video image data and depth original information, mix the two to encode;

The encoding method 2 encodes video image data and depth original information independently.

In the encoding method 1, the encoded information of the original depth information is located at an arbitrary position such as an information header, a sequence information header, an additional parameter set, or the like of the encoded information of the video image data.

In the encoding method 2, the original depth information itself is encoded individually by using other correlations such as spatial correlation and temporal correlation of the original depth information.

In the video image encoder, the original depth information corresponding to each video image may be encoded, or only the original depth information corresponding to the specified image or the specified image position may be encoded, and the original depth information corresponding to other non-specified images or non-specified image positions may not be encoded.

For an image processor, in a photographing or previewing scene, for the generation of depth of field, original depth information can be directly utilized to act on a video image to form a target video image with depth of field, and the target video image with depth of field is generated without overlapping the depth image and the video image.

In the process of encoding the original depth information by the encoding end, in order to compress the data volume, the following correlations may be used and not limited to the following correlations are used to eliminate redundancy:

1. If the original depth information comprises phase information of a plurality of video images, eliminating phase data redundancy by utilizing correlation among phases; if the original depth information is other data, eliminating data redundancy by utilizing the spatial correlation among the data and other correlation;

2. Eliminating data redundancy by utilizing the time correlation of the original depth information;

3. eliminating scene-based data redundancy using a specified depth;

4. converting the original depth information into a frequency domain, and eliminating data redundancy of the frequency domain by utilizing frequency domain correlation;

5. Utilizing the correlation between the encoded binary data to eliminate encoded bit redundancy; wherein the encoding here may be entropy encoding.

In the embodiment of the invention, in the video image code stream containing the original depth information formed by the video image encoder, the original depth information and the video image data can be independently decoded, namely the video image code stream has decoupling property or independence, so that a video image decoder adopting various video image standard coding and decoding protocols can only extract video images from the video image code stream without extracting the original depth information, and can also only extract the original depth information without extracting the video images.

As shown in fig. 9A to 9D, for a video image decoder, a depth image generator, and an image processor, the three cooperate to decode a video image code stream according to a video image standard codec protocol, and generate a processed image and original depth information; the video image standard coding and decoding protocol can be a private standard customized by manufacturers or an industry standard. The video image decoder, the depth image generator and the image processor are composed of the following three modes:

In the composition mode 1, as shown in fig. 9A, the video image decoder 1021, the depth image generator 1023 and the image processor 1022 are independent of each other, and after the video image decoder 1021 parses the video image code stream 803 to obtain the video image 805 and the original depth information 804, the original depth information 804 is sent to the depth image generator 1023 to generate the depth image 806, and the video image 805 and the original depth information 804 are sent to the image processor 1022 to generate the processed target video image 807;

In the composition mode 2, as shown in fig. 9B, the depth image generator 1023 and the image processor 1022 are embedded in the video image decoder 1021, and the video image code stream 803 is processed in the video image decoder 1021, so that the depth image 806 and the processed target video image 807 are directly output.

In the composition mode 3, as shown in fig. 9C, the depth image generator 1023 is embedded in the video image decoder 1021, processes the video image code stream 803 in the video image decoder 1021, outputs the depth image 806 and the video image 805, and then sends the video image 805 and the original depth information 804 to the image processor 1022 to output the processed target video image 807;

In the composition 4, as shown in fig. 9D, the image processor 1022 is embedded in the video image decoder 1021, processes the image video code stream 803 in the video image decoder 1021, outputs the original depth information 804 and the processed target video image 807, and then sends the original depth information 804 to the depth image generator 1023 to output the depth image 806.

In the information processing method provided by the embodiment of the invention, at a coding end, original depth information obtained by a depth information sensor is subjected to video image coding to form a video image code stream for transmission; at the decoding end, the depth image can be recovered through the video image code stream, and the original video image can be processed through the original depth information obtained through analysis, so that the target video image with higher image quality is obtained.

In an example, the original depth information is phase information, a depth image can be recovered by a plurality of phase images obtained by sampling at different time points, and when the original video image is blurred due to motion, the blurred original video image can be recovered by motion estimation based on the phase information because the plurality of phase images can carry more information at different time points, so as to obtain a clearer target video image.

In yet another example, the depth information sensor is a TOF architecture or a module, the original depth information is charge information, so that not only can a depth image be generated, but also noise and external visible light of a shooting scene can be judged according to the charge information, and the charge information is used for performing de-drying and white balance adjustment on an original video image so as to obtain a video image with better image quality, so that a user can have a more beautiful and more real image and video experience.

In the embodiment of the present invention, the acquisition modes of the original depth information include, but are not limited to, the following modes:

Mode one

The method comprises the steps of adopting a continuous modulation TOF method, obtaining 8 groups of optical signals with different phases through sampling by a TOF sensor by controlling integration time under two different transmitting signal frequencies, carrying out photoelectric conversion on the 8 groups of optical signals to obtain 8 groups of charge signals, and carrying out 10-bit quantization on the 8 groups of charge signals to generate 8 original charge images; the decoding end encodes the 8 original charge images and attribute parameters such as temperature of the TOF sensor as original depth information; or preprocessing the 8 original charge images to generate 2 pieces of process depth data and one piece of background data, and encoding the 2 pieces of process depth data and one piece of background data as original depth information.

Mode two

The principle of binocular imaging is adopted, two video images obtained by shooting with a binocular camera are utilized, parallax information and other information are obtained by calculation according to the pose of the two video images, and the parallax information, camera parameters and the like are used as original depth information for encoding.

In the embodiment of the invention, taking 3-dimensional high-performance video coding (3 Dimension High Efficiency Video Coding,3D HEVC) of a coding and decoding protocol as an example, when original depth information is coded, as a possible implementation manner, each view and corresponding original depth information are coded; as another possible implementation, the original depth information may be coded at intervals based on views, i.e. since there is a strong correlation between different views at the same time, such as a phase diagram or a charge image, the amount of video image code stream data transmitted may be reduced by using the correlation. In an example, for video coding of three viewpoints, at the coding end, only the original depth data of the left and right viewpoints needs to be reserved in the video image code stream, and at the decoding end, the original depth information of the middle viewpoint can be obtained by performing interpolation processing on the original depth information of the left and right viewpoints.

In the embodiment of the invention, taking the redundancy elimination of the original depth information based on time correlation as an example, as a possible implementation manner, all original depth information is not required to be encoded, but only the original depth information acquired by a depth information sensor is required to be sampled in a sampling mode by adopting a fixed step length, and the sampling signals are encoded by a video image encoder; after the decoding end recovers the sampling signals, the original depth information which is not sampled is recovered through methods such as interpolation and the like.

In one example, as shown in fig. 10, the original depth information includes: the numbers are signal 1, signal 2, signal 3, signal 4. Signal 1, signal 4, signal 7. For example, signal 1 and signal 4 are interpolated to obtain signal 2, signal 2 and signal 4 are interpolated to obtain signal 3, and so on.

In the embodiment of the invention, in an AR scene, as a possible implementation manner, original depth information corresponding to the whole depth image is not required to be encoded, but only a part of pictures are required to be encoded, so that the encoded transmission of the appointed local original depth information is realized.

In order to implement the above information processing method, an embodiment of the present invention further provides a terminal device, where, as shown in fig. 11, a terminal device 1100 includes:

A first obtaining unit 1101, configured to obtain, when obtaining depth information of a target object by using a depth information sensing unit, original depth information corresponding to the depth information, where the original depth information characterizes an acquisition state in which the depth information sensing unit acquires the depth information or information other than the acquired depth information;

A second acquisition unit 1102 configured to acquire video image data of the target object through an image sensing unit;

An encoding unit 1103 configured to perform merging encoding on the original depth information and the video image data to obtain a video image code stream;

an output unit 1104 configured to output the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured to:

And merging and encoding the original depth information corresponding to the appointed image frame view in the image frames corresponding to the video image data and the video image data to obtain the video image code stream.

and merging and encoding the original depth information corresponding to the designated image position and the video image data to obtain the video image code stream.

And carrying out mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain the video image code stream.

Encoding the original depth information to obtain first encoded information;

Writing the first encoded information into a designated location of the video image data;

and encoding the video image data written with the first encoding information to obtain the video image code stream.

Encoding the original depth information to obtain first encoded information;

encoding the video image data to obtain second encoded information;

And combining the first coding information and the second coding information to obtain the video image code stream.

In the embodiment of the present invention, the terminal device further includes:

a preprocessing unit configured to:

And preprocessing the original depth information before carrying out combined coding on the original depth information and the video image data to obtain a video image code stream.

An elimination unit configured to:

And before the original depth information and the video image data are combined and encoded to obtain a video image code stream, carrying out redundancy elimination processing on the original depth information so as to eliminate redundancy information in the original depth information.

In an embodiment of the present invention, the cancellation unit is further configured to at least one of:

In an embodiment of the present invention, the original depth information includes at least one of the following: charge information, phase information and attribute parameters of the depth information sensing unit.

The embodiment of the invention also provides a terminal device, which comprises a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the terminal device when the computer program is run.

It should be noted that, in the embodiment of the present invention, the depth information sensing unit, the image sensing unit, and the video image encoding unit may be a depth information sensor, an image sensor, and a video image encoder, respectively.

In order to implement the above information processing method, an embodiment of the present invention further provides a terminal device, where, as shown in fig. 12, a terminal device 1200 includes:

a receiving unit 1201 configured to receive a video image code stream, where the video image code stream is obtained by performing merging encoding on original depth information and video image data, the original depth information is obtained when depth information of a target object is obtained by a depth information sensing unit, and the video image data is obtained by an image sensing unit; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensing unit or information beyond the acquired depth information;

a decoding unit 1202 configured to decode the video image code stream to obtain a video image corresponding to the original depth information and the video image data;

And the processing unit 1203 is configured to perform image processing on the original depth information and the video image to obtain a target video image.

In the embodiment of the present invention, the decoding unit 1202 is further configured to decode the video image code stream by using a video image decoding unit, so as to obtain the video image corresponding to the original depth information and the video image data;

the processing unit 1203 is further configured to perform image processing on the original depth information and the video image by using the video image decoding unit, so as to obtain a target video image.

In the embodiment of the invention, the video image decoding unit and the image processing unit are mutually independent, or the image processing unit is integrated in the video image decoding unit.

In an embodiment of the present invention, the processing unit 1203 is further configured to:

And when the original depth information is charge information, denoising or white balance adjustment is carried out on the video image according to the charge information, so as to obtain the target video image.

And when the original depth information is phase information, deblurring the video image according to the phase information to obtain the target video image.

And the generating unit is configured to restore the original depth information to obtain a depth image.

In the embodiment of the present invention, the generating unit is further configured to restore the original depth information by using a depth image generating unit, so as to obtain a depth image, and obtain the depth image.

It should be noted that, the video image decoding unit, the image processing unit, and the depth image generating unit in the embodiment of the present invention may be a video image decoder, an image processor, and a depth image generator, respectively.

Fig. 13 is a schematic diagram of a hardware composition structure of an electronic device (terminal device) according to an embodiment of the present invention, and an electronic device 1300 includes: at least one processor 1301, a memory 1302, and at least one network interface 1304. The various components in the electronic device 1300 are coupled together by a bus system 1305. It is appreciated that the bus system 1305 is used to implement connected communications between these components. The bus system 1305 includes a power bus, a control bus, and a status signal bus in addition to the data bus. But for clarity of illustration, the various buses are labeled as bus system 1305 in fig. 13.

It is to be appreciated that the memory 1302 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Wherein the nonvolatile Memory may be ROM, programmable read-Only Memory (PROM, programmable Read-Only Memory), erasable programmable read-Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable read-Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk-Only (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random Access Memory) which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronous Dynamic Random Access Memory), double data rate synchronous dynamic random access memory (ddr SDRAM, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory 1302 described in embodiments of the present invention is intended to comprise, without being limited to, these and any other suitable types of memory.

The memory 1302 in embodiments of the invention is used to store various types of data to support the operation of the electronic device 1300. Examples of such data include: any computer program for operating on the electronic device 1300, such as application 13021. The program for implementing the method of the embodiment of the present invention may be contained in the application program 13021.

The method disclosed in the above embodiment of the present invention may be applied to the processor 1301 or implemented by the processor 1301. Processor 1301 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the method described above may be performed by integrated logic circuitry in hardware in processor 1301 or instructions in software. The Processor 1301 may be a general purpose Processor, a digital signal Processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. Processor 1301 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the invention can be directly embodied in the hardware of the decoding processor or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium in the memory 1302 and the processor 1301 reads information in the memory 1302, performing the steps of the method described above in connection with its hardware.

In an exemplary embodiment, the electronic device 1300 may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, programmable logic devices (PLDs, programmable Logic Device), complex programmable logic devices (CPLDs, complex Programmable Logic Device), FPGAs, general purpose processors, controllers, MCUs, MPUs, or other electronic elements for performing the aforementioned methods.

The embodiment of the invention also provides a storage medium for storing the computer program.

Optionally, the storage medium may be applied to a terminal device in the embodiment of the present invention, and the computer program makes a computer execute corresponding flows in each method in the embodiment of the present invention, which is not described herein for brevity.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is illustrative of the preferred embodiments of the present invention and is not to be construed as limiting the scope of the invention, but rather as covering any modifications, equivalents, and improvements within the spirit and principles of the invention.

Claims

1. An information processing method, the method comprising:

acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;

acquiring video image data of the target object through an image sensor;

2. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:

And merging and encoding the original depth information corresponding to the appointed image frame in the image frames corresponding to the video image data and the video image data to obtain the video image code stream.

3. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:

4. A method according to any one of claims 1 to 3, wherein said combining encoding said original depth information and said video image data to obtain a video image bitstream comprises:

5. The method of claim 4, wherein the merging encoding of the original depth information and the video image data results in a video image bitstream, further comprising:

and writing the first coding information corresponding to the original depth information into the appointed position of the second coding information corresponding to the video image data.

6. A method according to any one of claims 1 to 3, wherein said jointly encoding said original depth information and said video image data comprises:

and respectively and independently encoding the original depth information and the video image data to obtain an image video code stream comprising a first code stream and a second code stream, wherein the first code stream is the code stream obtained by encoding the original depth information, and the second code stream is the code stream obtained by encoding the image video data.

7. The method of any of claims 1 to 6, wherein prior to combining encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:

preprocessing the original depth information;

the step of carrying out merging coding on the original depth information and the video image data to obtain a video image code stream comprises the following steps:

and merging and encoding the preprocessed original depth information and the video image data to obtain a video image code stream.

8. The method of any of claims 1 to 7, wherein prior to jointly encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:

And performing redundancy elimination processing on the original depth information to eliminate redundant information in the original depth information.

9. The method of claim 8, wherein the performing redundancy elimination processing on the original depth information includes at least one of:

10. The method of any of claims 1 to 9, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.

11. An information processing method, the method comprising:

Receiving a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained under the condition that depth information of a target object is obtained through a depth information sensor, and the video image data is obtained through an image sensor; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;

12. The method of claim 11, wherein,

Decoding the video image code stream through a video image decoder to obtain the original depth information and a video image corresponding to the video image data;

13. The method of claim 12, wherein the video image decoder and the image processor are independent of each other or the image processor is integrated within the video image decoder.

14. The method of any of claims 11 to 13, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.

15. The method of claim 14, wherein when the original depth information is charge information, the performing image processing on the original depth information and the video image to obtain a target video image includes:

and denoising or adjusting white balance of the video image according to the charge information to obtain the target video image.

16. The method of claim 14, wherein when the original depth information is phase information, the performing image processing on the original depth information and the video image to obtain a target video image includes:

and performing deblurring processing on the video image according to the phase information to obtain the target video image.

17. The method of any one of claims 11 to 16, wherein the method further comprises:

and recovering the original depth information to obtain a depth image.

18. The method of claim 17, wherein the original depth information is restored by a depth image generator resulting in the depth image.

19. A terminal device, the terminal device comprising:

The first acquisition unit is configured to acquire original depth information corresponding to the depth information under the condition that the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;

and the output unit is configured to output the video image code stream.

20. The terminal device of claim 19, wherein the encoding unit is further configured to:

21. The terminal device of claim 19 or 20, wherein the encoding unit is further configured to:

Encoding the original depth information to obtain first encoded information;

encoding the video image data to obtain second encoded information;

22. The terminal device of any of claims 19 to 21, wherein the terminal device further comprises:

a preprocessing unit configured to:

23. The terminal device of any of claims 19 to 22, wherein the terminal device further comprises:

An elimination unit configured to:

24. A terminal device, the terminal device comprising:

the receiving unit is configured to receive a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained when the depth information of a target object is obtained through the depth information sensing unit, and the video image data is obtained through the image sensing unit; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensing unit or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;

and the processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.

25. The terminal device of claim 24, wherein the terminal device further comprises:

and the generating unit is configured to restore the depth information to obtain a depth image.

26. A terminal device comprising a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the information processing method of any of the preceding claims 1 to 10 or the steps of the information processing method of any of the preceding claims 11 to 18 when running the computer program.

27. A storage medium storing an executable program which, when executed by a processor, implements the information processing method of any one of the above claims 1 to 10, or implements the information processing method of any one of the above claims 11 to 18.