WO2022262858A1 - 图像传输方法、图像显示及处理设备、及图像传输系统 - Google Patents

图像传输方法、图像显示及处理设备、及图像传输系统 Download PDF

Info

Publication number
WO2022262858A1
WO2022262858A1 PCT/CN2022/099504 CN2022099504W WO2022262858A1 WO 2022262858 A1 WO2022262858 A1 WO 2022262858A1 CN 2022099504 W CN2022099504 W CN 2022099504W WO 2022262858 A1 WO2022262858 A1 WO 2022262858A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image data
sub
data
target
Prior art date
Application number
PCT/CN2022/099504
Other languages
English (en)
French (fr)
Inventor
吴健
Original Assignee
青岛小鸟看看科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛小鸟看看科技有限公司 filed Critical 青岛小鸟看看科技有限公司
Priority to US17/882,381 priority Critical patent/US11758108B2/en
Publication of WO2022262858A1 publication Critical patent/WO2022262858A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/398—Synchronisation thereof; Control thereof

Definitions

  • Embodiments of the present disclosure relate to the technical field of image transmission, and more specifically, to an image transmission method, an image display and processing device, and an image transmission system.
  • Head Mounted Display is a display device that can be worn on the user's head, which can realize virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR) and mixed reality (Mixed Reality) , MR) and other different effects.
  • VR Virtual Reality
  • AR Augmented Reality
  • MR Mixed reality
  • HMDs render binocularly. That is: the HMD renders the left-eye image corresponding to the user's left eye and the right-eye image corresponding to the user's right eye at the same moment, and the images of the left-eye image and the right-eye image are images at the same moment.
  • a three-dimensional stereoscopic image with depth and layering can be formed in the user's mind, and the experience of virtual reality is more realistic.
  • the HMD receives left-eye image data and right-eye image data from a personal computer (Personal Computer, PC) through two independent transmission channels, so as to render the left-eye image according to the left-eye image data, and render the left-eye image according to the right-eye image data. Render the right-eye image to realize the synchronous transmission of left-eye image data and right-eye image data.
  • PC Personal Computer
  • An object of the present disclosure is to provide a new technical solution for an image transmission method, an image display and processing device, and an image transmission system.
  • an image transmission method applied to an image display device comprising: receiving multiple sets of image data sent by the image processing device through multiple sending threads; each set of image data includes The image, and its corresponding first serial number and second serial number, the sub-image is any one of multiple sub-images obtained by segmenting and processing the target data, and the target data includes any target image in multiple image frames of the target video
  • the left-eye image data and the right-eye image data corresponding to the frame the first sequence number reflects the playback order of the target image frame in a plurality of image frames, and the second sequence number reflects the order in which each sub-image is arranged in a plurality of sub-images; according to multiple The first serial number and the second serial number in a group of image data, combine and process multiple sub-images corresponding to the same image frame in multiple groups of image data, and obtain left-eye image data and right-eye image data; synchronously play the left-eye image data A left-eye image corresponding to the eye
  • the multiple sub-images corresponding to the same image frame in the multiple sets of image data are combined to obtain the left-eye image data and the right-eye image data.
  • Image data including: according to the first serial numbers in multiple sets of image data, determine multiple sub-images corresponding to the same image frame; according to the second serial numbers of multiple sub-images corresponding to the same image frame, determine the corresponding Multiple sub-images of the image frame are sequentially stored in the preset buffer area, correspondingly obtaining left-eye image data and right-eye image data.
  • receiving multiple sets of image data sent by the image processing device through multiple sending threads includes: receiving multiple target RTP data packets corresponding to multiple sets of image data sent by the image processing device through multiple sending threads ; Wherein, a set of image data is encapsulated in the target RTP packet.
  • the image data further includes: a preset image data identifier; receiving multiple target RTP data packets corresponding to multiple groups of image data sent by the image processing device through multiple sending threads, including: receiving the RTP data packet ; When the RTP data packet includes a preset image data identifier, determine that the RTP data packet is the target RTP data packet.
  • an image transmission method applied to an image processing device comprising: performing segmentation processing on the target data to obtain multiple sub-images; wherein the target data includes multiple image frames of the target video Left-eye image data and right-eye image data corresponding to any target image frame in any target; Determine the first sequence number and the second sequence number corresponding to each sub-image; Wherein, the first sequence number reflects that the target image frame is in a plurality of image frames The playing sequence of the sub-images, the second serial number reflects the arrangement order of each sub-image in multiple sub-images; according to the determination result, multiple sets of image data corresponding to multiple sub-images are generated; each set of image data includes: sub-image, and its corresponding first sequence number and second sequence number; sending multiple sets of image data to the image display device through multiple sending threads, so that the image display device No., combine the sub-images in multiple sets of image data to obtain left-eye image data and right-eye image data.
  • generating multiple sets of image data in one-to-one correspondence with the multiple sub-images includes: sub-image in each set of image data, the first serial number and the second serial number corresponding to the sub-image Encapsulated in one target RTP data packet to obtain multiple target RTP data packets; sending multiple sets of image data to the image display device through multiple sending threads, including: according to the number of multiple sending threads and the number of multiple target RTP data packets The quantity determines the quantity of the target RTP packets corresponding to each sending thread; according to the quantity of the target RTP packets corresponding to each sending thread, a plurality of target RTP packets are distributed to a plurality of sending threads for sending.
  • the image data further includes: a preset image data identifier, so that when the image display device receives multiple RTP data packets sent by the image processing device through multiple sending threads, it will include the RTP data packet of the image data identifier Determined as the target RTP packet.
  • an image display device including: providing an image processing device, including a memory and a processor, the memory is used to store a computer program; the processor is used to execute the computer program, to A method for realizing the above-mentioned first aspect.
  • an image processing device including a memory and a processor, the memory is used to store a computer program; the processor is used to execute the computer program, so as to implement the method of the second aspect above.
  • an image transmission system including: the image display device according to the above third aspect and the image processing device according to the above fourth aspect.
  • a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the computer program according to the first aspect or the second aspect of the present disclosure is implemented. aspects of the method.
  • each set of image data includes a sub-image and its corresponding first The serial number and the second serial number, the sub-image is any of the multiple sub-images obtained after the target data is segmented and processed, and the target data includes the left-eye image data corresponding to any target image frame in the multiple image frames of the target video and
  • the first sequence number reflects the playback order of the target image frame in multiple image frames
  • the second sequence number reflects the order in which each sub-image is arranged in multiple sub-images
  • a serial number and a second serial number combining and processing multiple sub-images corresponding to the same image frame in multiple sets of image data to obtain left-eye image data and right-eye image data, and then synchronously play the left-eye image data corresponding to the left-eye image data
  • the embodiments of the present disclosure can transmit left-eye image data and right-eye image data through multiple sending threads, so as to effectively shorten the time for the image processing device to transmit image data to the image display device and improve image transmission efficiency.
  • the embodiment of the present disclosure can synchronously play the corresponding left-eye image and right-eye image after obtaining the left-eye image data and right-eye image data, so as to ensure the synchronization of displaying the left-eye image and the right-eye image.
  • FIG. 1 is a schematic diagram of image data processing of PC and VR equipment in the related art
  • Fig. 2 is a method flowchart of an image transmission method according to some embodiments of the present disclosure
  • Fig. 3A is a schematic diagram of cutting left-eye image data and right-eye image data of the same frame of original image according to some embodiments of the present disclosure
  • Fig. 3B is another schematic diagram of cutting left-eye image data and right-eye image data of the same frame of original image according to some embodiments of the present disclosure
  • 3C is a schematic diagram of the relationship between an entire data packet that needs to be transmitted by multiple sending threads at a time and multiple target RTP data according to some embodiments of the present disclosure
  • FIG. 4 is a schematic diagram of the format of a header of a target RTP data packet according to some embodiments of the present disclosure
  • FIG. 5 is a schematic diagram of the format of an extension header of a target RTP packet according to some embodiments of the present disclosure
  • Fig. 6 is a method flowchart of another image transmission method according to some embodiments of the present disclosure.
  • Fig. 7 is a schematic diagram of sub-image cache arrangement in multiple target RTP data packets in a preset buffer area according to some embodiments of the present disclosure
  • Fig. 8 is a functional structural block diagram of an image transmission device according to some embodiments of the present disclosure.
  • Fig. 9 is a functional structural block diagram of another image transmission device according to some embodiments of the present disclosure.
  • Fig. 10 is a schematic diagram of a hardware structure of an image processing device according to some embodiments of the present disclosure.
  • Fig. 11 is a schematic diagram of a hardware structure of an image display device according to some embodiments of the present disclosure.
  • Fig. 12 is a schematic diagram of an image transmission system according to some embodiments of the present disclosure.
  • VR devices can perform binocular rendering, so that a three-dimensional image with depth and layering can be formed in the user's mind, making the user's experience of virtual reality more realistic.
  • the VR device can be connected to a personal computer (Personal Computer, PC) in a wired or wireless manner.
  • PC Personal Computer
  • the PC runs the application and plays the application screen.
  • the PC runs the video application to play the video screen
  • the PC runs the game application to play the game screen.
  • the PC captures the application screen, and processes the same frame of application screen into a left-eye image corresponding to the left eye and a right-eye image corresponding to the right eye.
  • the PC has two transmission channels, the left-eye data channel and the right-eye data channel (Dual channel of PC in Fig. 1).
  • Encoders are respectively arranged in the two transmission channels for encoding data (as shown in the dual-channel encoding in FIG. 1 ).
  • the left-eye image data corresponding to the left-eye image on the PC is transmitted through the left-eye data channel.
  • the left-eye image data is transmitted to the encoder corresponding to the left-eye data channel, it is encoded by the encoder.
  • the encoded left-eye The image data continues to be transmitted in the left-eye data channel of the PC, and finally sent to the left-eye data channel of the VR device; similarly, the right-eye image data corresponding to the right-eye image on the PC is transmitted through the right-eye data channel, and the After the image data is transmitted to the encoder corresponding to the right-eye data channel, it is encoded by the encoder.
  • the encoded right-eye image data continues to be transmitted in the right-eye data channel on the PC side, and finally sent to the right-eye data channel of the VR device. .
  • a two-channel transmission as shown in FIG. 1 is realized.
  • the VR device also has two transmission channels, a left-eye data channel and a right-eye data channel (as shown in the dual channels in the VR device in Figure 1). Similar to the PC, the left-eye data channel and the right-eye data channel of the VR device are respectively provided with a decoder and a renderer.
  • the left-eye data channel of the VR device receives the encoded left-eye image data from the PC, and transmits the encoded left-eye image data through the left-eye data channel, and the encoded left-eye image data is transmitted to the left eye of the VR device. After being decoded by the decoder corresponding to the eye data channel, the left-eye image data is obtained.
  • the left-eye image data continues to be transmitted in the left-eye data channel of the VR device, and is transmitted to the left-eye data channel of the VR device. After the renderer, it is rendered by the renderer and displayed on the display screen of the VR device.
  • the right-eye data channel of the VR device receives the encoded right-eye image data from the PC.
  • the processing method for the encoded right-eye image data in the VR device is similar to the above-mentioned processing method for the encoded left-eye image data. Specifically, you can Participate in the above-mentioned processing process of the left-eye image data, which will not be repeated here.
  • the user's left eye sees the left-eye image displayed on the VR device display
  • the user's right eye sees the right-eye image displayed on the VR device display.
  • Channel rendering enables the user to form a three-dimensional image with depth and layering in the user's mind, making the user's experience of virtual reality more realistic.
  • a single thread is used to transmit the above image data.
  • the amount of data of the image frame in the image is large, such as an intra-frame coding frame in the image frame (ie: I frame, the frame is completely preserved) or an instantaneous decoding refresh (Instantaneous Decoding Refresh, IDR) frame
  • IDR Instantaneous Decoding Refresh
  • the present disclosure provides an image transmission method, an image display and processing device, and an image transmission system to effectively solve the problem of image data transmission in the process of transmitting image data to a binocular rendered image display device.
  • Fig. 2 is a schematic flowchart of an image transmission method according to some embodiments of the present disclosure.
  • the method embodiment may be implemented by an image processing device.
  • the image display device may be, for example, a PC.
  • the image transmission method includes the following steps S210-S250:
  • Step S210 Segment the target data to obtain a plurality of sub-images; wherein, the target data includes left-eye image data and right-eye image data corresponding to any target image frame in the multiple image frames of the target video.
  • the left-eye image data is the image data corresponding to the left-eye image corresponding to the target image frame
  • the right-eye image data is the image data corresponding to the right-eye image corresponding to the target image frame.
  • the left-eye image data may be segmented first, and then the right-eye image data may be segmented to obtain multiple sub-images.
  • the right-eye image data may be segmented first, and then the left-eye image data may be segmented to obtain multiple sub-images.
  • the left-eye image data and right-eye image data corresponding to the target image frame are divided to obtain a plurality of data lengths smaller sub-images, the data length of each sub-image is less than or equal to 1400 bytes.
  • Step S220 Determine the first sequence number and the second sequence number corresponding to each sub-image; wherein, the first sequence number reflects the playback order of the target image frame in multiple image frames, and the second sequence number reflects the sequence of each sub-image in multiple sub-images. The sort order in the image.
  • the multiple image frames may be numbered sequentially according to the playing sequence of the multiple image frames in the target video, and the number of each image frame is the first sequence number corresponding to the image frame.
  • the number of the target image frame is the first serial number corresponding to the target image frame.
  • the first serial number is set as frameIndex(0) ⁇ frameIndex(g).
  • the multiple sub-images are obtained by segmenting the target image frame, and the serial number of each sub-image in the multiple sub-images is the first serial number of the target image frame. In the case that the first sequence numbers of the multiple sub-images are the same, the multiple sub-images are from the same image frame.
  • the second sequence number is set as packetIndexInframe(0) ⁇ packetIndexInframe(k).
  • packetIndexInframe(0) 0
  • packetIndexInframe(1) 1
  • packetIndexInframe(2) 2
  • PacketIndexInframe(k) k.
  • k is a natural number and k>1. If the arrangement order of a sub-image among the multiple sub-images is 6, then the second sequence number of the sub-image is 6.
  • Step S230 According to the determination result, generate multiple sets of image data corresponding to multiple sub-images; each set of image data includes: a sub-image, and its corresponding first serial number and second serial number.
  • each group of image data can be encapsulated in a target real-time transport protocol (Real-time Transport Protocol, RTP) packet to obtain multiple target RTP packets.
  • RTP Real-time Transport Protocol
  • Each target RTP packet corresponds to a set of image data.
  • the sub-image and the first sequence number and the second sequence number corresponding to the sub-image may be written into a target RTP data packet.
  • the format of the target RTP data packet is the format of an RTP data packet.
  • V is the version number of the RTP protocol, which occupies 2 bits.
  • the current protocol version number of the RTP data packet shown in FIG. 4 is 2.
  • CC is a special source (CSRC) counter, which occupies 4 bits and indicates the number of CSRC identifiers.
  • M is a marker, which occupies 1 bit. Different payloads have different meanings.
  • PT is the type of payload, which occupies 7 bits, and is used to describe the type of payload in the RTP message, such as Chinese for Global System for Mobile Communications (Global System for Mobile Communications, GSM) audio, JPEM (Joint Photographic Experts Group) image, etc.
  • GSM Global System for Mobile Communications
  • JPEM Joint Photographic Experts Group
  • SN sequence number in Figure 4
  • the sender increases the value of this field by 1 after sending an RTP packet, and the receiver can detect packet loss and recovery from this field packet sequence. The initial value of the serial number is random.
  • timestamp in Figure 4 is a timestamp, which occupies 32 bits, and records the sampling time of the first byte of data in the packet. It is indispensable for removing jitter and achieving synchronization.
  • SSRC that is, “synchronization source (SSRC) identifier” in Figure 4
  • SSRC synchronization source
  • CSRC distributed source identifiers
  • the MTU is 1500 bytes.
  • the whole data packet may include the above-mentioned multiple target RTP data packets (such as the target RTP data packet RTP[0], the target RTP data packet RTP[1] in FIG. 3C ], target RTP packet RTP[2]).
  • the relationship between multiple target RTP data packets and the entire data packet is shown in FIG. 3C .
  • each set of image data can also be set in a format similar to the data format in the above-mentioned target RTP data packet.
  • the embodiment of the present disclosure does not limit the specific format of each group of image data.
  • Step S240 Send multiple sets of image data to the image display device through multiple sending threads, so that the image display device can perform an operation on sub-items in the multiple sets of image data according to the first sequence number and the second sequence number in the multiple sets of image data.
  • the images are combined and processed to obtain left-eye image data and right-eye image data.
  • each sending thread can send a group or w groups of image data; where w is natural data and w>1.
  • multiple sending threads may The quantity and the quantity of a plurality of target RTP data packets, determine the quantity of the target RTP data packets corresponding to each sending thread, then according to the quantity of the target RTP data packets corresponding to each sending thread, distribute the multiple target RTP data packets up to Sending thread to send.
  • the process of assigning multiple target RTP data packets to multiple sending threads for sending is as follows: the number of multiple sending threads is M, where M is a natural number and M>1.
  • the number of multiple target RTP data packets is N, where N is a natural number and N>1, then the number of target RTP data packets sent by each sending thread is (N/M).
  • Corresponding sequence numbers may be set for the N target RTP data packets, and N target RTP data packets with sequence numbers 0 to N are obtained.
  • the first sending thread sends the target RTP packet with the sequence number 0 to (N/M), and the second sending thread sends the target RTP packet with the sequence number [(N/M)+1] to The target RTP packet with the number [2*(N/M)]...and so on.
  • N is not an integral multiple of M
  • the number of target RTP packets sent by the last sending thread is smaller than the number of target RTP packets sent by other sending threads.
  • the M threads After the M threads send the N target RTP data packets, they temporarily enter the dormant state, and after being woken up to send the multiple target RTP data packets next time, the multiple target RTP data packets are sent.
  • the left-eye image data and right-eye image data corresponding to the target image frame in the target video can be divided into a plurality of sub-images with smaller data lengths, and then each sub-image and the corresponding sub-image
  • the first sequence number and the second sequence number are used as a set of image data to obtain multiple sets of image data, and send multiple sets of image data through multiple sending threads, so as to effectively shorten the time for the image processing device to transmit image data to the image display device time.
  • the time required for transmitting the video frame can be greatly shortened, and the image transmission efficiency can be improved.
  • the image data further includes: a preset image data identifier, so that when the image display device receives multiple RTP packets sent by the image processing device through multiple sending threads, it will include the RTP of the image data identifier The packet is determined to be a target RTP packet.
  • Fig. 6 is a schematic flowchart of an image transmission method according to some embodiments of the present disclosure.
  • the method embodiment can be implemented by an image display device.
  • the image display device may be a VR device such as an HMD.
  • the image display device can have a transmission channel.
  • the image transmission method includes the following steps S610-S630:
  • Step S610 Receive multiple sets of image data sent by the image processing device through multiple sending threads; wherein, the image data includes a sub-image, a first serial number and a second serial number corresponding to the sub-image.
  • step S610 multiple sets of image data sent by the multiple sending threads may be correspondingly received through multiple receiving threads.
  • Multiple receiving threads correspond to multiple sending threads one by one.
  • Each set of image data includes a sub-image, and its corresponding first serial number and second serial number.
  • the sub-image is any one of multiple sub-images obtained after segmentation of the target data.
  • the target data includes multiple images of the target video.
  • the first serial number reflects the playback order of the target image frame in multiple image frames
  • the second serial number reflects the sequence of each sub-image in multiple sub-images. sort order in .
  • the multiple sets of image data sent by the receiving image processing device through multiple sending threads may be: multiple target RTPs corresponding to the multiple sets of image data sent by the receiving image processing device through multiple sending threads data packet; wherein, a set of image data is encapsulated in each target RTP data packet.
  • Step S620 According to the first serial number and the second serial number in the multiple sets of image data, combine the multiple sub-images corresponding to the same image frame in the multiple sets of image data to obtain the left-eye image data and the right-eye image data.
  • the first sequence number reflects the playing order of the target image frame in the multiple image frames of the target video.
  • the first sequence numbers of the left-eye image data and right-eye image data corresponding to the same image frame are the same.
  • multiple sub-images corresponding to the same image frame may be determined according to the first sequence numbers in multiple sets of image data, and then the multiple sub-images corresponding to the same image frame may be determined according to the second sequence numbers of the multiple sub-images
  • a plurality of sub-images are sequentially stored in a preset buffer area, correspondingly obtaining left-eye image data and right-eye image data.
  • the first sequence number and the second sequence number are stored in the target The position of the extension header of the RTP data packet, so first read the first sequence number and the second sequence number of the extension header of the target RTP data packet, after determining multiple target RTP data packets corresponding to the same image frame according to the first sequence number , according to the second sequence numbers in the multiple target RTP data packets, sequentially store the sub-images in the multiple target RTP data packets into a preset buffer area.
  • the sub-pictures in a plurality of target RTP data packets are arranged sequentially RTP[0], RTP[1]...RTP[packetIndexInframe(K-1)], RTP[packetIndexInframe( K)].
  • the second serial number RTP[packetIndexInframe(K-1)] and RTP[packetIndexInframe(K)] corresponding to the adjacent two sub-images are adjacent, and the starting positions of the adjacent sub-images differ by the data length of one sub-image; where , K is a natural number, and k ⁇ K ⁇ 0.
  • Step S630 synchronously playing the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data.
  • the left-eye image data and right-eye image data corresponding to the same original image frame are transmitted to the decoder for decoding, and the decoded left-eye image data and right-eye image data are rendered synchronously, so that the left-eye image data can be displayed synchronously
  • the corresponding left-eye image and right-eye image data correspond to the right-eye image.
  • the image data further includes: a preset image data identifier.
  • receiving a plurality of target RTP data packets sent by the image processing device through a plurality of sending threads corresponding to a plurality of sets of image data includes: receiving the RTP data packet; including preset image data in the RTP data packet In the case of identification, determine that the RTP data packet is the target RTP data packet.
  • each set of image data includes a sub-image and its corresponding first The serial number and the second serial number, the sub-image is any of the multiple sub-images obtained after the target data is segmented and processed, and the target data includes the left-eye image data corresponding to any target image frame in the multiple image frames of the target video and
  • the first sequence number reflects the playback order of the target image frame in multiple image frames
  • the second sequence number reflects the order in which each sub-image is arranged in multiple sub-images
  • a serial number and a second serial number combining and processing multiple sub-images corresponding to the same image frame in multiple sets of image data to obtain left-eye image data and right-eye image data, and then synchronously play the left-eye image data corresponding to the left-eye image data
  • the embodiments of the present disclosure can transmit left-eye image data and right-eye image data through multiple sending threads, so as to effectively shorten the time for the image processing device to transmit image data to the image display device and improve image transmission efficiency.
  • the embodiment of the present disclosure can synchronously play the corresponding left-eye image and right-eye image after obtaining the left-eye image data and right-eye image data, so as to ensure the synchronization of displaying the left-eye image and the right-eye image.
  • Fig. 8 is a functional structural block diagram of an image transmission device provided according to some implementations of the present disclosure.
  • the image transmission device is applied to image processing equipment.
  • the image transmission device 80 includes:
  • the segmentation module 81 is configured to perform segmentation processing on the target data to obtain multiple sub-images; wherein, the target data includes left-eye image data and right-eye image data corresponding to any target image frame in multiple image frames of the target video.
  • the determination module 82 is used to determine the first sequence number and the second sequence number corresponding to each sub-image; wherein, the first sequence number reflects the playback order of the target image frame in multiple image frames, and the second sequence number reflects the sequence number of each sub-image Sorting order among multiple subimages.
  • the generating module 83 is configured to generate multiple sets of image data corresponding to multiple sub-images according to the determination result; each set of image data includes: a sub-image and its corresponding first serial number and second serial number.
  • the sending module 84 is configured to send multiple sets of image data to the image display device through multiple sending threads, so that the image display device can send multiple sets of image data to the multiple sets of image data according to the first sequence number and the second sequence number in the multiple sets of image data.
  • the sub-images in the image are combined to obtain left-eye image data and right-eye image data.
  • the image processing device can be, for example, a PC.
  • Fig. 9 is a functional structural block diagram of an image transmission device provided according to some implementations of the present disclosure.
  • the image transmission device is applied to image display equipment.
  • the image transmission device 90 includes:
  • the receiving module 91 is used to receive multiple sets of image data sent by the image processing device through multiple sending threads; wherein, each set of image data includes a sub-image and its corresponding first serial number and second serial number, and the sub-image It is any of multiple sub-images obtained by segmenting the target data.
  • the target data includes the left-eye image data and right-eye image data corresponding to any target image frame in the multiple image frames of the target video.
  • the first serial number reflects The playing sequence of the target image frame in the plurality of image frames, and the second sequence number reflects the arrangement sequence of each sub-image in the plurality of sub-images.
  • the processing module 92 is configured to combine and process multiple sub-images corresponding to the same image frame in the multiple sets of image data according to the first sequence number and the second sequence number in the multiple sets of image data, to obtain left-eye image data and Right eye image data.
  • the playing module 93 is used for synchronously playing the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data.
  • the image display device may be, for example, a VR device such as an HMD.
  • Fig. 10 is a schematic diagram of a hardware structure of an image processing device according to some embodiments of the present disclosure.
  • the image processing device 1000 includes a processor 1010 and a memory 1020, the memory 1020 is used to store an executable computer program, and the processor 1010 is used to execute any of the above methods according to the control of the computer program. example method.
  • the image processing device 1000 may be a PC.
  • Each module of the above image transmission device 80 may be implemented by the processor 1010 in this embodiment executing a computer program stored in the memory 1020, or may be implemented by other circuit structures, which are not limited here.
  • Fig. 11 is a schematic diagram of a hardware structure of an image display device according to some embodiments of the present disclosure.
  • the image display device 1100 includes a processor 1110 and a memory 1120, the memory 1120 is used to store an executable computer program, and the processor 1110 is used to execute any method implemented as above according to the control of the computer program. example method.
  • the image display device 1100 may be a VR device such as an HMD.
  • Each module of the above image transmission device 90 may be implemented by the processor 1110 in this embodiment executing a computer program stored in the memory 1120, or may be implemented by other circuit structures, which are not limited here.
  • FIG. 12 is a schematic diagram of an image transmission system according to some embodiments of the present disclosure. As shown in FIG. 12 , an image transmission system 1200 includes an image processing device 1000 and an image display device 1100 .
  • This embodiment provides a computer-readable storage medium, and executable commands are stored in the storage medium.
  • executable command is executed by a processor, the method described in any method embodiment of this specification (such as the above steps S210- The method corresponding to S240 or the method corresponding to the above steps S610-S630).
  • An embodiment or embodiments of the present description may be a system, method and/or computer program product.
  • a computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement various aspects of the present description.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • flash memory static random access memory
  • SRAM static random access memory
  • CD-ROM compact disc read only memory
  • DVD digital versatile disc
  • memory stick floppy disk
  • mechanically encoded device such as a printer with instructions stored thereon
  • a hole card or a raised structure in a groove and any suitable combination of the above.
  • computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
  • Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • the computer program instructions for carrying out the operations of the embodiments of the present specification may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or programmed in one or more Source or object code written in any combination of languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as “C” or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect).
  • LAN local area network
  • WAN wide area network
  • an electronic circuit such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA)
  • FPGA field programmable gate array
  • PDA programmable logic array
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
  • each block in a flowchart or block diagram may represent a module, a program segment, or a portion of an instruction that contains one or more executable instruction.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by means of hardware, implementation by means of software, and implementation by a combination of software and hardware are all equivalent.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

本公开涉及一种图像传输方法、图像显示及处理设备、及图像传输系统，涉及图像传输技术领域。本公开在图像处理设备侧将对应于目标视频中同一原始图像帧的左眼图像数据和右眼图像数据分割为多个子图像，生成与多个子图像一一对应的多组图像数据，通过多个发送线程发送多组图像数据至图像显示设备；每组图像数据包括子图像、该子图像对应的第一序列号和第二序列号，图像显示设备根据多组图像数据中的第一序列号和第二序列号对多组图像数据中的子图像进行组合，得到对应同一原始图像帧的左眼图像数据和右眼图像数据，最后播放左眼图像数据对应的左眼图像和所右眼图像数据对应的右眼图像。

Description

图像传输方法、图像显示及处理设备、及图像传输系统
优先权信息
本申请要求于2021年06月18日提交的，申请名称为“图像传输方法、图像显示及处理设备、及图像传输系统”的、中国专利申请号“202110681037.5”的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开实施例涉及图像传输技术领域，更具体地，涉及图像传输方法、图像显示及处理设备、及图像传输系统。
背景技术
头戴式显示设备(Head Mounted Display，HMD)是一种可以穿戴在用户头部的显示设备，能够实现虚拟现实(Virtual Reality，VR)、增强现实(Augmented Reality，AR)和混合现实(Mixed Reality，MR)等不同效果。
通常情况下，HMD为双目渲染。即：HMD在同一时刻渲染对应于用户左眼的左眼图像和对应于用户右眼的右眼图像，左眼图像和右眼图像的画面为同一时刻的画面。在此情况下，用户脑中能够形成一个具有景深和层次感的三维立体图像，对虚拟现实的体验更具真实感。
目前，HMD通过两个独立的传输通道，分别接收来自个人计算机(Personal Computer，PC)端的左眼图像数据和右眼图像数据，以便根据左眼图像数据渲染左眼图像、以及根据右眼图像数据渲染右眼图像，实现左眼图像数据和右眼图像数据的同步传输。实际情况中，PC端向HMD传输左眼图像数据和右眼图像数据的时间较长，传输效率较低。
发明内容
本公开的一个目的是提供一种图像传输方法、图像显示及处理设备、及图像传输系统的新的技术方案。
根据本公开的第一方面，提供了一种图像传输方法，应用于图像显示设备，该方法包括：接收图像处理设备通过多个发送线程发送的多组图像数据；每组图像数据包括子图像、及其对应的第一序列号和第二序列号，子图像是对目标数据进行分割后处理得到的多个子图像中任一，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反 映每个子图像在多个子图像中的排列顺序；根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据；同步播放左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。
可选地，根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据，包括：根据多组图像数据中的第一序列号，确定对应于同一图像帧的多个子图像；根据对应于同一图像帧的多个子图像的第二序列号，将该对应于同一图像帧的多个子图像依次存入预设缓存区，对应得到左眼图像数据和右眼图像数据。
可选地，接收图像处理设备通过多个发送线程发送的多组图像数据，包括：接收图像处理设备通过多个发送线程发送的与多组图像数据一一对应的多个目标RTP数据包；其中，目标RTP数据包中封装有一组图像数据。
可选地，图像数据还包括：预设的图像数据标识；接收图像处理设备通过多个发送线程发送的与多组图像数据一一对应的多个目标RTP数据包，包括：接收RTP数据包；在RTP数据包包括预设的图像数据标识的情况下，确定RTP数据包为目标RTP数据包。
根据本公开的第二方面，提供了一种图像传输方法，应用于图像处理设备，该方法包括：对目标数据进行分割处理，得到多个子图像；其中，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据；确定每个子图像对应的第一序列号和第二序列号；其中，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序；根据确定结果，生成与多个子图像一一对应的多组图像数据；每组图像数据包括：子图像、及其对应的第一序列号和第二序列号；通过多个发送线程发送多组图像数据至图像显示设备，以便图像显示设备根据多组图像数据中的第一序列号和第二序列号，对多组图像数据中的子图像进行组合处理，得到左眼图像数据和右眼图像数据。
可选地，根据确定结果，生成与多个子图像一一对应的多组图像数据，包括：将每组图像数据中的子图像、该子图像对应的第一序列号和第二序列号封装在一个目标RTP数据包中，得到多个目标RTP数据包；通过多个发送线程发送多组图像数据至图像显示设备，包括：根据多个发送线程的数量和多个目标RTP数据包的数量，确定每个发送线程对应的目标RTP数据包的数量；根据每个发送线程对应的目标RTP数据包的数量，将多个目标RTP数据包分配至多个发送线程进行发送。
可选地，图像数据还包括：预设的图像数据标识，以便图像显示设备接收到图像处理设备通过多个发送线程发送的多个RTP数据包的情况下，将包括图像数据标识的RTP数据包确定为目标RTP数据包。
根据本公开的第三方面，提供了一种图像显示设备，包括：提供了一种图像处理设备，包括存储器和处理器，该存储器用于存储计算机程序；该处理器用于执行该计算机程序，以实现上述第一方面的方法。
根据本公开的第四方面，提供了一种图像处理设备，包括存储器和处理器，该存储器用于存储计算机程序；该处理器用于执行该计算机程序，以实现上述第二方面的方法。
根据本公开的第五方面，提供了一种图像传输系统，包括：包括上述第三方面的图像显示设备和如上述第四方面的图像处理设备。
根据本公开的第六方面，还提供了一种计算机可读存储介质，该计算机可读存储介质上存储计算机程序，该计算机程序在被处理器执行时实现根据本公开的第一方面或第二方面的方法。
本公开实施例的一个有益效果在于，在图像显示设备一侧，能够接收图像处理设备通过多个发送线程发送的多组图像数据；每组图像数据包括子图像、及其对应的第一序列号和第二序列号，子图像是对目标数据进行分割后处理得到的多个子图像中任一，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序；然后根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据，然后同步播放左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。由此可见，本公开实施例能够通过多个发送线程传输左眼图像数据和右眼图像数据，以此来有效缩短图像处理设备向图像显示设备传输图像数据的时间，提高图像传输效率，在此过程中，本公开实施例能够在得到左眼图像数据和右眼图像数据后，同步播放对应的左眼图像和右眼图像，保证左眼图像和右眼图像显示的同步性。
通过以下参照附图对本说明书的示例性实施例的详细描述，本说明书的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本说明书的实施例，并且连同其说明一起用于解释本说明书的原理。
图1是相关技术中PC和VR设备的图像数据处理示意图；
图2是根据本公开一些实施例的一种图像传输方法的方法流程图；
图3A是根据本公开一些实施例的一种对同一帧原始图像的左眼图像数据和右眼图像数据进行切割的示意图；
图3B是根据本公开一些实施例的另一种对同一帧原始图像的左眼图像数据和右眼图像数据进行切割的示意图；
图3C是根据本公开一些实施例的一种一次需要多个发送线程传输的整个数据包与多个目标RTP数据包括的关系示意图；
图4是根据本公开一些实施例的一种目标RTP数据包的头的格式示意图；
图5是根据本公开一些实施例的一种目标RTP数据包的扩展头的格式示意图；
图6是根据本公开一些实施例的另一种图像传输方法的方法流程图；
图7是根据本公开一些实施例的一种预设缓存区中多个目标RTP数据包中的子图像缓存排列示意图；
图8是根据本公开一些实施例的一种图像传输装置的功能结构框图；
图9是根据本公开一些实施例的另一种图像传输装置的功能结构框图；
图10是根据本公开一些实施例的一种图像处理设备的硬件结构示意图；
图11是根据本公开一些实施例的一种图像显示设备的硬件结构示意图；
图12是根据本公开一些实施例的一种图像传输系统的示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本发明及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
目前，VR设备(例如HMD)可以进行双目渲染，以使得用户脑中能够形成一个具有景深和层次感的三维立体图像，使用户对虚拟现实的体验更具真实感。
如图1所示，VR设备可以通过有线或无线的方式连接个人计算机(Personal Computer，PC)。在PC端，PC运行应用，播放应用画面，例如PC运行视频应用播放视频画面，又如PC端运行游戏应用播放游戏画面等。在此情况下，PC抓取应用画面，将同一帧应用画面 处理为左眼对应的左眼图像和右眼对应的右眼图像，PC中具有左眼数据通道和右眼数据通道两个传输通道(如图1中PC的双通道)。两个传输通道中分别设置有编码器，用于对数据进行编码(如图1中的双通道编码)。
PC端中左眼图像对应的左眼图像数据通过左眼数据通道进行传输，在左眼图像数据被传输至左眼数据通道对应的编码器后，被该编码器进行编码，编码后的左眼图像数据继续在PC端的左眼数据通道中传输，最终发送给VR设备的左眼数据通道；同样地，PC端中右眼图像对应的右眼图像数据通过右眼数据通道进行传输，在右眼图像数据被传输至右眼数据通道对应的编码器后，被该编码器进行编码，编码后的右眼图像数据继续在PC端的右眼数据通道中传输，最终发送给VR设备的右眼数据通道。在此情况下，实现如图1中的双通道发送。
VR设备中也具有左眼数据通道和右眼数据通道两个传输通道(如图1中VR设备中的双通道)。与PC类似，VR设备的左眼数据通道和右眼数据通道中分别设置有解码器和渲染器。VR设备的左眼数据通道接收来自PC端的编码后的左眼图像数据，并通过左眼数据通道传输该编码后的左眼图像数据，在编码后的左眼图像数据被传输至VR设备的左眼数据通道对应的解码器后，被该解码器进行解码，得到左眼图像数据，该左眼图像数据继续在VR设备的左眼数据通道中传输，在传输至VR设备的左眼数据通道的渲染器后，被渲染器渲染，显示在VR设备的显示屏中。VR设备的右眼数据通道接收来自PC端的编码后的右眼图像数据，VR设备中对编码后的右眼图像数据的处理方式与上述对编码后的左眼图像数据的处理方式类似，具体可以参加上述对左眼图像数据的处理过程，此处不再赘述。
对于穿戴VR设备的用户而言，用户的左眼看到VR设备显示屏所显示的左眼图像，用户的右眼看到VR设备显示屏所显示的右眼图像，VR设备中通过双通道解码和双通道渲染，使得用户脑中能够形成一个具有景深和层次感的三维立体图像，使得用户对虚拟现实的体验更具真实感。
目前，在PC向VR设备传输图像数据的过程中，均采用单线程进行传输上述图像数据。在图像中图像帧的数据量较大的情况下，例如图像帧中的帧内编码帧(即：I帧，这一帧画面完整保留)或者瞬时解码刷新(Instantaneous Decoding Refresh，IDR)帧，采用单线程传输上述图像数据的时间较长，传输效率较低。
针对以上实施方式存在的技术问题，本公开提供一种图像传输方法、图像显示及处理设备、及图像传输系统，以有效解决在向双目渲染的图像显示设备传输图像数据的过程中图像数据传输效率偏低的问题。
下面，参照附图描述根据本发明的各个实施例和例子。
<方法实施例>
图2是根据本公开一些实施例的一种图像传输方法的方法流程示意图。该方法实施例可以由图像处理设备实施。该图像显示设备可以是例如可以是PC。
如图2所示，该图像传输方法包括如下步骤S210～S250：
步骤S210：对目标数据进行分割处理，得到多个子图像；其中，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据。
左眼图像数据即目标图像帧对应的左眼图像所对应的图像数据，右眼图像数据即目标图像帧对应的右眼图像所对应的图像数据。
示例地，如图3A所示，可以先对左眼图像数据进行分割，然后对右眼图像数据进行分割，得到多个子图像。
示例地，如图3B所示，可以先对右眼图像数据进行分割，然后对左眼图像数据进行分割，得到多个子图像。
示例地，由于网络最大传输单元(Maximum Transmission Unit，MTU)的限制(MTU为1500字节)，将目标图像帧对应的左眼图像数据和右眼图像数据进行分割，得到多个数据长度较小的子图像，每个子图像的数据长度小于或等于1400字节。
步骤S220：确定每个子图像对应的第一序列号和第二序列号；其中，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序。
可以根据目标视频中的多个图像帧的播放顺序，依次对多个图像帧进行编号，每个图像帧的编号即该图像帧对应的第一序列号。目标图像帧的编号即目标图像帧对应的第一序列号。
示例地，设置第一序列号为frameIndex(0)～frameIndex(g)。其中frameIndex(0)＝0，frameIndex(1)＝1，frameIndex(2)＝2……frameIndex(g)＝g，g为自然数且g≥0。若目标图像帧在目标视频中的多个图像帧的播放顺序为5，则目标图像帧的第一序列号为5。
多个子图像是由目标图像帧分割得到的，该多个子图像中每个子图像的序列号即该目标图像帧的第一序列号。多个子图像的第一序列号相同情况下，该多个子图像来自同一图像帧。
在确定每个子图像的第二序列号时，无论是先对左眼图像数据进行分割，然后对右眼图像数据进行分割，还是先对右眼图像数据进行分割，然后对左眼图像数据进行分割，将得到多个子图像先后顺序作为该多个子图像的排列顺序，根据该排列顺序对该多个子图像进行编号，得到该多个子图像对应的第二序列号。
要说明的是，该多个子图像按照上述该排列顺序进行组合后，可以得到上述左眼图像数据和右眼图像数据。
示例地，设置第二序列号为packetIndexInframe(0)～packetIndexInframe(k)。其中，packetIndexInframe(0)＝0，packetIndexInframe(1)＝1，packetIndexInframe(2)＝2……packetIndexInframe(k)＝k。k为自然数且k>1。若多个子图像中一子图像在该多个子图像中的排列顺序为6，则该子图像的第二序列号为6。
步骤S230：根据确定结果，生成与多个子图像一一对应的多组图像数据；每组图像数据包括：子图像、及其对应的第一序列号和第二序列号。
在一些实施例中，可以将每组图像数据封装在一个目标实时传输协议(Real-time Transport Protocol，RTP)数据包中，得到多个目标RTP数据包。每个目标RTP数据包对应一组图像数据。
示例地，对于多个子图像中的任一子图像，可以将该子图像和该子图像对应的第一序列号和第二序列号写入一个目标RTP数据包。目标RTP数据包的格式即一个RTP数据包的格式。
示例地，每个目标RTP数据包的头的格式如图4所示。V为RTP协议的版本号，占2位，如图4所示的RTP数据包的当前协议版本号为2。P为填充标志，占1位，如果P＝1，则在该报文的尾部填充一个或多个额外的八位组，它们不是有效载荷的一部分。X为扩展标志，占1位，如果X＝1，则在RTP报头后跟有一个扩展报头。CC为特约信源(CSRC)计数器，占4位，指示CSRC标识符的个数。M为标记，占1位，不同的有效载荷有不同的含义，对于视频，标记一帧的结束；对于音频，标记会话的开始。PT为有效载荷类型，占7位，用于说明RTP报文中有效载荷的类型，如中文为全球移动通讯系统(Global System for Mobile Communications，GSM)音频、JPEM(Joint Photographic Experts Group)图像等，在流媒体中大部分是用来区分音频流和视频流的，这样便于客户端进行解析。SN(即图4中的sequence number)为序列号，占16位，发送方在每发送完一个RTP数据包后就将该域的值增加1，接收方可以由该域检测包的丢失及恢复包序列。序列号的初始值是随机的。图4中“timestamp”为时间戳，占32位，记录了该包中数据的第一个字节的采样时刻。它是去除抖动和实现同步不可缺少的。SSRC(即图4中“synchronization source(SSRC)identifier”)为同步源标识符，占32位，同步源就是指RTP包流的来源。在同一个RTP数据包会话中不能有两个相同的SSRC值。CSRC(即图4中“contributing source(CSRC)identifiers”)为特约信源标识符，每个CSRC标识符占32位，可以有0～15个。每个CSRC标识了包含在该RTP报文有效载荷中的所有特约信源。其中，目标RTP数据包的头固定占用12字节。
该子图像对应的第一序列号和第二序列号可以写入该目标RTP数据包的扩展头(如图5所示的“header extension”部分)，目标RTP数据包扩展头的格式如图5所示，包括defined by profile字段和length。defined by profile和length共占用4个字节，第一序列号和第二序列号分别占用4个字节。在此情况下，RTP协议字段的总长度为12+4+4+4＝24字节。MTU为1500字节，为了每个目标RTP数据包能够在网络中正常传输，设定每个目标RTP数据内的子图像的长度为1400字节，在此情况下，一个目标RTP数据包的总长度为1400+24＝1424字节，保证每个目标RTP数据包能够在网络中正常传输。
示例地，对于一次需要多个发送线程传输的整个数据包，该整个数据包可以包括上述多个目标RTP数据包(如图3C中目标RTP数据包RTP[0]、目标RTP数据包RTP[1]、目标RTP数据包RTP[2]……)。多个目标RTP数据包与该整个数据包的关系如图3C所示。
当然，可以理解的是，也可以采用与上述目标RTP数据包中数据格式类似的格式设置每组图像数据。本公开实施例对每组图像数据的具体格式不作限定。
步骤S240：通过多个发送线程发送多组图像数据至图像显示设备，以便图像显示设备根据多组图像数据中的第一序列号和第二序列号，对多组图像数据中的子图像进行组合处理，得到左眼图像数据和右眼图像数据。
示例地，每个发送线程可以发送一组或w组图像数据；其中w为自然数据且w>1。
在一些实施例中，在将每组图像数据中的子图像、该子图像对应的第一序列号和第二序列号封装在一个目标RTP数据包中的情况下，可以根据多个发送线程的数量和多个目标RTP数据包的数量，确定每个发送线程对应的目标RTP数据包的数量，然后根据每个发送线程对应的目标RTP数据包的数量，将多个目标RTP数据包分配至多个发送线程进行发送。
示例地，将多个目标RTP数据包分配至多个发送线程进行发送的过程如下：多个发送线程的数量为M，M为自然数且M>1。多个目标RTP数据包的数量为N，N为自然数且N>1，则每个发送线程发送的目标RTP数据包的数量为(N/M)个。可以对N个目标RTP数据包设置对应的序列号，得到序列号为0～序列号为N的N个目标RTP数据包。在此情况下，第一个发送线程发送序列号为0～序列号为(N/M)的目标RTP数据包，第二个发送线程发送序列号为[(N/M)+1]～序列号为[2*(N/M)]的目标RTP数据包……依次类推。在N不是M的整倍数的情况下，最后一个发送线程发送的目标RTP数据包的数量小于其他发送线程发送的目标RTP数据包的数量。M个线程发送完N个目标RTP数据包后，暂时进入休眠状态，在下一次发送多个目标RTP数据包时被唤醒以后发送该多个目标RTP数据包。
在本公开实施例中，能够将目标视频中目标图像帧对应的左眼图像数据和右眼图像数据分割为多个数据长度较小的子图像，然后将每个子图像、该子图像对应的第一序列号和 第二序列号作为一组图像数据，得到多组图像数据，通过多个发送线程发送多组图像数据，以此来有效缩短图像处理设备向图像显示设备传输图像数据的时间。尤其对于数据量较大的视频帧，能够大大缩短传输该视频帧所需的时间，提高图像传输效率。
在一些实施例中，图像数据还包括：预设的图像数据标识，以便图像显示设备接收到图像处理设备通过多个发送线程发送的多个RTP数据包的情况下，将包括图像数据标识的RTP数据包确定为目标RTP数据包。
图6是根据本公开一些实施例的一种图像传输方法的方法流程示意图。该方法实施例可以由图像显示设备实施。该图像显示设备可以是例如HMD等VR设备。该图像显示设备可以具有一个传输通道。
如图6所示，该图像传输方法包括如下步骤S610～S630：
步骤S610：接收图像处理设备通过多个发送线程发送的多组图像数据；其中，图像数据包括子图像、该子图像对应的第一序列号和第二序列号。
图像处理设备及其通过多个发送线程发送的多组图像数据的过程可以参见上述实施例中的对应描述，此处不再赘述。
在步骤S610中，可以通过多个接收线程，对应接收上述多个发送线程发送的多组图像数据。多个接收线程与多个发送线程一一对应。
每组图像数据包括子图像、及其对应的第一序列号和第二序列号，子图像是对目标数据进行分割后处理得到的多个子图像中任一，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序。
第一序列号和第二序列号的获取方式可以参见上述实施例中的对应描述，此处不再赘述。
在一些实施例中，接收图像处理设备通过多个发送线程发送的多组图像数据可以是：接收图像处理设备通过多个发送线程发送的与多组图像数据一一对应的多个目标RTP数据包；其中，每个目标RTP数据包中封装有一组图像数据。
步骤S620：根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据。
第一序列号反映目标图像帧在目标视频的多个图像帧中的播放顺序，在此情况下，对应于同一图像帧的左眼图像数据和右眼图像数据的第一序列号相同。
在一些实施例中，可以根据多组图像数据中的第一序列号，确定对应于同一图像帧的多个子图像，然后根据对应于同一图像帧的多个子图像的第二序列号，将该多个子图像依 次存入预设缓存区，对应得到左眼图像数据和右眼图像数据。
在一些实施例中，在接收图像处理设备通过多个发送线程发送的与多组图像数据一一对应的多个目标RTP数据包的情况下，第一序列号和第二序列号存储在目标RTP数据包的扩展头的位置，因此先读取目标RTP数据包的扩展头的第一序列号和第二序列号，在根据第一序列号确定对应同一图像帧的多个目标RTP数据包后，根据该多个目标RTP数据包中的第二序列号，将该多个目标RTP数据包中的子图像依次存入预设缓存区。如图7所示，多个目标RTP数据包中的子图像按照第二序列号的顺序依次排列RTP[0]、RTP[1]……RTP[packetIndexInframe(K-1)]、RTP[packetIndexInframe(K)]。相邻的两个子图像对应的第二序列号RTP[packetIndexInframe(K-1)]和RTP[packetIndexInframe(K)]相邻，且相邻子图像的起始位置相差一个子图像的数据长度；其中，K为自然数，且k≥K≥0。
步骤S630：同步播放左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。
将得到对应于同一原始图像帧的左眼图像数据和右眼图像数据传输至解码器进行解码，对解码后的左眼图像数据和右眼图像数据同步进行渲染，便可以同步显示左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。
在一些实施例中，图像数据还包括：预设的图像数据标识。在此情况下，接收图像处理设备通过多个发送线程发送的与多组图像数据一一对应的多个目标RTP数据包，包括：接收RTP数据包；在RTP数据包包括预设的图像数据标识的情况下，确定RTP数据包为目标RTP数据包。
本公开实施例的一个有益效果在于，在图像显示设备一侧，能够接收图像处理设备通过多个发送线程发送的多组图像数据；每组图像数据包括子图像、及其对应的第一序列号和第二序列号，子图像是对目标数据进行分割后处理得到的多个子图像中任一，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序；然后根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据，然后同步播放左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。由此可见，本公开实施例能够通过多个发送线程传输左眼图像数据和右眼图像数据，以此来有效缩短图像处理设备向图像显示设备传输图像数据的时间，提高图像传输效率，在此过程中，本公开实施例能够在得到左眼图像数据和右眼图像数据后，同步播放对应的左眼图像和右眼图像，保证左眼图像和右眼图像显示的同步性。
<设备实施例>
图8是根据本公开一些实施提供的一种图像传输装置的功能结构框图。该图像传输装置应用于图像处理设备。如图8所示，该图像传输装置80包括：
分割模块81，用于对目标数据进行分割处理，得到多个子图像；其中，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据。
确定模块82，用于确定每个子图像对应的第一序列号和第二序列号；其中，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序。
生成模块83，用于根据确定结果，生成与多个子图像一一对应的多组图像数据；每组图像数据包括：子图像、及其对应的第一序列号和第二序列号。
发送模块84，用于通过多个发送线程发送多组图像数据至图像显示设备，以便图像显示设备根据多组图像数据中的第一序列号和第二序列号，对多组图像数据中的子图像进行组合处理，得到左眼图像数据和右眼图像数据。
图像传输装置80的各模块的功能的实现过程可以参见上述方法实施例中的对应描述，此处不赘述。
图像处理设备例如可以是PC。
图9是根据本公开一些实施提供的一种图像传输装置的功能结构框图。该图像传输装置应用于图像显示设备。如图9所示，该图像传输装置90包括：
接收模块91，用于接收图像处理设备通过多个发送线程发送的多组图像数据；其中，每组图像数据包括子图像、及其对应的第一序列号和第二序列号，子图像是对目标数据进行分割后处理得到的多个子图像中任一，目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，第一序列号反映目标图像帧在多个图像帧中的播放顺序，第二序列号反映每个子图像在多个子图像中的排列顺序。
处理模块92，用于根据多组图像数据中的第一序列号和第二序列号，将多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到左眼图像数据和右眼图像数据。
播放模块93，用于同步播放左眼图像数据对应的左眼图像和右眼图像数据对应的右眼图像。
图像传输装置90的各模块的功能的实现过程可以参见上述方法实施例中的对应描述，此处不赘述。
图像显示设备例如可以HMD等VR设备。
图10是根据本公开一些实施例的图像处理设备的硬件结构示意图。
如图10所示，该图像处理设备1000包括处理器1010和存储器1020，该存储器1020用于存储可执行的计算机程序，该处理器1010用于根据该计算机程序的控制，执行如以上任意方法实施例的方法。
该图像处理设备1000可以是PC。
以上图像传输装置80的各模块可以由本实施例中的处理器1010执行存储器1020存储的计算机程序实现，也可以通过其他电路结构实现，在此不做限定。
图11是根据本公开一些实施例的图像显示设备的硬件结构示意图。
如图11所示，该图像显示设备1100包括处理器1110和存储器1120，该存储器1120用于存储可执行的计算机程序，该处理器1110用于根据该计算机程序的控制，执行如以上任意方法实施例的方法。
该图像显示设备1100可以是HMD等VR设备。
以上图像传输装置90的各模块可以由本实施例中的处理器1110执行存储器1120存储的计算机程序实现，也可以通过其他电路结构实现，在此不做限定。
<系统实施例>
图12是根据本公开一些实施例的图像传输系统的示意图。如图12所示，图像传输系统1200包括图像处理设备1000和图像显示设备1100。
<计算机可读存储介质实施例>
本实施例提供了一种计算机可读存储介质，该存储介质中存储有可执行命令，该可执行命令被处理器执行时，执行本说明书任意方法实施例中描述的方法(例如上述步骤S210～S240对应的方法或者上述步骤S610～S630对应的方法)。
本说明书的一个实施例或者多个实施例可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本说明书的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、 机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本说明书实施例操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本说明书的各个方面。
这里参照根据本说明书实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本说明书的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本说明书的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人物来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本说明书的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人物来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人物能理解本文披露的各实施例。本申请的范围由所附权利要求来限定。

Claims (10)

  1. 一种图像传输方法，其特征在于，应用于图像显示设备，所述方法包括：
    接收图像处理设备通过多个发送线程发送的多组图像数据；每组图像数据包括子图像、及其对应的第一序列号和第二序列号，所述子图像是对目标数据进行分割后处理得到的多个子图像中任一，所述目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据，所述第一序列号反映所述目标图像帧在所述多个图像帧中的播放顺序，所述第二序列号反映每个子图像在所述多个子图像中的排列顺序；
    根据所述多组图像数据中的第一序列号和第二序列号，将所述多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到所述左眼图像数据和所述右眼图像数据；
    同步播放所述左眼图像数据对应的左眼图像和所述右眼图像数据对应的右眼图像。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述多组图像数据中的第一序列号和第二序列号，将所述多组图像数据中对应于同一图像帧的多个子图像进行组合处理，得到所述左眼图像数据和所述右眼图像数据，包括：
    根据所述多组图像数据中的第一序列号，确定对应于同一图像帧的多个子图像；
    根据对应于同一图像帧的多个子图像的第二序列号，将该对应于同一图像帧的多个子图像依次存入预设缓存区，对应得到所述左眼图像数据和所述右眼图像数据。
  3. 根据权利要求1或2所述的方法，其特征在于，所述接收图像处理设备通过多个发送线程发送的多组图像数据，包括：
    接收图像处理设备通过多个发送线程发送的与所述多组图像数据一一对应的多个目标RTP数据包；其中，所述目标RTP数据包中封装有一组图像数据。
  4. 根据权利要求3所述的方法，其特征在于，所述图像数据还包括：预设的图像数据标识；
    所述接收图像处理设备通过多个发送线程发送的与所述多组图像数据一一对应的多个目标RTP数据包，包括：
    接收RTP数据包；
    在所述RTP数据包包括所述预设的图像数据标识的情况下，确定所述RTP数据包为所述目标RTP数据包。
  5. 一种图像传输方法，其特征在于，应用于图像处理设备，所述方法包括：
    对目标数据进行分割处理，得到多个子图像；其中，所述目标数据包括目标视频的多个图像帧中任一目标图像帧对应的左眼图像数据和右眼图像数据；
    确定每个子图像对应的第一序列号和第二序列号；其中，所述第一序列号反映所述目标图像帧在所述多个图像帧中的播放顺序，所述第二序列号反映每个子图像在所述多个子图像中的排列顺序；
    根据确定结果，生成与所述多个子图像一一对应的多组图像数据；每组图像数据包括：子图像、及其对应的第一序列号和第二序列号；
    通过多个发送线程发送所述多组图像数据至图像显示设备，以便所述图像显示设备根据所述多组图像数据中的第一序列号和第二序列号，对所述多组图像数据中的子图像进行组合处理，得到所述左眼图像数据和所述右眼图像数据。
  6. 根据权利要求5所述的方法，其特征在于，所述根据确定结果，生成与所述多个子图像一一对应的多组图像数据，包括：
    将每组图像数据中的子图像、该子图像对应的第一序列号和第二序列号封装在一个目标RTP数据包中，得到多个目标RTP数据包；
    所述通过多个发送线程发送所述多组图像数据至图像显示设备，包括：
    根据所述多个发送线程的数量和所述多个目标RTP数据包的数量，确定每个发送线程对应的目标RTP数据包的数量；
    根据每个发送线程对应的目标RTP数据包的数量，将所述多个目标RTP数据包分配至所述多个发送线程进行发送。
  7. 根据权利要求6所述的方法，其特征在于，所述图像数据还包括：预设的图像数据标识，以便图像显示设备接收到图像处理设备通过多个发送线程发送的多个RTP数据包的情况下，将包括所述图像数据标识的RTP数据包确定为所述目标RTP数据包。
  8. 一种图像显示设备，其特征在于，包括存储器和处理器，所述存储器用于存储计算机程序；所述处理器用于执行所述计算机程序，以实现根据权利要求1-5中任意一项所述的方法。
  9. 一种图像处理设备，其特征在于，包括存储器和处理器，所述存储器用于存储计算机程序；所述处理器用于执行所述计算机程序，以实现根据权利要求6或7所述的方法。
  10. 一种图像传输系统，其特征在于，包括如权利要求8所述的图像显示设备和如权利要求9所述的图像处理设备。
PCT/CN2022/099504 2021-06-18 2022-06-17 图像传输方法、图像显示及处理设备、及图像传输系统 WO2022262858A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/882,381 US11758108B2 (en) 2021-06-18 2022-08-05 Image transmission method, image display device, image processing device, image transmission system, and image transmission system with high-transmission efficiency

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110681037.5 2021-06-18
CN202110681037.5A CN113473106A (zh) 2021-06-18 2021-06-18 图像传输方法、图像显示及处理设备、及图像传输系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/882,381 Continuation US11758108B2 (en) 2021-06-18 2022-08-05 Image transmission method, image display device, image processing device, image transmission system, and image transmission system with high-transmission efficiency

Publications (1)

Publication Number Publication Date
WO2022262858A1 true WO2022262858A1 (zh) 2022-12-22

Family

ID=77868773

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/099504 WO2022262858A1 (zh) 2021-06-18 2022-06-17 图像传输方法、图像显示及处理设备、及图像传输系统

Country Status (2)

Country Link
CN (1) CN113473106A (zh)
WO (1) WO2022262858A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113473106A (zh) * 2021-06-18 2021-10-01 青岛小鸟看看科技有限公司 图像传输方法、图像显示及处理设备、及图像传输系统
US11758108B2 (en) 2021-06-18 2023-09-12 Qingdao Pico Technology Co., Ltd. Image transmission method, image display device, image processing device, image transmission system, and image transmission system with high-transmission efficiency
CN115103175B (zh) * 2022-07-11 2024-03-01 北京字跳网络技术有限公司 图像传输方法、装置、设备及介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101600099A (zh) * 2009-04-09 2009-12-09 上海交通大学 多视角视频码流的实时传输同步控制方法
CN101668192A (zh) * 2008-09-04 2010-03-10 新奥特（北京）视频技术有限公司 一种基于c/s模式实时传输大数据量图像的方法
WO2013012884A1 (en) * 2011-07-20 2013-01-24 X6D Limited Controlling display device using 3d glasses
US20160337639A1 (en) * 2015-05-12 2016-11-17 Delta Electronics, Inc. Projection apparatus
CN106991665A (zh) * 2017-03-24 2017-07-28 中国人民解放军国防科学技术大学 基于cuda图像融合并行计算的方法
US20170302978A1 (en) * 2015-01-14 2017-10-19 Tencent Technology (Shenzhen) Company Limited Video coding method, system and server
CN110192391A (zh) * 2017-01-19 2019-08-30 华为技术有限公司 一种处理的方法及设备
CN113473106A (zh) * 2021-06-18 2021-10-01 青岛小鸟看看科技有限公司 图像传输方法、图像显示及处理设备、及图像传输系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100486140C (zh) * 2004-02-26 2009-05-06 上海大学 基于因特网的立体视频监视方法及系统
US20050289626A1 (en) * 2004-06-23 2005-12-29 Aboulgasem Abulgasem H IP based interactive multimedia communication system
CN103220543B (zh) * 2013-04-25 2015-03-04 同济大学 基于kinect的实时3d视频通信系统及其实现方法
CN105025347B (zh) * 2015-07-21 2018-10-09 暴风集团股份有限公司 一种gop图像组的发送和接收方法
US10873754B2 (en) * 2017-12-12 2020-12-22 Coherent Logix, Incorporated Low latency video codec and transmission with parallel processing
CN108616722B (zh) * 2018-04-18 2021-02-26 中南大学 一种嵌入式高清视频采集与数据流传输系统
CN111698386B (zh) * 2020-05-26 2021-07-09 中国科学院上海微系统与信息技术研究所 一种多路图像数据同步发送装置、接收装置和传输系统

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101668192A (zh) * 2008-09-04 2010-03-10 新奥特（北京）视频技术有限公司 一种基于c/s模式实时传输大数据量图像的方法
CN101600099A (zh) * 2009-04-09 2009-12-09 上海交通大学 多视角视频码流的实时传输同步控制方法
WO2013012884A1 (en) * 2011-07-20 2013-01-24 X6D Limited Controlling display device using 3d glasses
US20170302978A1 (en) * 2015-01-14 2017-10-19 Tencent Technology (Shenzhen) Company Limited Video coding method, system and server
US20160337639A1 (en) * 2015-05-12 2016-11-17 Delta Electronics, Inc. Projection apparatus
CN110192391A (zh) * 2017-01-19 2019-08-30 华为技术有限公司 一种处理的方法及设备
CN106991665A (zh) * 2017-03-24 2017-07-28 中国人民解放军国防科学技术大学 基于cuda图像融合并行计算的方法
CN113473106A (zh) * 2021-06-18 2021-10-01 青岛小鸟看看科技有限公司 图像传输方法、图像显示及处理设备、及图像传输系统

Also Published As

Publication number Publication date
CN113473106A (zh) 2021-10-01

Similar Documents

Publication Publication Date Title
WO2022262858A1 (zh) 图像传输方法、图像显示及处理设备、及图像传输系统
US11405699B2 (en) Using GLTF2 extensions to support video and audio data
TWI846795B (zh) 用於經串流媒體資料之多個解碼器介面
TW201924323A (zh) 用於浸入式媒體資料之內容來源描述
JP2017153127A (ja) メディアコンテンツ復号装置
JP6254291B2 (ja) Ｄａｓｈのロバストなライブ動作
JP2018521538A (ja) ウェブソケットサブプロトコルを使用したメディアデータの転送
US12035020B2 (en) Split rendering of extended reality data over 5G networks
CN104243920A (zh) 一种基于基本流视频数据封装的图像拼接方法及装置
KR20240007142A (ko) 5g 네트워크들을 통한 확장 현실 데이터의 분할 렌더링
CN114930862A (zh) 用于流式媒体数据的多解码器接口
CN112771876B (zh) 检索媒体数据的方法和设备以及发送媒体数据的方法和设备
US20220335694A1 (en) Anchoring a scene description to a user environment for streaming immersive media content
WO2022116822A1 (zh) 沉浸式媒体的数据处理方法、装置和计算机可读存储介质
WO2023062271A1 (en) A method, an apparatus and a computer program product for video coding
US11758108B2 (en) Image transmission method, image display device, image processing device, image transmission system, and image transmission system with high-transmission efficiency
JP2024509524A (ja) ストリーミングされるメディアデータによって定義される仮想インタラクティブシーンにおける仮想カメラのためのオブジェクト衝突データ
TW202249493A (zh) 將場景描述錨定到使用者環境以用於串流沉浸式媒體內容
JP2024509416A (ja) ストリーミングされるメディアデータによって定義される仮想インタラクティブシーンにおける仮想カメラのためのカメラ制御データ
CN114009030B (zh) 视频解码的方法、装置及可读介质
US11265357B2 (en) AV1 codec for real-time video communication
CN113453006B (zh) 一种图片封装方法、设备以及存储介质
US20240098130A1 (en) Mixed media data format and transport protocol
WO2024164714A1 (zh) 一种音频编解码方法、音频编解码装置、计算机设备及存储介质
US20240273833A1 (en) Signaling pose information to a split rendering server for augmented reality communication sessions

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22824330

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.04.2024)