TWI786694B - Data streaming method and data streaming system - Google Patents

Abstract

A data streaming method and a data streaming system are provided. The data streaming system includes at least one transmission device and a receiving device. In the method, each transmission device packs streaming data into multiple packets according to a predefined format in response to a transmission request for the streaming data from the receiving device, and transmits the packets to the receiving device. The receiving device stores data of the received packets in a storage device and records a time point when receiving each packet. When running the streaming data, the receiving device reads the data of each packet stored in the storage device, and performs interpolation calculation on the data of each packet according to the time point of each packet, so as to obtain corresponding data required by the receiving device to run the streaming data at a current time point. The data streaming method and the data streaming system can simultaneously take into account the real-time and interactive nature of virtual and real interaction.

Description

Data streaming method and system

The present invention relates to a data processing method and system, and in particular to a data streaming method and system.

In virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR) or mixed reality (Mixed Reality, MR), virtual-real interaction is an important application in the human-machine interface (Human-machine interface) One of the situations. The main purpose of virtual-real interaction is to interact with hardware devices such as the Head Mounted Display (HMD) through the user, and to confuse the user's senses or give the user illusion during the interaction process, thereby integrating abstract information, virtual Information that does not exist in the real world, such as images and virtual objects, is presented in front of the user's eyes.

Among the current mainstream algorithm modules or systems, virtual-real interactive systems mostly include "inertial navigation system (Inertial Navigation System, INS)", "visual inertial odometer (Visual Inertial Odometry, VIO)", "real-time positioning and map construction (Simultaneous localization and mapping, SLAM)", "hand gesture tracking module" and "virtual-real interaction module" and other modules or systems with five main functions. Find out the corresponding real scenes in the virtual space with the map building module, and trigger field events through gesture actions in the virtual space, so as to present virtual messages, virtual images or virtual objects that match the field in the in front of the user's eyes.

However, due to the different computing requirements of each module or system, the delays generated during the computing process are not consistent, and there are dependencies between modules or systems or the order of processing procedures. As a result, each module or system When the results of calculations are transmitted, there will be a problem of out-of-synchronization of information between modules or systems. Due to the delay of the operation of the modules or systems, from the input of sensors (such as cameras, inertial measurement units, etc.) The accumulation of delays results in a significant delay in the entire virtual-real interaction system process, thereby reducing the user experience.

In detail, since the processing speed and delay time between each input source and each module or system are not equal, it will be difficult for the virtual-real interaction module in the last step to synchronize information. FIG. 1 is a schematic diagram of a conventional virtual-real interaction system. Taking the virtual-real interactive system 10 in FIG. 1 as an example, the camera (or video camera) 11 processes a photo for about 1/30 second or 1/60 second, and the value transmitted by the inertial measurement unit 12 is about 1/200 second. With the same hardware and the same computing power, the processing speed of gesture tracking module 13 is about 1/15 second, and the processing speed of visual inertial odometry (VIO) 14 and real-time localization and mapping (SLAM) 16 is about 1/25 second to 1/20 second, and the processing speed of the inertial navigation system (INS) 15 is about 1/100 second. Therefore, in terms of the synchronization time of each module or system, the virtual-real interaction module 17 will be synchronized based on the slowest processing speed of the gesture tracking module 13, which is 1/15 second. However, because the processing speed of the slowest module is too slow and the time points of data transmission of each module or system are not synchronized, the result will cause problems such as lack of immediacy and low interactivity in virtual-real interaction.

The present invention provides a data streaming method and system, which can simultaneously take into account the immediacy and interactivity of virtual-real interaction.

The invention provides a data streaming method suitable for a data streaming system including at least one transmitting device and a receiving device. The method includes the following steps: each transmitting device responds to the receiving device's transmission request for the streaming data, encapsulates the streaming data into multiple packets according to a predefined format, and transmits the packets to the receiving device; the receiving device stores the received packets The data is stored in the storage device and records the time point of receiving each packet; and when the receiving device is running streaming data, it reads the data of each packet stored in the storage device, and interpolates the data of each packet according to the time point of each packet (interpolation) operation to obtain the corresponding data required by the receiving device when running the streaming data at the current point in time.

The invention provides a data streaming system, which includes at least one transmitting device and receiving device. In response to the transmission request of the streaming data, each transmission device encapsulates the streaming data into multiple packets according to a predefined format, and sends these packets. The receiving device is connected to each transmitting device to send a transmission request to each transmitting device, and receives packets sent by each transmitting device in response to the transmission request, and stores the received packets in the storage device and records the time point when each packet is received. Wherein, when the receiving device is running the streaming data, it reads the data of each packet stored in the storage device, and performs interpolation calculation on the data of each packet according to the time point of each packet, and obtains the running data of the receiving device at the current time point. corresponding information required.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

The embodiment of the present invention proposes to package the data processed by each input source and each module into a form of "streams" to communicate information between modules. The embodiment of the present invention defines "streaming" as: a stable, continuous and time-dependent way through the Application Programming Interface (Application Programming Interface, API), Universal Serial Bus (Universal Serial Bus, USB) or network Send or receive data in a predefined format, and allow the remaining data to be sent at the same time while still receiving.

FIG. 2 is a schematic diagram of a data streaming system according to an embodiment of the present invention. Please refer to FIG. 2 , the data streaming system 20 of the embodiment of the present invention includes a first module 21 and a second module 22 . The direction indicated by the unidirectional arrow in FIG. 2 is that the source end (ie, the first module 21 ) sends information to the receiving end (ie, the second module 22 ). Among them, the present invention adopts the form of streaming to carry out the communication between the first module 21 and the second module 22, and its purpose is to use the asynchronous method for the time-dependent streaming data to achieve the effect of instant response .

Specifically, in the data streaming system 20 , the second module 22 has reserved a section of buffer (buffer) space in the storage device 23 to store the packets streamed in by the first module 21 . Therefore, when the second module 22 is running, it can run asynchronously to obtain unknown data corresponding points through interpolation operations, thereby improving the performance of the virtual-real interactive system.

FIG. 3 is a schematic diagram of a one-to-one data stream interpolation operation according to an embodiment of the present invention. Please refer to FIG. 2 and FIG. 3 at the same time, assuming that the data processing results of the first module 21 are a0-a8 of stream one, and the data processing results of the second module 22 are b0-b5 of stream two. Wherein, when the second module 22 processes the data at the b1 time point, it can take out the data at the a0 and a1 time points of the stream 1 from the buffer space reserved by the data stream system 20, and use linear interpolation (linear interpolation), polynomial interpolation, spline interpolation or exponential moving average interpolation and other interpolation algorithms to calculate the data of the approximate unknown corresponding point a1', so the second module 22 The result of the current operation (time point b1 ) can be obtained immediately without waiting until the time point a2 of the first module 21 .

FIG. 4 is a schematic diagram of a data streaming system according to an embodiment of the present invention. Please refer to FIG. 4 , the data streaming system 40 of the embodiment of the present invention includes one or more transmitting devices 42 and receiving devices 44 . In some embodiments, the transmitting device 42 and the receiving device 44 are, for example, computing devices such as file servers, database servers, application program servers, workstations or personal computers, or smart phones, tablet computers, etc. Mobile devices, but not limited to.

In some embodiments, the data streaming system 40 is, for example, a virtual reality system, an augmented reality system or a mixed reality system, and the transmitting device 42 and the receiving device 44 are, for example, an "inertial navigation system" (Inertial Navigation System, INS), "Visual Inertial Odometry (VIO)", "Simultaneous localization and mapping (SLAM)", "Gesture Tracking Module" or "Virtual Reality Interactive Module", nor limited to this.

In some embodiments, the transmitting device 42 and the receiving device 44 include any corresponding wired or wireless interface devices, which can be used to transfer data between the transmitting device 42 and the receiving device 44 . For the wired method, the interface device can be a universal serial bus (universal serial bus, USB), EIA-RS-232 (Electronic Industry Association - Recommended Standard - 232, RS232), a universal asynchronous receiver/transmitter (universal asynchronous receiver/transmitter, UART), internal integrated circuit (Inter-Integrated Circuit, I ² C), serial peripheral interface (serial peripheral interface, SPI), display port (display port) or thunderbolt port (thunderbolt) and other interfaces, but not limited to this. For wireless methods, the interface device can support wireless fidelity (wireless fidelity, Wi-Fi), radio frequency identification (Radio Frequency Identification, RFID), Bluetooth, infrared, near-field communication (near-field communication, NFC) or device-to-device (device-to-device, D2D) communication protocols, but not limited thereto. In some embodiments, the interface device may also be a network card supporting Ethernet or wireless network standards such as 802.11g, 802.11n, 802.11ac, etc., and is not limited thereto.

In some embodiments, the transmitting device 42 and the receiving device 44 each include a time counter 422 and a time counter 442 . Wherein, the transmission device 42 uses the time counter 422 to record the corresponding time point of each packet data in the streaming data. The receiving device 44 uses the time counter 442 to record the time when it receives each packet.

In some embodiments, the transmitting device 42 and the receiving device 44 each include a storage device 424 and a storage device 444 for storing computer programs and data, such as any type of fixed or removable random access memory (Random Access Memory, RAM), Read-Only Memory (Read-Only Memory, ROM), Flash memory (Flash memory), hard disk or similar components or a combination of the above components.

In some embodiments, the transmitting device 42 and the receiving device 44 each include a processor (not shown in the figure) for loading a computer program from the storage device 424 and the storage device 444 to execute the data streaming method of the embodiment of the present invention , which is, for example, a central processing unit (Central Processing Unit, CPU), other programmable general-purpose or special-purpose microprocessors (Microprocessor), microcontrollers (Microcontroller), digital signal processors (Digital Signal Processor, DSP) ), Programmable Controller, Application Specific Integrated Circuits (Application Specific Integrated Circuits, ASIC), Programmable Logic Device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, but this embodiment is not limited to this.

FIG. 5 is a flowchart of a data streaming method according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 5 at the same time. The method of this embodiment is applicable to the data streaming system 40 in FIG. 4 . The detailed steps of the data streaming method of this embodiment will be described below in conjunction with various devices of the data streaming system 40 .

In step S502 , in response to the transmission request of the receiving device 44 for the streaming data, each transmitting device 42 encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the packets to the receiving device 44 . The predefined format includes the target transmission device of the packet, the size of the data, and the time stamp of the corresponding time point in the streaming data of the packet data. Wherein, each transmitting device 42 sets the transmission path of the packet according to the transmission request, for example, and the receiving device 44 sets the receiving path of the received packet. The transmission path and the reception path may, for example, transmit and receive packet data through the wired and wireless methods described in paragraph [0017] of the above specification.

In step S504, the receiving device 44 stores the data of the received packets in the storage device and records the time point when each packet is received.

In step S506, when the receiving device 44 is running the streaming data, it will read the data of each packet stored in the storage device 444, and perform an interpolation operation on the data of each packet according to the time point of each packet to obtain the received The corresponding data required by the device 44 when running the streaming data at the current point in time.

In some embodiments, the receiving device 44 receives packets from only a single transmitting device 42 , while in other embodiments, the receiving device 44 receives packets from multiple transmitting devices 42 . The receiving device 44 will sort the packets and perform an interpolation operation based on the time point when it receives these packets, so as to obtain the corresponding data required for running the streaming data. The following examples are given in detail.

FIG. 6 is a flowchart of a data streaming method according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 6 at the same time. In this embodiment, the receiving device 44 only receives packets from a single transmitting device 42 . In response to the transmission request of the receiving device 44 for the streaming data, the transmitting device 42 encapsulates the streaming data into N packets according to a predefined format.

In step S602, the receiving device 44 estimates M pieces of corresponding data required for running the streaming data according to the average processing frequency of the streaming data.

In step S604, the receiving device 44 reads the data of the received N packets from the storage device 444, and sorts the data of the N packets according to the time points recorded by the timestamps, wherein M and N are positive integer.

In step S606, the receiving device 44 performs an interpolation operation according to the data of the sorted N packets, and obtains M pieces of corresponding data required for running the streaming data.

FIG. 7 is a flowchart of a data streaming method according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 7 at the same time. In this embodiment, after each transmitting device 42 encapsulates the streaming data into individual packets, for example, it will record whether it is abnormal on each packet, as a subsequent receiving device 44. The basis for interpolation calculations using packaged data. In this embodiment, the number of the transmission device 42 may be one or more.

In step S702 , in response to the transmission request of the receiving device 44 for the streaming data, each transmitting device 42 encapsulates the streaming data into a plurality of packets according to a predefined format.

In step S704, each transmitting device 42 determines whether the standard deviation of the processing frequency of the processed packets is greater than a threshold or whether the data of the processed packets is NULL.

In step S706 , each transmission device 42 marks the flag of the packet as abnormal when the standard deviation of the processing frequency of the packet is greater than a threshold or when the data of the processed packet is empty. On the contrary, in step S708, when the data of the processed packet has content and the standard deviation of the processing frequency is less than or equal to the threshold, each transmission device 42 marks the flag of the packet as normal. The flag is, for example, T for normal and F for abnormal. In detail, if the transmission device 42 processes data abnormally, the length of time it takes to process the data each time varies (the standard deviation of the processing frequency becomes larger), and the transmission device 42 can send the abnormal packet flag is set to F. Or, when the transmission device 42 has an output abnormality, for example, when some data are empty, it will set the flag of the empty packet to F.

In step S710 , each transmitting device 42 transmits the processed packet to the receiving device 44 .

Correspondingly, in step S712, the receiving device 44 stores the data of the received packets in the storage device 444 and records the time point when each packet is received.

In step S714 , when the receiving device 44 is running the streaming data, for example, it reads the flags of each packet, and does not use the packets whose flags are recorded as abnormal when performing interpolation calculations using the data of the packets.

FIG. 8 is a flowchart of a data streaming method according to an embodiment of the present invention. Please refer to FIG. 4 and FIG. 8 at the same time. In this embodiment, the receiving device 44 receives packets from a plurality of transmitting devices 42 . In response to the transmission request of the receiving device 44 for the streaming data, the multiple transmitting devices 42 encapsulate the streaming data into a plurality of packets according to a predefined format.

In step S802, the receiving device 44 estimates the M pieces of corresponding data needed to run the streaming data according to the average processing frequency of the streaming data, and divides the time sequence for processing the M pieces of corresponding data into multiple sub-time sequences, wherein M is a positive integer.

In step S804, the receiving device 44 reads the data of the multiple packets received from the multiple transmitting devices 42 from the storage device 444, and sorts the data of these packets according to the time points recorded by the time stamps.

In step S806, the receiving device 44 uses the sorted data of multiple packets to perform interpolation operation according to each sub-sequence, and obtain M pieces of corresponding data required to run the streaming data at each sub-sequence.

Specifically, in virtual reality (VR), augmented reality (AR) or mixed reality (MR) applications, many-to-one streaming scenarios (that is, multiple modules stream to the same A target module, for example, multiple modules or systems in FIG. 1 flow to the virtual-real interaction module 17). At this time, the receiving device 44 needs to use the packets received from a plurality of different transmitting devices 42 to calculate the corresponding data required for running the streaming data.

For example, FIG. 9 is a schematic diagram of an interpolation operation of many-to-one data streams according to an embodiment of the present invention. Please refer to FIG. 9. In this embodiment, when a many-to-one data stream occurs, the receiving device 90 first arranges each packet stored in the storage device 92 according to its timestamp (as shown in FIG. 9 stream one, stream two, ..., stream N). Wherein, the receiving device 90 will, for example, calculate the time point T4 according to the average processing frequency between the previous time points T1-T3. Since the receiving device 90 needs M data points from the synchronized stream 1, stream 2, ..., to stream N, the timing between T3 and T4 will be cut into M equal sub-sequences in the stream timing , the sub-sequences are, for example, T3 ₁ , T3 ₂ and T3 ₃ . For example, interpolate data at time points a6, a7, and a8 of stream 1 into data at time points a6', a6'', a7', and a8', and interpolate data at time points b3, b4, and b5 of stream 2 Interpolate data at time points b3', b4', b4'', b5', and interpolate data at time points c5, c6, c7 of stream N into c5', c6', c6'', c7' point in time data. The interpolation time points of the above-mentioned stream 1, stream 2-stream N correspond to the above-mentioned sub-sequences and timings, for example, the time points of a6', b3' and c5' correspond to the sub-sequence T3 ₁ , a6'', b4' and c6' time points correspond to sub-sequence T3 ₂ , a7', b4'' and c6'' time points correspond to sub-sequence T3 ₃ , a8', b5' and c7' time points correspond to time sequence T4 . By means of the above method, the data corresponding to the sub-sequence and time sequence can be obtained, and the effect of synchronizing multiple stream data in real time can be achieved.

To sum up, the data streaming method and system of the embodiments of the present invention define the transmission path and reception path between modules in advance, and effectively use interpolation to obtain unknown data points, so that multiple devices can Or the modules can run independently without data synchronization. By applying the data streaming system of the embodiment of the present invention to the virtual-real interaction system, the delay of the virtual-real interaction can be improved without affecting the accuracy of the virtual-real interaction system, thereby improving user experience.

Although the present invention has been disclosed above with the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be defined by the scope of the appended patent application.

10: virtual reality interactive system 11: Camera 12: Inertial Measurement Unit 13: Gesture tracking module 14:Visual-inertial odometer 15: Inertial Navigation System 16: Instant positioning and map construction 17:Virtual-real interactive module 20, 40: data streaming system 21: The first module 22: Second module 23: storage device 42: Teleporter 422, 442: time counter 424, 444, 92: storage device 44, 90: receiving device a0~a8, b0~b5, c0~c7, a1', a3', a4', a6', a6'', a7', a8', b3', b4', b4'', b5', c5', c6', c6'', c7', T1~T4: time points S502~S506, S602~S606, S702~S714, S802~S806: steps

FIG. 1 is a schematic diagram of a conventional virtual-real interaction system. FIG. 2 is a schematic diagram of a data streaming system according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a one-to-one data stream interpolation operation according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a data streaming system according to an embodiment of the present invention. FIG. 5 is a flowchart of a data streaming method according to an embodiment of the present invention. FIG. 6 is a flowchart of a data streaming method according to an embodiment of the present invention. FIG. 7 is a flowchart of a data streaming method according to an embodiment of the present invention. FIG. 8 is a flowchart of a data streaming method according to an embodiment of the present invention. FIG. 9 is a schematic diagram of a many-to-one data stream interpolation operation according to an embodiment of the present invention.

S502~S506: steps

Claims (20)

A data streaming method, suitable for a data streaming system including at least one transmitting device and a receiving device, the method includes the following steps: Each of the at least one transmitting device responds to a transmission request of the receiving device for a streaming data, encapsulates the streaming data into a plurality of packets according to a predefined format, and transmits the packets to the receiving device; The receiving device stores the data of the received packets in a storage device and records the time point when each of the packets is received; and When the receiving device is running the streaming data, it reads the data of each of the packets stored in the storage device, and performs an interpolation on the data of each of the packets according to the time point of each of the packets ( interpolation) operation to obtain the corresponding data required for the receiving device to run the streaming data at the current point in time. The data streaming method as described in claim item 1, wherein the predefined format includes the target transmission device of each of the packets, the size of the data, and the time of the corresponding time point in the streaming data of the data of each of the packets stamp. The data streaming method as described in Claim 2, wherein the step of running the streaming data by the receiving device further includes: According to the average processing frequency of processing the streaming data, estimate the M pieces of corresponding data required to run the streaming data; Reading the data of the received N packets from the storage device, and sorting the data of the N packets according to the time point recorded by the timestamp, wherein M and N are positive integers; and The interpolation operation is performed according to the sorted data of the N packets to obtain the M pieces of corresponding data required for running the streaming data. The data streaming method as described in Claim 2, wherein the step of running the streaming data by the receiving device further includes: According to the average processing frequency of processing the streaming data, estimate the M pieces of corresponding data required to run the streaming data, and divide and process the time series of the M corresponding data into a plurality of sub-time series, wherein M is a positive integer; reading from the storage device the data of the packets received from each of the at least one transmission device, and sorting the data of the packets according to the time point recorded by the timestamp; and Using the sorted data of the packets, perform the interpolation operation according to each of the sub-sequences, and obtain the M pieces of corresponding data required to run the streaming data at each of the sub-sequences. The data streaming method as described in claim 1, wherein the step of encapsulating the streaming data into each of the packets by each of the at least one transmission device further includes marking whether an abnormality is marked on each of the packets. The data streaming method as described in claim 5, wherein when the standard deviation of the processing frequency of the transmission device is greater than a threshold or when the data of the packet is NULL, the flag of the packet is abnormal. The data streaming method as described in claim item 5, wherein the step of running the streaming data by the receiving device further includes: Reading the flag of each of the packets, and not using the flag to record the packet as abnormal when performing the interpolation operation using the data of each of the packets. The data streaming method according to claim 1, wherein the interpolation operation includes linear interpolation, polynomial interpolation, spline interpolation or exponential moving average interpolation. The data streaming method according to claim 1, wherein the data streaming system includes a virtual reality system, an augmented reality system or a mixed reality system. A data streaming system, comprising: At least one transmission device, wherein each of the at least one transmission device responds to a transmission request of a stream data, encapsulates the stream data into a plurality of packets according to a predefined format, and sends the packets; The receiving device is connected to each of the at least one transmission device, sends the transmission request to each of the at least one transmission device, and receives the packets sent by each of the at least one transmission device in response to the transmission request, and stores the received These packets are stored in a storage device and record the time point when each of these packets is received, wherein When the receiving device is running the streaming data, it reads the data of each of the packets stored in the storage device, and performs an interpolation operation on the data of each of the packets according to the time point of each of the packets to obtain the corresponding data required by the receiving device when running the streaming data at the current point in time. The data streaming system as claimed in claim 10, wherein the receiving device includes a time counter for recording the time point when each of the packets is received. The data streaming system as described in claim 10, wherein the predefined format includes the target transmission device of each of the packets, the size of the data, and the time of the corresponding time point in the streaming data of the data of each of the packets stamp. The data streaming system as claimed in claim 12, wherein the transmission device includes another time counter for recording the corresponding time point of each of the packets of data in the streaming data. The data streaming system as described in claim 12, wherein the receiving device further includes, according to the average processing frequency of processing the streaming data, estimating the M pieces of corresponding data required to run the streaming data, and reading from the storage device Take the data of the N packets that have been received, and sort the data of the N packets according to the time point recorded by the timestamp, where M and N are positive integers, and according to the sorted N packets The interpolation operation is performed on the data to obtain the M pieces of corresponding data required to run the streaming data. The data streaming system as described in claim 12, wherein the operation of the receiving device is based on the average processing frequency of processing the streaming data, estimates the M pieces of corresponding data required to run the streaming data, and divides and processes the M The timing of the corresponding data is a plurality of sub-timings, wherein M is a positive integer, read the data of the packets received from each of the at least one transmission device respectively from the storage device, and store the data of the packets according to The time points recorded by the timestamp are sorted, and the data of the sorted packets are used to perform the interpolation operation according to each of the sub-time sequences, so as to obtain the data required for running the stream data at each of the sub-time sequences. The corresponding data of the M pens. In the data streaming system as claimed in claim 10, each of the at least one transmitting device marks whether an abnormality is marked on each of the packets. The data stream system as described in claim 16, wherein when the standard deviation of the processing frequency of each of the at least one transmission device is greater than a threshold value or when the data of the packet is empty, the transmission device sends the flag of the packet marked as abnormal. The data streaming system as described in claim 16, wherein the receiving device reads the flag of each of the packets when running the streaming data, and uses the data of each of the packets to perform the interpolation operation , do not use this flag to record the packet as abnormal. The data streaming system according to claim 10, wherein the interpolation operation includes linear interpolation, polynomial interpolation, spline interpolation or exponential moving average interpolation. The data streaming system as claimed in claim 10, wherein the data streaming system includes a virtual reality system, an augmented reality system or a mixed reality system.

Images