CN106100803A - The method and apparatus determined is retransmitted for making - Google Patents

The method and apparatus determined is retransmitted for making Download PDF

Info

Publication number
CN106100803A
CN106100803A CN201610566114.1A CN201610566114A CN106100803A CN 106100803 A CN106100803 A CN 106100803A CN 201610566114 A CN201610566114 A CN 201610566114A CN 106100803 A CN106100803 A CN 106100803A
Authority
CN
China
Prior art keywords
network
data
video data
packet
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610566114.1A
Other languages
Chinese (zh)
Inventor
吕秀平
I.U.曼德里卡
R.佩鲁马纳姆
刘航
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Thomson Licensing SAS
Original Assignee
Thomson Licensing SAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing SAS filed Critical Thomson Licensing SAS
Priority to CN201610566114.1A priority Critical patent/CN106100803A/en
Priority claimed from CN2010800658622A external-priority patent/CN103109485A/en
Publication of CN106100803A publication Critical patent/CN106100803A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/12Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/1607Details of the supervisory signal
    • H04L1/1671Details of the supervisory signal the supervisory signal being transmitted together with control information
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/12Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1829Arrangements specially adapted for the receiver end
    • H04L1/1835Buffer management
    • H04L1/1838Buffer management for semi-reliable protocols, e.g. for less sensitive applications such as streaming video
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/12Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1867Arrangements specially adapted for the transmitter end
    • H04L1/1874Buffer management
    • H04L1/1877Buffer management for semi-reliable protocols, e.g. for less sensitive applications like streaming video

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

This document describes and include analyzing (410) level of abstraction head and the method and apparatus to packet assigned priority to be sent.The networkmonitor (270,405) of transmitter is connected to network interface (250) and retransmits between resolver (275), for basis, such as, the priority specified and the network statistics of collection or situation, such as, packet loss rate determines to retransmit which packet.Networkmonitor (271) can also be equipped with on the receiver, for collecting current network statistic as the accumulated quantity of lost packets, and such statistic is supplied to transmitter (200).The method farther includes to cache data to be sent together with the priority specified, via datagram protocol (240,245) data of retrieval from buffer (235) are sent, receive the request of (405) retransmission data, determine (415) requested data the most in a buffer, and via providing end-to-end confirmation and the protocol retransmission requested data of error correcting of data.

Description

The method and apparatus determined is retransmitted for making
The application is filing date on January 28th, 2010, Application No. 201080065862.2, invention entitled " is used for Make and retransmit the method and apparatus determined " the divisional application of application for a patent for invention.
Cross
The application is relevant with U.S. Patent application the most co-pending, that own together: conduct on October 7 in (1) 2009 year Invention entitled " the AN that international patent application (application number PCT/US09/005499, The Thomson Corporation's Reference Number PU090136) is submitted to EFFICIENT APPLICATION-LAYER AUTOMATIC REPEAT REQUEST(ARQ)RETRANSMISSION SCHEME FOR RELIABLE REAL-TIME STREAMING IN WIRELESS NETWORKS " No. XXX;And (2) invention entitled " A submitted to as international patent application (application number XXX, The Thomson Corporation's Reference Number PU090157) METHOD AND APPARATUS FOR PARSING A NETWORK ABSTRACTON-LAYER FOR RELIABLE DATA COMMUNICA TION " No. XXX.
Technical field
Present application relates generally to digital data network, particularly relate to networkmonitor and data that reliable digit data send Retransmit resolver.
Background technology
In multicast or broadcasted application, generally on wiredly and/or wirelessly network, data are sent to multiple from server Receptor.As used herein multicast system is the system that identical data is sent to multiple receptor by server simultaneously, its In these receptors formed up to and include the subset of Servers-all of Servers-all.Broadcast system is that server is same Time identical data is sent to the system of all receptors.It is to say, multicast system can include broadcast system by definition.
Data are typically formatted as packet and/or frame sends.Format it is to say, packet and/or frame are data Scheme.As used herein, data can be formatted into any convenient form sent including packet and/or frame.Therefore, " it is grouped " and is used for defining any data formatting scheme known to those of ordinary skill in the art in this article.
As the example of a kind of application of this paper digital data network, the video in wireless network is used to send or distribution. Video in radio data network sends and is typically involved in by channel error situation as interference, channel fading, conflict etc. The packet loss caused.When there is such channel error situation, the radio link layer of protocol stack may attempt to when fixing Between interval in packet retransmission fixed number of times.If these repeating transmission are the most unsuccessful, radio link layer just abandons packet.Based on the Internet The video of agreement (IP) network sends and generally uses the agreement of RTP (RTP) that video packets is transported to destination (receptor;Sometimes herein called client computer), the agreement of this RTP (RTP) uses again transmitting control protocol (TCP) host-host protocol or less RUDP (UDP) host-host protocol.When such as using less reliable UDP agreement Time, this agreement does not provide detection confusion packet or recovers the means of lost packets, and giving correction packet conveying mistake responsibility Application by mistake.On the contrary, when using Transmission Control Protocol, it is provided that end-to-end confirmation, in order to this agreement manages strictly to divide by this application management The order of group sends and/or receives media (audio frequency, video, multimedia ...) packet (data).When packet error being detected Time, TCP provides the sliding window mechanism for data flow con-trol, and reduces packet transmission rate.TCP continues to retransmit dividing of loss Group is until they are restored.
It is the reception with data to occur and have and the application of user's viewing experience that reproduction is associated in real time that video sends Example.Within it the packet delay that must be carried or recover or time-constrain should not affect the viewing body of end user Test.Therefore, packet error ought to be corrected in finite time, and otherwise, data are probably not visible.TCP at present can not To provide for the control of packet recovery to ensure according to time-constrain.Therefore, as host-host protocol, TCP is used for wireless network will lead Cause bad user's viewing experience.And, TCP requires the positive acknowledgment for all transmission packets.TCP up-link is true Recognize (from data sink to data transmitter (sender)) with downlink data traffic (from transmitter (sender) to receiving Device) competition wireless bandwidth.If clashed between downlink sends and up-link sends, then these conflicts will cause Handling capacity reduces further.
PCT application US/09/005,499 that on October 7th, 2009 submits to discloses a kind of effectively application layer and automatically repeats to ask Seeking repeating method, the data that wherein will send are buffered or cache is in the module realizing unreliable medium agreement, in order to Recover the packet of loss and contribute to such as real-time streaming (such as video) market demand.Referring briefly to Fig. 1 (on being derived from PCT application US/09/005 stated, Fig. 5 of 499), real-time protocol (RTP) packetization module 120 is provided in such as real-time server 100 On, in order to receive digital video data input 105.There is provided real-time protocol (RTP) (RTP) packetization module 120 of streaming application via example As one or more similar socket application programming interface (API) 115 are connected with unreliable medium protocol module 130.Unreliable medium Protocol module 130 accept as cache or buffer sizes, wait packet recovery maximum time etc. configure data Parameter 125.These parameters are such as by digital application, such as, and phone, audio frequency, video or multimedia and other known applications Determine.Local buffer or the interim storing initial of cache memory 135 send packet, until may determine that and needing to retransmit Still them are abandoned.Server 100 is by one of UDP/IP sender interface 140 or TCP/IP transmitter/receiver interface 145 With numerical data network interface, such as, Ethernet/802.11 interface 150 sends numerical data.Along with packet in real time, such as, exist When controlling the reception of 155 etc. ACK/NACK to be retransmitted or definite time-lag in advance, it is temporarily stored in cache/buffer 135, can Can send the most in real time via to initial transmission 160 generation of network 110.
Summary of the invention
Have increase further as be disclosed in above-mentioned PCT application US/09/005,499 that based on reliably The effective ways of the reliability of media protocol real-time data transmission system and device are favourable.The present invention solve these and/ Or other problems.
According to one aspect of the present invention, disclose a kind of method.According to an one exemplary embodiment, the method comprises prison Control digital data network is with collection network transmission statistic amount;And decide whether to reception according to the network transmission statistic amount collected Think highly of and send out digital data packets.
According to another aspect of the present invention, disclose a kind of device.According to an one exemplary embodiment, as numerical data Device as transmitter comprises digital data network watch-dog, for collection network transmission statistic amount;And repeating transmission resolver, Decide whether that retransmitting digital data divides for the network transmission statistic amount according to the digital data packets priority specified and collection Group.
According to another aspect of the present invention, disclose a kind of device.According to an one exemplary embodiment, as numerical data Device as transmitter comprises as networkmonitor, the device of collection network transmission statistic amount;And determine as retransmitting Device is such, decides whether retransmitting digital data according to the network transmission statistic amount of the digital data packets priority specified and collection The device of packet.
According to another aspect of the present invention, disclose a kind of device.According to an one exemplary embodiment, as numerical data As receptor, device comprises networkmonitor, for from the visual angle collection network statistic receiving device;And network connects Mouthful, the control channel massage of described collection network statistic is included for output.
According to another aspect of the present invention, disclose a kind of device.According to an one exemplary embodiment, as numerical data Device as receptor comprises as networkmonitor, from the device of the visual angle collection network statistic receiving device;With And as network interface, output includes the device controlling channel massage of described collection network statistic.
Accompanying drawing explanation
By the embodiment of the present invention described below with reference in conjunction with the accompanying drawings, the above and other feature of the present invention and Advantage and realize their mode and will become apparent from, and make the present invention be best understood from, in the accompanying drawings:
Fig. 1 be derived from Fig. 5 of above-mentioned relevant PCT application US/09/005,499 containing unreliable medium protocol module and The schematic block diagrams of the server of the such as real-time local cache being grouped of storage or buffer when waiting to be retransmitted decision.
Fig. 2 A is the schematic block diagrams of the server of the principle according to the present invention, and it shows according to the present invention the most current Network statistical data monitors network interface and provides input to the network abstract layer (NAL) of unreliable medium agreement (RMP) module Head analyzer, networkmonitor and repeating transmission resolver;Fig. 2 B is containing of the principle according to the present invention to collect receptor network Peculiar statistical data is to be sent to the schematic block diagrams of the associated client of the networkmonitor of the server of Fig. 2 A.
Fig. 3 A is the simplification block schematic diagram of the streaming server of the principle according to the present invention, and it contains from telescopic video Coding (SVC) encoder receives NAL head analyzer, networkmonitor and the repeating transmission resolver of input.Fig. 3 A is further depicted as The streamed reception device of numerical data is received from streaming server.Fig. 3 B is to illustrate that NAL head analyzer is read from local MP4 file Read the block schematic diagram that device receives the streaming server of Fig. 3 A inputted.Fig. 3 C shows and receives from network via de-packetizer The NAL head analyzer of Fig. 3 A of input.
Fig. 4 shows that the principle according to the present invention carries out level of abstraction head and analyzes and the operation of network monitoring in sender side Flow chart, wherein sender can be the transmitter of Fig. 2 A or Fig. 3, server or streaming server.Fig. 4 further illustrates Data sink and network of relation policer operation, wherein receptor can be client computer or the streamed reception device of Fig. 3 of Fig. 2 B.
Fig. 5 shows that scalable video decoder (SVC) network abstract layer (NAL) of the principle analysis according to the present invention expands The exemplary data content representation form of exhibition head.
Fig. 6 shows that the principle according to the present invention can send priority data and can therefrom calculate network statistics The exemplary format of real-time protocol (RTP) (RTP) head.
Fig. 7 shows the continuous counter (CC) including to monitor with derived grid statistic according to some aspects of the present invention Mpeg transport stream (TS) head of field, such as, the exemplary format of MPEG-2.
Fig. 8 shows and includes, according to some aspects of the present invention, the request that sends and can monitor the end of statistic with derived grid Exemplary format to the message of end packet loss rate.
Fig. 9 shows that some aspects according to the present invention can be with ARQ request (NACK) point of derived grid monitoring statistic First exemplary format of group.
Figure 10 show some aspects according to the present invention can derive and send network monitoring statistic RMP ARQ please Seek the second exemplary format that (NACK) is grouped.
Detailed description of the invention
The present invention is directed on transmitter or sender for analyzing as the improvement unreliable medium for Fig. 2 A and Fig. 2 B Protocol module is the side of the level of abstraction head of the input of real-time (such as, real-time protocol (RTP) or RTP) packetization module of a module Method and device, and the priority output of analyzer according to the network condition of monitoring, such as, packet loss rate, can be utilized to have The selectively networkmonitor of the transmitter/server 200 (Fig. 2 A) of retransmission data and repeating transmission resolver.According to an enforcement Example, the network abstract layer of Real-Time Video Encoder as MPEG-4 scalable video decoder (SVC), such as, H.264/ AVC network abstract layer comprises provides the head that can be analyzed to determine the field of numerical data priority.Such as, NAL is Network friendliness, visual telephone can be represented, such as, video session application and picture will download movies storage in memory, extensively Broadcast or multicast application or streaming application or other non-session application such non-session application.NAL can be defined as, such as, Multiple Abstraction Layer units with the block form containing one or more bytes.First byte of each NAL unit may refer to Show the header byte of the data type of this unit, and remaining byte comprises the payload data of type of NAL head indication.
According to an embodiment, level of abstraction head analyzer comprise analysis scalable video decoder level of abstraction head so that Data are obtained from one of multiple fields.According to some aspects of this embodiment, level of abstraction head analyzer is permissible, such as, The output digital video data of MPEG 4 document reader are operated, or accept as in de-packetizer receive from The numerical data of network.Such as, level of abstraction head analyzer can analyze the DID representing the interlayer coding degree of dependence that layer represents Field.The quality level that QID field references medium size scalability (MGS) layer represents.TID field represents the time that layer represents Level.The result analyzing the level of abstraction head comprising such field identifies that payload data is associated subsequently Layer.The identification of response payload data type, can be that each identification layer specifies different priorities.Such as, Primary layer is permissible Receive the high priority retransmitted from buffer or cache memory, because Primary layer for the decoding completely on receptor is Indispensable.The identification of response payload data type, analyzer can be that each identification layer of following data is specified not Same priority.Such as, Primary layer can receive the high priority retransmitted from buffer or cache memory, because Primary layer Data are indispensable for the decoding completely on receptor.Enhancement layer can be specified high medium priority, because enhancement layer Data are necessary for the decoding of higher level.On the other hand, can be low preferentially to the high enhancement layer appointment of payload data Level.Once specify priority, such as, low, medium and high, if requiring to retransmit on the level of the transport layer, then can be at real-time protocol (RTP) (RTP) " PT Payload Type (PT) " of head or similar field represent precedence information, or can be at the clothes of IP head Service type (TOS) field represents precedence information.Type of service field is referred to as in the prior art, for example, Different service (DS) field.The first two position is referred to as ecn (explicit congestion notification) (ECN) position, then six position quilts of DS field bytes It is referred to as different service code point (DSCP) position.COS is generally used for these of finger offer COS data in this article And extended formatting.Therefore, when being modified to be incorporated in the level of abstraction head analyzer according to an embodiment and method, reliably The real-time packet retransmission that media control protocol module is made determines it can is layer consciousness.
In a further embodiment, it is provided that via data network interface monitoring data network quality and, such as, receive Collection is input to retransmit the current network data statistics amount in resolver, such as, packet loss rate, available bandwidth and round-trip delay Networkmonitor.Retransmit resolver and the input to unreliable medium agreement (RMP) module can be provided again, in order to retransmitting decision can With based on according to the current network conditions of data network statistical data collected in networkmonitor, and preferential according to it The network condition of level and monitoring determines packet retransmission selectively.
Then, digital transmission methods comprises the level of abstraction head analyzing digital data packets and represents data to obtain layer, with And respond this analysis priority is assigned to represent layer digital data packets.Obtain layer and represent that data can comprise one or institute Have the following steps: determine that interlayer encodes degree of dependence;Determine the quality level of granular scalability;And determine that layer represents time Between grade.Additionally, the method can be further contained in the payload type field of real-time Transmission packet header or mutually The type of service field of networking protocol packet header represents one of priority level.As it has been described above, to level of abstraction head analyzer Input can receive from the de-packetizer of receiving network data or from as be associated with mpeg file reader that The local server memory of sample receives.
Additionally, in a further embodiment, digital data transmitter can comprise level of abstraction head analyzer, for from Digital data packets obtains layer represent data and layer represents data assigned priority;Numerical data watch-dog, is used for receiving Collection network transmission statistic amount;And repeating transmission resolver, for according to the digital data packets priority specified and the network of collection Transmission statistic amount decides whether that retransmitting digital data is grouped.As it has been described above, the most such level of abstraction is first Mark analyzer can be the network abstract layer head analyzer of digital of digital video data packet application.
More specifically, with reference to Fig. 2-Figure 10, the level of abstraction head analyzer of server, data monitor and repeating transmission determine For RMP module 235, the networkmonitor of device and receptor as further described herein, be made whether that repeating transmission is sending control High reliability operation when reporting the decision of the packet that lost on channel 255 provides guarantee.Video in wireless network is sent out Send or distribute RTP (RTP) moving picture expert group 2 transmission stream (MPEG2TS) generally used on UDP.Regard in real time Frequency can be distributed to single destination (unicast mode) from single source or be distributed to multiple destination (multicasting mode) from single source. Owing to channel conditions is change in the wireless network, so when channel conditions is bad, if link layer error correcting does not becomes Merit, then packet transmission causes packet loss.In these conditions, in packet sequence, existence causes end user's viewing quality poor Gap.The invention provides the level of abstraction based on the effective re-transmission scheme of application layer being called unreliable medium agreement (RMP) herein Head analytic function recovers lost packets, to contribute to the application of reliable real-time streaming.The present invention is also in transmitter side with connecing Receive device side and network monitoring capability is provided, in order to improve making of the decision of the packet retransmission in the server/transmitter 200 of Fig. 2 A. Therefore, the receptor/client computer 201 of Fig. 2 B is equipped with networkmonitor 271.
With reference to Fig. 2 A and Fig. 2 B, in unreliable medium agreement (RMP) method of the present invention, by the principle according to the present invention It is incorporated to NAL head analyzer 210 to help RTP packetization module 220 and unreliable medium agreement (RMP) module 230.Additionally, with The networkmonitor 270 of network interface 250 handing-over collects current network statistic, and improves RMP mould via retransmitting resolver 275 The reliability of block 230.Highlight NAL head analyzer 210, networkmonitor 270 in fig. 2 and retransmit resolver 275, with Show that Fig. 2 A as will be described as further below is different from Fig. 1.
Initially, RMP module 230, via network interface 250, uses UDP 240 send conventional unicast and multi-case data or divide Group, in order to use the initial channel 260 that sends will send packets to network 110.Data initially via RTP packetization module 220 with The priority that NAL head analyzer 210 is specified is stored in local cache 235.In addition, the source at Fig. 2 A (sends Device, sender) set up between 200 and destination's (receptor, the stay of two nights, client computer) 201 of Fig. 2 B and add reliably based on TCP control Channel 245.TCP/IP 245 controls channel 255 via the repeating transmission ACK/NACK to network 110, is asked by network interface 250 Retransmit and receive the packet lost.In order to make this mechanism properly functioning, transmitter (sender, server) 200 retains transmission To being grouped recently in cache 235 of its receptor/client computer.One or more receptor/client computer 210 is from transmission Device/server 200 receives packet, and can use and be present in RTP (Fig. 6) or mpeg transport stream (TS) head (Fig. 7) In sequence-number field receive packet in detect sequence gaps.If receptor 201 detects sequence gaps, then connect Receive device 201 in the request controlling to send on channel 255 for there being Selective resending lost data packet based on TCP.When transmitter/clothes Business device 200 is when its receptor/client computer 210 one or more receive repeating transmission request, and it is just at its local cache Packet recently is found in 235.If finding the packet of request in local cache 235, then transmitter/server 200 warp By network 110, controlling on channel 255 with the copy of unicast fashion packet retransmission to receptor 201 based on TCP.If at it Do not find the packet of request in local cache/buffer 235, then sender 200 continues as remaining and retransmits request service.Connect Receiving device/client computer 201 maintains conveying queue (buffer/cache) 236 to retain from data channel and to control channel two All reception packets of person.Packet retransmission is also reset the normal sequence in this queue by receptor/client computer 201 In (position), and with suitable order, packet being flowed to application in orthochronous, such as, video player application 221 is to add To show 206.
Receptor/client computer 201 (Fig. 2 B) keeps configurable time window to wait any repeating transmission rather than forevermore etc. Go down, in order to packetization delay and delay jitter can be made to be maintained in application boundary line.Additionally, according to an embodiment, receptor/ Client computer 201 is furnished with the visual angle collection network statistic from receptor/client computer 201 to be sent to via controlling channel 255 The networkmonitor 271 of transmitter/server 200.Emphasize in fig. 2b to show networkmonitor 271, to show as herein It is further described that according to collection network statistic of the present invention and provide it to transmitter/server 200 to improve and retransmitting certainly Cooperating between the network interface 251 being set for out and networkmonitor 271.
If not timely receiving the repeating transmission response of some lost packets, then receptor/the client computer 201 of Fig. 2 B is by remaining Receive packet and pass application from conveying force biographies, such as, video player 221 and display 206.If when acceptable recovery Between receive some packet retransmission outside window, then receptor discards them.It will be noted that Video Applications decodes at video Time Error concealment techniques can be used to tolerate some packet loss.
Referring again to Fig. 2 A, make to retransmit decision by means of networkmonitor 270 and 271 packetizing and improvement, this Unreliable medium agreement (RMP) scheme in invention is operated in via RMP module 230,231 by the help of analyzer 210 in real time Between application/RTP/MPEG TS and UDP/TCP/IP.In server/transmitter 200, level of abstraction head analyzer, such as, This locality is provided by network abstract layer (NAL) head analyzer 210 or the digital of digital video data 205 of network is operated.NAL head Analyzer 210 is real-time protocol (RTP) (RTP) packetization module 220 output priority and data.Additionally, networkmonitor 270 is again Obtain the statistic of current network 114, and control data are exported repeating transmission resolver 275 to help RMP module 230.Such as, Enhancement layer is designated high medium priority and Primary layer be designated the example of high priority and push away further, net faced by RMP module 230 When the available bandwidth of network watch-dog 270 detection is less than given threshold value, retransmits high priority base layer data and do not send medium excellent First level enhancement data.Such as, replace and send all packets being saved in speed buffering/buffer storage 235, retransmit and determine Device 275 can divide according to networkmonitor collect statistics and NAL head as packet loss rate or other network conditions The priority that parser 210 is specified selects the part packet being saved in memorizer 235 for retransmitting.Therefore, as combined Fig. 4 Discussion further described, on analyzer 210 to data assigned priority with according to retransmit resolver 275 repeating transmission determine Highly reliable media protocol (RMP) module 230 that is combined into made provides guarantee.
Referring again to Fig. 2 A, representative network interface 250 is displayed on network 110 and the RMP in server/transmitter 200 Between module 230.The example of Network Interface Module is Ethernet card, the IEEE 802.11/ being connected with computer network 110 WiFi card.
Fig. 2 B shows the schematic diagram of the exemplary realization of customer equipment 201.This client computer can include video player/ Streaming application module 221, display 206, buffer (cache memory) 236, optional media protocol (RMP) module 231, UDP/IP module 241, TCP/IP module 246 and network interface 251.Network interface 251 it may be that such as, Ethernet interface or IEEE 802.11 interface or other known network interfaces.Network interface 251 receives all arrival message.These message arrives not Same socket/address.Networkmonitor 271 collects current network from the visual angle of receptor/client computer 201 from network interface 251 Statistic, and export and suitably control message on the channel 255 of report collect statistics.Therefore, network interface 251 may determine that Forward the place of reception message of automatic network 110, and export from the statistical data of networkmonitor 271 with from TCP/IP The message of module 246.New arrival packet is transmitted to UDP/IP interface by Network Interface Module 251.Retransmitted data packet Request and retransmitted data packet are transmitted to TCP/IP module 246 by RMP module 231.RMP module 231 determines that the data of reception are divided Group is damaged the most, and utilizes the elaborately planned packet recovery of UDP/IP and TCP/IP.RMP module 231 is any damage number The request of repeating transmission is generated according to packet.Then, repeating transmission request is transmitted to TCP/IP module 246 so that at network 110 by RMP module 231 Upper transmission.On network interface 251, any network statistical data that networkmonitor 271 is collected be incorporated to retransmit request in Just send on network 110.The packet of reception is also stored in local buffer 236 to resequence by RMP module 231. Once controlling channel via TCP/IP module 246 from TCP and receive packet retransmission, RAMP module 231 is just arranged by proper order Packet.RMP 231 keeps the queue pressing serial number sequence, and packet is resequenced and insert this buffer areas/queue In.When recovering window and expiring, packet is flowed to such as player/streaming application 221 to be shown by RMP module 231 206.RMP module 231 is data transmission and the integrated offer similar socket application protocol interface (API) 216 with application.Note Meaning, some are grouped in when recovery window expires and may not be restored.
Recover the window packet arrived afterwards that expires to abandon by Fig. 4.Depend on application, such as, for video, stream Formula/player application 221 is by data de-packetization and/or decoding, and passes data to display/speaker 206.To arrive Packet is stored in RMP " buffer areas ", and when application request be grouped, just hands to, and such as, applies so that in addition Render (display) 206.Mark square frame and the RMP module 231 " configuration interface " of " configuration " 226.RMP module can be initially Configure the when of change, in order to parameter as cache memory sizes, the maximum time waiting packet recovery etc. is set.
According in the receptor/client computer 201 of Fig. 2 B, networkmonitor 271 can be from receptor/client computer 201 Visual angle via the current network statistic on interface 251 collection network watch-dog 271, and via control channel 255 by receptor/ The network of client computer 201 experiences output to the server being attached thereto via digital data network 110.Once at transmitter/clothes Business device 200 receives via network 110, via interface 250, the network statistics so collected is exported networkmonitor 270, To supplement the statistic that networkmonitor 270 is collected as discussed above from the visual angle of the transmitter/server of network 110. Digital data network 110 can be including, but not limited to satellite network, continental rise wireless network, fiber optic network, coaxial electric cable network Any digital data network of network, twisted-pair wire net, LAN, wide area network and other known digital data networks.Representative network Interface 251 is displayed between the RAM module 231 on network 110 and client/server 201.The example of Network Interface Module Including the Ethernet card being connected with computer network 110, IEEE 802.11/WiFi card etc..
Before Fig. 3 is discussed, it is important to notice that, similar label represents the similar components in figure.Additionally, as network The first digit of label as the first digit 1 of the label 110 of 110 indicates the figure number that this element occurs for the first time.Cause This, such as, RAM module 230 occurs in fig. 2 for the first time, similar with unreliable medium agreement (RAM) module 130 of Fig. 1, but with Unreliable medium agreement (RAM) module 130 of Fig. 1 is different, because introducing the NAL head analyzer all not appeared in Fig. 1 210, networkmonitor 270 and repeating transmission resolver 275.Additionally, according to the principle of the present invention, the client computer/receptor 201 of Fig. 2 B Networkmonitor 271 improve the RMP module 230 of Fig. 2 A and retransmit the operation of resolver 275.Analyzer 210, network monitoring Some advantages of device 270 and 271 and repeating transmission resolver 275 are described above, but below in conjunction with the discussion of Fig. 3 and Fig. 4 It is further described.
It is first that Fig. 3 provides the NAL level of abstraction relating to sending to the one or more streamed reception devices connected via network 110 Directly representing of mark analyzer 210, networkmonitor 270 and repeating transmission resolver 275.With reference to Fig. 3 A, illustrated therein is and can locate The encoder 300 of the scalable video coder (SVC) in streaming server or away from it.As further discussed below, (such as, scalable video coder (SVC) provides time or frame rate scalability, space or picture size scalability With multiple resolution and the ratio of width to height encoded video) and signal to noise ratio (SNR) or quality/fidelity scalability.Enter as combined Fig. 4 Described in one step, NAL head analyzer 210 can be H.264 compatible AVC network abstract layer (NAL) head analyzer or have phase Other level of abstraction head analyzers like function.Such as, can deposit from local storage with reference to Fig. 3 B, NAL head analyzer 210 Chu Ti, such as, MP-4 document reader 302 receives input.Exemplary streaming server example can be to provide Internet Protocol Television frequency Road or the streaming server of movie services.In alternate embodiments Fig. 3 C, it is shown that receive, such as, carry out the stream of automatic network 110 The packet received also is exported the de-packetizer 304 arriving NAL head analyzer 210 by formula video.The streaming server of Fig. 3 C Exemplary Video Applications can be describe streaming server be to another video communication terminal from a video communication terminal Network path in the visual telephone of a server.
In each of Fig. 3 A, Fig. 3 B and Fig. 3 C, NAL head analyzer 210 will be able to divide via packetizer 310 Group/frame priority is supplied to local cache 235.Sending for initial, analyzer 210 is by defeated to analyzer data and priority Go out to packetizer 310.Via path 312, copy is placed in local cache 235, and via path 314, packetizing Grouped data is sent to the de-packetizer 320 of streaming server by device.Network status transmission data are carried by networkmonitor 270 Resolver 275 is retransmitted in supply, in order to for having selection to determine via the repeating transmission on network 110 to the path 316 of de-packetizer 320 Whether repeating transmission is stored in the packet/frame in local cache 235.Path 316 is for specifying according to status transmission and analyzer Priority retransmit select packet.
Now, with reference to Fig. 4, with reference to illustrating sender or transmitter/server side operation and receptor/client computer/terminal The described flow chart of side's operation further describes level of abstraction head analyzer, networkmonitor and repeating transmission according to the present invention certainly Determine the feature operation of device.In square 400, the server/transmitter of Fig. 3 obtains such as from local file, by Fig. 3 B, file The data of reader 302 or reception are from network interface, such as, from network 110 and the number of de-packetizer 304 of Fig. 3 C According to.The data of reception are supplied to the square 410 of the level of abstraction head analyzer 210 of representative graph 2A and Fig. 3.
Referring briefly to Fig. 5, illustrated therein is the typical level of abstraction head of the field providing to be analyzed, such as, SVC NAL expands Exhibition head.Such as, level of abstraction head analyzer 410 can be analyzed to be shown as occurring in and represent the interlayer coding dependence that layer represents DID field on the 1-3 position of second byte (byte 1) of the level of property or space/resolution scalability.Such as second QID field on the 4-7 position of byte (byte 1) can be followed, represents medium size scalability (MGS) layer and represents Quality level., such as, the TID field on the 0-2 position of the 3rd byte (byte 2) represents the time grade that layer represents.Analyze One result of the level of abstraction head comprising such field is to identify the layer that payload data is associated subsequently.Response has The identification of effect load data type, the analyzer on the analysis block 410 of Fig. 4 can be that each identification layer of following data is specified not Same priority: such as, high, medium and low priority.The priority of more than three rank can be applied in alternative embodiments. Such as, Primary layer can receive the high priority retransmitted from buffer or cache memory, because Primary layer is for receiving Decoding completely on device is indispensable.Enhancement layer can be specified high medium priority, because enhancement data is for higher The decoding of layer is necessary.The SVC encoding stream of numerical data can comprise Primary layer and an enhancement layer.On the other hand, in order to Enjoy higher image quality, it is provided that the second or higher enhancement layer, therefore, for relevant payload data, Ke Yi Higher enhancement layer is specified low priority by cache 235.
In order to provide simplification example, if SVC coding video frequency data has the resolution of 416 × 240 and the position of 600kbps The Primary layer of speed and the higher resolution of 832 × 480 and the position therewith of 1.2 Gigabits per second (Mbps) for video flowing One Primary layer of speed, then, analyzer 410 can be according to its DID field identification Primary layer NAL unit and to Primary layer Specify higher priority.On the other hand, enhancement layer can be specified lower priority (so that high-resolution is provided).In this example In to Primary layer specify than strengthen floor height priority.Output priority can not show RTP in figure 6 on 9-15 position PT Payload Type (PT) field of head indicates.Once specify priority, such as, high, neutralize low, if requirement Retransmit on the level of the transport layer, then can also represent precedence information in the similar field of real-time protocol (RTP) (RTP) head, or permissible Precedence information is represented in COS (TOS) field of IP head.Initially, along with passing through " sending packet " 450 via net Network 110 sends packet, the local cache/buffer storage being buffered on 430 by copy in square 420 to receptor In.Packet may be lost in a network or be received.Packet in local cache 430 waits related network situation and whether Receive the determination of packet.
If reception is grouped and network condition performance is good, then on 460, reception is grouped and receptor determines pre-on 465 Phase packet loss (YES) is again without losing (no).If lost (YES), then send to retransmit on 485 and ask, and via The recovery timing of receptor is set back in sender/transmitter/server 200 by the channel 255 that controls of Fig. 2 B.If it is not, Then in 490, such as, by suitable order reception packet is placed on for the display function 206 in receptor/client computer 201 and connects Receive in buffer.
If sending, via 485, the request of repeating transmission, then activating network monitor functions 405, the repeating transmission then processing reception please Ask.As further described herein, can determine on networkmonitor 271 that the end-to-end packet loss rate of preset time is as coming From current network 110 situation of receptor side.By current network conditions, such as, end-to-end packet loss rate and repeating transmission request one Play the output as 405 to be supplied to retransmit resolver 415.
The example use of networkmonitor 405 can be that sender/server is on cable network 110, such as, and machine Top box (cable or satellite) or home gateway and receptor are the mobile device being associated with wireless access point AP or individual's meter The situation of calculation machine.Intermediate node/WAP (AP/ router) can be to Sender Report network and radio channel state. Therefore, when being modified to be incorporated in the level of abstraction head analyzer according to an embodiment and method, former according to the present invention The real-time packet retransmission that unreliable medium agreement (RMP) module 230 of reason is made determines to have layer consciousness.
Retransmitting resolver 415 to inquire, such as, whether current end-to-end transmission packet loss rate is high, it is meant that it is On no threshold level in the memorizer being arranged on RMP module 230.If answer is "Yes", then 445 abandon relatively low preferentially Level packet, and only recover from cache/buffer 430 and retransmit higher priority packets.In addition to packet loss rate Other tolerance of status transmission can be available bandwidth (such as, the lowest available bandwidth in end-to-end path) and round-trip delay The time-out of the packet being saved in retransmit in cache/buffer 235 (the longer round-trip delay may require).Permissible By include each of these status transmissions of available bandwidth compared with relevant threshold level in memory is set, to determine Packet retransmission.One or more status transmissions can be applied and make the packets retransmitted decision and arranged by analysis 410 In priority.If returning the answer of negative, then on 435, via " send packet " 440 and network 110 from cache/slow Rush device 430 and retransmit all request packets.
Once receive, on 460, receive packet retransmission, and on 470, be identified as packet retransmission.Given viewing in real time Experience, on 475, check the reception time.Ask: retransmit evening?If packet retransmission is received too late, say, that program Viewing experience is transferred on next frame, then abandon packet retransmission on 480.On the other hand, if packet retransmission is the most late (that is, answer is negative), then be placed on packet retransmission in reception buffer in order to be shown on 490.
The RMP method of the present invention can be at flexible software library, hardware, firmware, any computer or processor, special integrated Circuit (ASIC), Reduced Instruction Set Computer (RISC), field programmable gate array (FPGA) or combinations thereof body realize. The RMP method of the present invention use similar socket user's space API and bottom transmission means be convenient to streaming server and Player application is integrated.The streaming application that it is supported by the RMP method of the present invention is transparent.Internal holding UDP message channel Channel is controlled with TCP.The RMP method of the present invention is extendible, to support that other mistake is entangled as FEC and mixing ARQ Direction-determining board.
With reference now to Fig. 6, Fig. 7 and Fig. 8, further describe the networkmonitor function 405 of the present invention.A kind of possible means In figure 6, wherein networkmonitor 271 can be such as from serial number in display, i.e. detect between sequence in the 6-1 position of the first row Gap.Another kind of means are as it is shown in fig. 7, utilize continuous counter (CC) field of the mpeg 2 transport stream head of a byte.
The statistic that the networkmonitor 271 of the receptor/client computer 201 of Fig. 2 B is collected is sent to the service of Fig. 2 A The exemplary format of the message of device/transmitter 200 shows in fig. 8.This message format may look like RTP Control Protocol (RTCP) receptor report (RR) message.Especially, such as, the 0-7 position of the 4th row can provide " loss ratio " field. May be provided for other related data as the accumulated quantity of lost packets, can by the accumulated quantity of this lost packets with The previously received accumulated quantity of the lost packets received on same channel compares.This difference provides and is become in sequential messages The quantity of the packet lost in the time defined interval between merit reception.
Fig. 9 provides sender/server/transmitter by the repeating transmission acquisition request received from receptor/client computer 201 The form of Network packet loss.The networkmonitor function 405 of Fig. 4 is from " starting the basic sequence number of packet " and " end of packet Side-play amount " field calculates the quantity of lost packets.Similarly, provide can in RMP ARQ request (NACK) packet of Figure 10 With storage and the basic sequence number compared with receiving basic sequence number subsequently.The basic sequence number described and bitmap instruction are wanted The sequence of the packet of the receptor request retransmitted.Determine that another means of network status transmission are monitoring lower level such as nets Network/MAC/PHY level sends statistic, to estimate channel conditions and available bandwidth.
In the RMP scheme of the above-mentioned present invention, change is not made in the packet to sending on data channel 255.Therefore, Maintain backward compatibility.Additionally, the RMP scheme of the present invention is effectively utilized bandwidth, because only at the control channel of low overhead Media packet is lost in upper request and repeating transmission.Lost packets request is used as NACK (Negative Acknowledgement), and also offer arrives sender's Feedback.Because lost packets can repeatedly be retransmitted in recovery time in window, it is possible to carry under extensive different channels situation For high reliability.Additionally, the RMP scheme of the present invention strengthens by allowing repeating transmission have maximum latency (that is, recovering window) Application delay constraints, therefore acts on best endeavors conveying model under retraining preset time.
Noting, above embodiment uses video to send and illustrates.But the present invention can also be applied to sending out of audio frequency Send, such as, phone and the application of other real-time multimedia streaming.
Although the such scheme of the present invention describes for wireless network, but the program can also be used in and involve point In any type network that group is lost.
Should be understood that the present invention can be with the various forms of hardware, software, firmware, application specific processor or combinations thereof Realize.Preferably, the present invention is implemented as the combination of hardware and software.Additionally, software can be preferably implemented into visibly The application program being embodied on program storage device.Application program can upload to comprise on the machine of any suitable framework also Performed by it.Preferably, containing as one or more CPU (" CPU "), random access storage device This machine is realized on the computer platform of hardware as (" RAM ") and input/output (" I/O ") interface.This computer platform Also include operating system and micro-instruction code.Various processes and functions described herein can perform via operating system Part micro-instruction code or certain applications program (or combinations thereof).It addition, as additional data storage device and printing device Such other ancillary equipment various can be connected with computer platform.
It should also be understood that preferably come real with software because of some the subsystem parts described in the accompanying drawings and method step It is existing, so the actual connection between system unit (or process steps) may be different with the mode of the programming present invention.Given this paper Teaching, those of ordinary skill in the related art are it is contemplated that the similar realization of the present invention or configuration.

Claims (6)

1. a method, including:
Receive the video data of coding from digital video data source, the video data of described coding is scalable coding and takes Band video content, the video data of described coding includes network abstract layer NAL extension header, for the first of described video content Layer, and the second layer for described video content, described ground floor includes being used for from the video data of described coding obtaining First video data of the video content of gross, the described second layer includes the use being used together with described first video data The second video data of the video content strengthening quality is obtained in the video data from described coding;
Resolve NAL extension header to identify described first video data and described second video data;
Described first video data will be distributed to relative to the described second higher priority of video data be used for retransmitting;
In buffer storage, caching has the first video data and second video data of the priority of distribution;
Via the first data channel of digital data network to the video data of coding initially sent receptor, described numeral Data network also include the second data channel, described first data channel utilize the first host-host protocol and provide from conveyer to The one way data communication of described receptor, described second data channel utilize the second host-host protocol and provide described conveyer and Bidirectional data communication between described receptor, the reliability of described first host-host protocol can less than described second host-host protocol By property;
Monitor digital data network with from the angular collection of described conveyer about described digital data network statistics first Network data, described first network data instruction first network status transmission, described first network status transmission includes the first end Opposite end packet loss rate, the first available bandwidth of described digital data network, and at least one in the first round-trip delay;
The repeat requests transmitted via receptor described in described second data channel reception, described repeat requests includes connecing from described Receiving the second network data of the statistics of the relevant described digital data network of the angle of device, described second network data indicates in institute Stating the second network status transmission of the described digital data network determined at receptor, described second network status transmission includes Two port end packet loss rate, the second available bandwidth of described digital data network, and at least in the second round-trip delay ?;
Utilize described second network data that described first network data are supplemented;And
Determine whether to recover at least described first video data and described second video data from described buffer storage Individual and whether in response to described repeat requests in view of the first network data after supplementing and according to the priority warp distributed By the second data channel, the video data recovered is sent to described receptor.
Method the most according to claim 1, wherein, described digital video data source includes MP4 document reader and packet One of de-packetizer.
Method the most according to claim 1, wherein, described first host-host protocol includes UDP UDP, described Second host-host protocol includes transmission control protocol TCP.
Method the most according to claim 1, wherein, described digital data network include satellite network, roadbed wireless network, At least one in fiber optic network, coax network.
Method the most according to claim 3, wherein, described digital data network include satellite network, roadbed wireless network, At least one in fiber optic network, coax network.
Method the most according to claim 1, wherein said network abstract layer NAL extension header includes H.264/AVC network The NAL extension header of level of abstraction.
CN201610566114.1A 2010-01-28 2010-01-28 The method and apparatus determined is retransmitted for making Pending CN106100803A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610566114.1A CN106100803A (en) 2010-01-28 2010-01-28 The method and apparatus determined is retransmitted for making

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2010800658622A CN103109485A (en) 2010-01-28 2010-01-28 A method and apparatus for retransmission decision making
CN201610566114.1A CN106100803A (en) 2010-01-28 2010-01-28 The method and apparatus determined is retransmitted for making

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
CN2010800658622A Division CN103109485A (en) 2010-01-28 2010-01-28 A method and apparatus for retransmission decision making

Publications (1)

Publication Number Publication Date
CN106100803A true CN106100803A (en) 2016-11-09

Family

ID=57240753

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610566114.1A Pending CN106100803A (en) 2010-01-28 2010-01-28 The method and apparatus determined is retransmitted for making

Country Status (1)

Country Link
CN (1) CN106100803A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109428683A (en) * 2017-08-24 2019-03-05 福建省华渔教育科技有限公司 Packet acknowledgement method and terminal based on bitmap
CN113900816A (en) * 2021-10-15 2022-01-07 无锡江南计算技术研究所 Multi-depth buffer activation retransmission method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6021516A (en) * 1996-01-23 2000-02-01 Ntt Mobile Communications Network, Inc. Communication system and transmission station
CN1476267A (en) * 2002-05-10 2004-02-18 ���ǵ�����ʽ���� Device and method for retransmitting data in mobile communication system
CN101180831A (en) * 2005-05-24 2008-05-14 诺基亚公司 Method and apparatus for hierarchical transmission/reception in digital broadcast
CN101543018A (en) * 2007-01-12 2009-09-23 庆熙大学校产学协力团 Packet format of network abstraction layer unit, and algorithm and apparatus for video encoding and decoding using the format, qos control algorithm and apparatus for ipv6 label switching using the fo

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6021516A (en) * 1996-01-23 2000-02-01 Ntt Mobile Communications Network, Inc. Communication system and transmission station
CN1476267A (en) * 2002-05-10 2004-02-18 ���ǵ�����ʽ���� Device and method for retransmitting data in mobile communication system
CN101180831A (en) * 2005-05-24 2008-05-14 诺基亚公司 Method and apparatus for hierarchical transmission/reception in digital broadcast
CN101543018A (en) * 2007-01-12 2009-09-23 庆熙大学校产学协力团 Packet format of network abstraction layer unit, and algorithm and apparatus for video encoding and decoding using the format, qos control algorithm and apparatus for ipv6 label switching using the fo

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109428683A (en) * 2017-08-24 2019-03-05 福建省华渔教育科技有限公司 Packet acknowledgement method and terminal based on bitmap
CN109428683B (en) * 2017-08-24 2021-06-08 福建省华渔教育科技有限公司 Bitmap-based packet acknowledgement method and terminal
CN113900816A (en) * 2021-10-15 2022-01-07 无锡江南计算技术研究所 Multi-depth buffer activation retransmission method and device
CN113900816B (en) * 2021-10-15 2024-03-22 无锡江南计算技术研究所 Multi-depth buffer activation retransmission method and device

Similar Documents

Publication Publication Date Title
US9306708B2 (en) Method and apparatus for retransmission decision making
EP2529528B1 (en) A method and apparatus for parsing a network abstraction-layer for reliable data communication
US8175036B2 (en) Multimedia wireless distribution systems and methods
KR100941562B1 (en) Medium streaming distribution system
JP3757857B2 (en) Data communication system, data transmission apparatus, data reception apparatus and method, and computer program
US20110044338A1 (en) Throughput in a lan by managing tcp acks
US9565482B1 (en) Adaptive profile switching system and method for media streaming over IP networks
US9084177B2 (en) Adaptive time allocation in a TDMA MAC layer
US9781488B2 (en) Controlled adaptive rate switching system and method for media streaming over IP networks
JP2004187286A (en) Mpeg-4 live unicast video streaming system in wireless network equipped with congestion control of end-to-end bit rate reference
CN108781139A (en) Data in packet network retransmit
KR20120082416A (en) An efficient application-layer automatic repeat request retransmission method for reliable real-time data streaming in networks
JP2012504352A (en) Method and apparatus for dynamically adapting the number of retransmissions
WO2010054551A1 (en) Method, device and system for video stream transmission
CN106100803A (en) The method and apparatus determined is retransmitted for making
EP1450535A1 (en) A relay for hierarchical retransmissions in multimedia streaming
EP1947859A1 (en) Video transmission method and system
JP2013013093A (en) Improving throughput in lan by managing tcp acks
Wang et al. Performance of MPEG-4 transmission over SCTP multi-streaming in wireless networks
Wang et al. MPEG-4 optimal transmission over SCTP multi-streaming in 802.11 wireless access media
Quaritsch et al. An adaptive multi-purpose transmission scheme for H. 264 encoded video in wireless networks

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
AD01 Patent right deemed abandoned

Effective date of abandoning: 20191022

AD01 Patent right deemed abandoned