KR100623017B1 - Mobile Terminal Client and Method of Sensing and Recovery of Abnormality of Decorder - Google Patents

Abstract

The present invention relates to a method for resolving and detecting a decoder abnormality caused by data loss by software using a motion analysis technique. The present invention relates to a network monitor for monitoring a current loss state of streaming data. And a motion analyzer for determining whether the loss of the streaming data is due to a malfunction of a decoder or a loss occurring during radio transmission when the loss of the streaming data is detected.

Streaming, Motion Analyzer, Network Monitor

Description

Mobile Terminal Client and Method of Sensing and Recovery of Abnormality of Decorder             

1 is a block diagram illustrating a wireless streaming process according to an embodiment of the prior art.

2 is a block diagram illustrating a wireless streaming process according to an embodiment of the present invention.

3 is a block diagram of a network monitor according to an embodiment of the present invention.

Figure 4 shows the operation of the motion analyzer according to an embodiment of the present invention.

5 is a flow chart showing a method of operation according to an embodiment of the present invention.

{Description of Major Symbols in Drawings}

204: source filter 205: network render filter

209: network source filter 210: MPEG2 splitter filter

211: decoder filter 212: display overlay

213: MPEG2 Decoder 214: Motion Analyzer

215: stream client application

216 network monitor 301 RTP header analyzer

302: Motion Analyzer Handler

The present invention relates to a mobile terminal client for detecting and recovering a decoder abnormality. More specifically, the present invention relates to data loss that may be caused by temporary radio channel deterioration when the mobile terminal client communicates with a streaming server through a wireless network. The present invention relates to a mobile terminal client that detects and recovers a malfunction of a decoder based on motion analysis.

Currently, in order to support media streaming to users, various applications have been developed for transmitting encoded data using an encoding technique such as MPEG to a network, decoding the client, and displaying the encoded data.

Such applications are generally composed of a system for encoding data, a transmission / reception subsystem for transmitting data to a network, and a system for decoding transmitted data. Such a system has a large amount of loss of data transmitted to the network. Does not have a recovery plan.

Therefore, when real-time data is transmitted and received through the wireless network, a large amount of data loss may occur depending on the channel environment of the network. Accordingly, in order to support continuous service, recovery of an error of the decoding system caused by data loss is required. A solution is needed.

1 is a block diagram illustrating a wireless streaming process according to an embodiment of the prior art.

Referring to FIG. 1, the stream server encodes real time data obtained from a real time media source (TV tuner) through an encoder and transmits this data through an air interface. The transmitted data is delivered to the client's decoding system, decoded and sent to the screen.

The server encodes the real-time stream data in a predetermined format and transmits it to the network interface, and the client decodes the data and displays the decoded result on the screen, thereby performing a real-time media streaming service.

Real-time streaming applications have a certain amount of buffers for transmission delays and small amounts of data loss.

However, when the server and the client communicate over the wireless network, if the wireless channel condition worsens and a large amount of data loss occurs, the decoder cannot decode the data, freezes the screen, and restarts the media application to continue the streaming service. I can receive it.

In addition, since an error in the decoding subsystem does not automatically recover, there is a problem in that data normally transmitted through a network cannot be decoded.

In order to solve the above problems, when a malfunction of a decoder occurs due to a large amount of data loss and a picture freezes, the present invention detects this and initializes the decoder so that the data transmitted thereafter can be normally processed. To provide a device.

Another object of the present invention is to provide a method for improving the decoder abnormal state recognition rate by analyzing the data loss amount and the characteristic pattern change of the frame buffer memory according to the network state.

In order to achieve the above object, a mobile terminal client receiving a real-time media streaming service of the present invention includes a network monitor that monitors a current loss state of the streaming data among the received streaming data, and when the loss state of the streaming data is detected, the streaming data. And a motion analyzer for determining whether the loss of the signal is due to a malfunction of the decoder or a loss that occurs during radio transmission.

In the present invention, the network monitor checks the sequence number included in the RTP header extracted from the received streaming data to determine whether the data is lost by the RTP header analyzer and the RTP header analyzer. Preferably, a motion analyzer for detecting a motion analyzer and a decoder for controlling a decoder for decoding streaming data are included.

In the present invention, when the motion analyzer handler detects data loss due to a decoder malfunction from the motion analyzer, the motion analyzer handler resets the decoder and waits for the decoder to be notified that the decoder is in a normal state. It is desirable to stop the motion analyzer and resume the decoding process.

In the present invention, a method for detecting and recovering an abnormal shape of a decoder through motion analysis includes a) storing a decoding result, b) extracting a 'Y' value related to motion among the result values, and c) from the 'Y' data. Generating a data set from which the data field is extracted; d) extracting the generated data set and calculating a correlation according to a sampling rate; and generating a feature related matrix. Determining whether the error occurs due to a decoder abnormality or caused by maintaining the same streaming data for a predetermined time.

In the method of forming the data set of step c) in the present invention, it is preferable to extract the 'Y' data field centered on the field among the most frequent motion changes.

In the present invention, the method of determining the reason for the screen freeze of step e) may be determined to be an abnormal phenomenon of the decoder if the feature-related matrix is a unit matrix.

The present invention further includes the step of continuously operating the decoder if the data of the feature-related matrix is the same when it is determined that the decoder is abnormal, otherwise stopping the decoding operation, and restarting the decoder after the decoder reset. It is desirable to.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram illustrating a wireless streaming process according to an embodiment of the present invention.

In this embodiment, the stream server is a real time media source 202, MPEG2 encoder 203, source filter 204, network render filter 205, Winsock API 206, TCP / IP layer 207, MAC layer 208, the mobile terminal client includes a MAC layer 208, a TCP / IP layer 207, a Winsock API 206, a network source filter 209, an MPEG2 splitter filter 210, a decoder filter 211. , Display overlay 212, MPEG2 decoder 213, motion analyzer 214, stream client application 215, and network monitor 216.

The above embodiment schematically illustrates a process of transmitting streaming data from a stream server to a mobile terminal client.

Referring to FIG. 2, the stream server encodes real-time media data 202 and transmits data in accordance with the direct show structure of the source filter 204 and the network render filter 205.

The mobile terminal client decodes MPEG2 stream data through a network source filter 209, an MPEG2 splitter filter 210, and a decoder filter 211 to transmit a screen. The stream client application 215 structures these filters and manages connection control.

The network monitor 216 monitors the current data loss situation. When data loss is detected by the network monitor 216, the motion analyzer 214 is activated.

The motion analyzer 214 analyzes the data of the video memory currently being overlaid and determines whether the overlay screen is frozen or not according to the decoder 213 malfunction. If the motion analyzer 214 determines that the overlay is not updated due to a malfunction of the decoder 213, the service is initialized to initialize the state of the decoder 213 to enable continuous service.

Detailed description of each of the components will be described in more detail below, and other components described above are already known and apparent to those skilled in the art, the detailed description thereof will be omitted.

3 is a block diagram of a network monitor 216 according to an embodiment of the present invention.

In this embodiment, the network monitor 216 includes an RTP header analyzer 301 and a motion analyzer 214 handler.

This embodiment schematically illustrates how network monitor 216 monitors for loss of streaming data.

Referring to FIG. 3, the network monitor 216 monitors a loss state of data transmitted to a network. When the network loss is detected, the motion analyzer 214 is operated, and the state of the current decoder 213 is fed back through the motion analyzer handler 302.

The network source filter 209 receives the stream data from the server, extracts the RTP header, and delivers the media data to the MPEG2 splitter 210 / decoder filter 211 for decoding. The RTP header includes a sequence number entered before transmission from the server, and through this, the RTP header analyzer 301 detects that there is data loss.

Information about data loss is sent to the motion analyzer handler 302. When the motion analyzer handler 302 is notified that the data loss is detected by the RTP header analyzer 301, the motion analyzer handler 302 operates the motion analyzer 214 for detecting whether the decoder 213 is malfunctioning.

When the decoder 213 anomaly detection status is notified from the motion analyzer 214, the decoding operation is temporarily stopped, and the decoder 213 is overcome by reactivating the decoder 213 after resetting the decoder 213.

When the data loss is detected, the motion analyzer 214 waits until the decoder 213 is notified of the normal state, and when the motion analyzer 214 is notified that the decoder 213 is in the normal state, the motion analyzer 214 is notified. ) And enter normal operation.

4 is a view showing the operation of the motion analyzer 214 according to an embodiment of the present invention.

In this embodiment, the operation of the motion analyzer 214 includes a feature extraction step, a feature correlation matrix generation step, and a motion check step.

The embodiment shows an approximate process of analyzing the cause of still images.

Referring to FIG. 4, the motion analyzer 214 stores decoding results in a frame buffer 401, obtains a feature-related matrix from data stored in the frame buffer 401, and based on the obtained matrix, the decoder 213 or more. Determine the state.

The decoding result is stored in the frame buffer 401 to be overlaid. In this data, a 'Y' value related to motion is extracted. If it is determined that there is a data loss, feature extraction is used to obtain 'Y' data.

A data group consisting of data obtained by selecting a specific position from 'Y' data obtained through feature extraction is called a feature related matrix. The data group 403 is obtained by extracting the 'Y' data field centered on the field in which the motion change occurs the most from the 'Y' data.

The feature-related matrix is generated by extracting a certain data group from the generated data and calculating correlations according to sampling rates. The feature-related matrix is used as an important index to determine the abnormality of the decoder 213.

When the related matrix obtained above is in the form of an identity matrix, motion may not be detected, and thus it may be sensed that no screen change occurs. The unit matrix indicates a square matrix in which all of the elements on the main diagonal have a value of 1 and all of the remaining elements have a value of 0. The related matrix is a square matrix. The remaining elements all have a value of 0 and may have the same shape as the unit matrix. Whether the screen change does not occur is due to an abnormal phenomenon of the decoder 213 or whether the same source signal is caused by being maintained for a predetermined time.

In general, in the case of a system that encodes and transmits real-time data such as a TV screen and decodes it, even in the case of a still picture, some vibration is detected by noise of an analog signal. This vibration causes a slight motion change and shows a difference in the feature set. Thus, the correlation matrix does not become a unitary matrix.

However, this difference does not appear when data is processed by processing a signal regardless of the source signal. For example, when the screen of a channel where a TV signal is not recognized is processed in blue, noise is not generated in this case because artificially filling all data fields with constant data.

In this case, the feature correlation matrix becomes a unit matrix, but since all data of the feature set have the same value, the motion check module determines the above case and determines that the decoder 213 is not an abnormal phenomenon.

5 is a flowchart showing a method of operation according to an embodiment of the present invention.

Referring to FIG. 5, steps 501 to 502 are processes of checking time.

When the screen on which the streaming data is displayed stops, the timer inside the mobile terminal client checks the screen change for a predetermined time. The said time can be adjusted according to the objective of this invention.

Step 503 is a process for checking whether data is lost.

When the predetermined time has elapsed, it is checked whether the displayed screen is stopped due to loss in data transmission or whether it is an original still screen.

Step 504 is a process in which the motion analyzer 214 operates.

If it is determined in step 503 that the reason for freezing is due to data loss, the motion analyzer 214 is operated.

Steps 505 to 506 are for calculating a feature correlation matrix.

As described above with reference to FIG. 4, when the motion analyzer 214 operates in step 504, the decoding result is stored in the frame buffer 401, and a feature correlation matrix is obtained from the data. Since the details have already been described above, the description thereof will be omitted below.

Steps 507a to 508 are for determining the reason for freezing based on the feature correlation matrix.

When the feature correlation matrix obtained in step 506 is a unit matrix, in step 508 it is determined whether the elements of each matrix are the same. If the value of each element is the same, this step is repeated as it is not an error of the internal decoder 213 because it is a channel on which the TV signal is not recognized.

Steps 509 to 511 control the decoder 213.

If the element values of the feature correlation matrix in step 508 are not the same, the stop of the screen is caused by an error of the decoder 213 inside the mobile terminal client, so that the motion analyzer handler 302 stops the decoding process. The decoder 213 is reset and decoding starts again.

As described above, although described with reference to a preferred embodiment of the present invention, those skilled in the art will be able to vary the present invention without departing from the spirit and scope of the invention described in the claims below. It will be understood that modifications and changes can be made.

As described above, the present invention detects an abnormality of a decoder that may occur in a WLAN-based stream service that may cause a large amount of data loss according to a channel environment, thereby preventing the decoder abnormality from continuing.

In addition, it is possible to provide a continuous service by automatically detecting and overcoming the decoder abnormality caused without causing the user to restart and process the service.

In addition, since the decoder abnormal state is implemented in software through the frame memory, it can be used universally regardless of the hardware characteristics of the decoder, and can be extended to other applications using the frame memory such as digital television and set-top box. .

Claims (7)

delete delete delete Detecting and recovering decoder abnormality through motion analysis, a) storing the decoding result; b) extracting a 'Y' value related to motion from the result value; c) generating a data set from which data fields are extracted from the 'Y' data; d) extracting the generated data set and calculating correlations according to sampling rates to generate feature related metrics; and and e) determining whether the screen freeze is caused by a decoder abnormality or whether the same streaming data is maintained for a predetermined time by using the feature-related metrics. The method of claim 4, wherein the method of forming a data set of step c) comprises: Decoder detection and recovery method characterized by extracting the 'Y' data field centered on the field among the most changes in motion. The method of claim 4, wherein the determining of the reason for freezing in step e) If the feature-related matrix is a unit matrix, it is determined as an abnormality of the decoder, otherwise it is determined that the same streaming data is caused by maintaining for a predetermined time. The method of claim 6, wherein the determination of the abnormality of the decoder And if the data of the feature-related matrix is the same, continue operating the decoder, otherwise stop the decoding operation, and restart the decoder after the decoder reset. Way.

Images