KR102384249B1 - Method for processing data by use of multi-threads on a web browser - Google Patents

Abstract

The present invention relates to a method for processing data in multiple threads on a web browser. The multi-thread includes a first thread that is a main thread and a second thread that is a lower thread. According to the present invention, it is possible for the first thread to transfer data to the second thread without overflow of the message queue of the second thread and to process the data on the web browser. Therefore, it is possible to stably process data with multi-threads on the web browser, and through this, the reliability of a device that processes data with multi-threads on the web browser can be increased.

Description

Method for processing data by use of multi-threads on a web browser

The present invention relates to a method of processing data on a web browser, and more particularly, to a method of processing data while transferring data between a plurality of threads without overflow.

JavaScript, a programming language used in web browsers, is one of the elements that compose web pages along with HTML and CSS. While HTML is responsible for the composition of the web page and CSS is responsible for the design of the web page, JavaScript is responsible for the operation of the web page.

JavaScript is an object-based scripting language. And since JavaScript is a dynamic language, when JavaScript is used, a dynamic web page can be implemented unlike static HTML. Also, JavaScript has a simpler syntax compared to other languages, so many programmers can easily use it. In addition to these characteristics of JavaScript, there is one important characteristic, which is that JavaScript is single-threaded.

Single-threading means that one process is executed as one thread. Since JavaScript operates as a single thread, it cannot process multiple tasks at the same time. For this reason, conventionally, while the script is operating, the web page does not respond until the script is completed. Accordingly, the developers tried to imitate the simultaneous execution of scripts through setTimeout, setInterval, XMLHttpRequest, and event handlers, but it was not possible to fundamentally solve the single-threading characteristic of JavaScript.

Web workers are designed to overcome the single-threaded characteristics of JavaScript. Web Worker is a script that runs in the background on an HTML page, and was developed by the World Wide Web Consortium (W3C) and the Web Hypertext Application Technology Working Group (WHATWG). Web Worker is fully supported from Chrome 4.0, Internet Explorer 10.0, Firefox 3.5, Safari 4.0, Opera 11.5, and later. In addition, it started to be supported from iOS 5 and Android 2.1, and in the case of Android, support was stopped in versions 2.2-4.3 and is supported again from version 4.4.

Since the web worker runs in the background of the web page, it does not affect the web page. In other words, whenever a web worker is created in a web page, it creates its own thread that can execute JavaScript, so the web worker operates as a separate thread from the web page, and the web page affects the process running in the web worker. You can run the script without receiving the .

Data is transmitted and received between the web page and the web worker so that the web page and the web worker perform a task in the web browser with multi-threads. In this case, the postMessage method and onMessage event handler are used. The postMessage method is used to pass data from the web page to the web worker, and the onMessage event handler is used to receive data from the web worker. Conversely, the same method is used when passing data from a web worker to a web page.

Data delivered through postMessage from the web page is stored in the message queue of the web worker. Data stored in the message queue in the web worker is moved to the call stack and processed. However, if an excessive number of data is transferred from a web page to a web worker compared to the number of data moving to the call stack, the message queue may overflow due to a bottleneck.

Since the web worker is a script designed to perform a long-time task in the background, there is no device in place to prevent overflow of the message queue due to excessive data being delivered to the web worker in a short time. However, web workers do not necessarily have to be used only for time-consuming tasks such as image processing. Also, in image processing, when image data is divided into small pieces, a situation may arise in which a plurality of divided image data needs to be transmitted from a web page to a web worker.

When the web worker's message queue overflows, the script being executed by the web worker may be locked or the web browser may not work. This can be a fatal problem when performing tasks on a web browser in real time, such as real-time video streaming. A web worker designed to perform multi-threaded tasks in a web browser should not cause a situation in which the script does not work due to an overflow of the message queue. A method of handling is required.

Korean Publication No. 10-2014-0103038

An object of the present invention is to provide a method for processing data in a web browser in which an overflow does not occur in a message queue of a web worker due to data transfer between a web page and a web worker.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A data processing method according to an embodiment of the present invention for solving the above problems is a data processing method performed by at least one processor using a thread supported by the web browser on a data processing device having a web browser. , creating a first thread on the web browser; generating, by the first thread, a second thread; transmitting data to be processed by the first thread to the second thread; storing, by the second thread, the transferred data in a message queue of the second thread; and processing, by the second thread, the stored data, wherein the transferring the data to be processed to the second thread includes the first thread according to the current state of the message queue of the second thread. It may include adjusting the number of data to be transmitted, and transmitting the adjusted number of data to the second thread.

Other specific details of the invention are included in the detailed description and drawings.

According to the embodiments of the present invention, there are at least the following effects.

According to the above problem solving means, it is possible to process a task that involves transferring a large number of data from a web page to a web worker without overflow of the message queue of the web worker. Therefore, it has the effect of broadening the range of tasks that can be processed by the web worker.

In addition, according to the above problem solving means, since the web worker can be used stably without overflow of the message queue of the web worker, the reliability of a device that processes data in multiple threads on a web browser can be improved. Accordingly, the present invention will be very usefully used in situations where real-time performance is important, such as real-time image processing.

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

1 shows a multi-thread operating on a web browser.
2 shows how the second thread 200 processes data.
3 shows data 211 enqueued in the message queue 210 and data 212 dequeued.
4 shows data 213 stored in a message queue 210 .
5 shows a method of transferring data from the first thread 100 to the second thread 200 in a conventional manner.
6 illustrates a method of generating intermediate data in the first thread 100 and transmitting the intermediate data to the second thread 200 according to an embodiment of the present invention.
7 shows an embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .
8 shows another embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .
9 shows another embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .
10 shows an apparatus for processing image data in a multi-threaded manner.
11 shows RTP data and frame data.
12 shows an image processing apparatus according to an embodiment of the present invention.
13 shows RTP data, RTPInt data, and frame data.
14 shows message queues 210a and 210b in which RTP data and RTPInt data are enqueued.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly defined in particular.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a multi-thread operating on a web browser. The multi-thread is a thread for executing a script written in JavaScript, and includes a first thread 100 and a second thread 200 .

The first thread 100 may be a main thread created on a web browser. Typically, the main thread is used to run user interface scripts. That is, the first thread 100 may be a web page that performs a user interface (UI) function so that the user performs various operations. The second thread 200 is a sub-thread generated from the first thread 100 and may be a web worker. Although the second thread 200 is represented as one thread in FIG. 1 , the second thread 200 may be a plurality of threads.

The first thread 100 may create the second thread 200 by using a web worker in the method shown in Table 1 below.

var myWorker = new Worker("worker.js");

If only the first thread 100, which is the main thread, is created and used on the web browser, the web page may not respond until the running process ends due to the single-thread characteristic of JavaScript. On the other hand, when the second thread 200 is created as a web worker, since the second thread 200 executes the script as a thread independent of the first thread 100 in the background of the web page, the first thread 100 is In the two threads 200 , the script can be freely executed regardless of whether the process has been completed.

In order to perform a process using the first thread 100 and the second thread 200 on the web browser, data transfer may be performed between threads. In this case, data transfer between threads can be done through the postMessage method and onMessage event handler. In this case, only a single variable is allowed for the transferred data. The postMessage method is used to transmit data from the first thread 100 to the second thread 200 , and the onMessage event handler is used to receive data from the second thread 200 . Data transfer from the second thread 200 to the first thread 100 may be performed in the same manner. A method of transferring data between the first thread 100 and the second thread 200 using the postMessage method and the onMessage event handler is shown in Table 2 below.

Passing data from first thread to second thread:
myWorker.postMessage(aMessage, transferList);

Receive data in first thread:
myWorker.onmessage = function(event) {
value = event.data;
}

Passing data from second thread to first thread:
postMessage(aMessage, transferList);

Receive data in second thread:
onmessage = function(event) {
value = event.data;
}

In FIG. 1 , the first thread 100 transmits data to be processed through postMessage to the second thread 200 . The second thread 200 performs an operation with data received from the first thread 100 . While the second thread 200 performs a task, the first thread 100 may wait for a task result to be received or may perform a separate task. When the work is completed, the second thread 200 delivers the processed data to the first thread 100 through postMessage. At this time, the reason that the first thread 100 can execute the process even though the second thread 200 is in the middle of executing the script is because the second thread 200 performs the task as an independent thread in the background.

The apparatus for processing data on a web browser by the method described above may process data in multi-threads using the first thread 100 , the second thread 200 , and postMessage.

2 shows how the second thread 200 processes data. In FIG. 2 , the second thread 200 includes a message queue 210 independent of the first thread 100 , an event loop 220 , and a call stack 230 .

The message queue 210 is a module for storing tasks to be processed. In the call stack 230, call information of a function for performing an operation, etc. are accumulated. The call stack 230 operates in a last in first out (LIFO) method in which a function pushed last is called and executed first. When all the functions stacked in the call stack 230 are executed according to the LIFO, the call stack 230 is emptied. The event loop 220 performs a function of moving a job waiting in the message queue 210 to the call stack 230 when the call stack 230 is empty.

A method of processing data by the second thread 200 based on the functions of the message queue 210 , the event loop 220 , and the call stack 230 will be described.

Data received from the first thread 100 is stored in the message queue 210 . Data stored in the message queue 210 moves from the message queue 210 to the call stack 230 by the event loop 220 when the call stack 230 becomes empty. At this time, data moving to the call stack 230 is data stored first in the message queue 210 , and data in the message queue 210 is enqueued and dequeued in a first in first out (FIFO) manner. The call stack 230 processes data by executing functions accumulated in the call stack 230 according to the LIFO.

In the process where the second thread 200 processes data, the message queue 210 enqueues the data received from the first thread 100 and dequeues the data to move to the call stack 230 repeatedly. . The number of data stored in the message queue 210 varies depending on the number of data flowing in from the first thread 100 and the number of data flowing out of the call stack 230 . In this case, data transmitted by one postMessage from the first thread 100 is regarded as one data. For example, when data is transferred to the second thread 200 by postMessage three times, the number of transferred data is three.

When the number of data flowing in from the first thread 100 is greater than the number of data flowing out to the call stack 230 , data is accumulated in the message queue 210 . If this situation continues, an overflow may occur in the message queue 210 due to a bottleneck. In this case, a situation in which the execution of the script of the second thread 200 is stopped may occur, or even a situation in which the entire web browser does not operate may occur.

Hereinafter, in FIGS. 3 and 4 , as a method for preventing overflow of the message queue 210 , the second thread 200 checks the current state of the message queue 210 and transmits the state information to the first thread 100 . Let's see how to do it.

The occurrence of an overflow in the message queue 210 is associated with the moment when data starts to accumulate in the message queue 210 . That is, when the number of data enqueued in the message queue 210 becomes greater than the number of data dequeued, a bottleneck may occur. Accordingly, the second thread 200 may check the current state of the message queue 210 by comparing the number of data enqueued in the message queue 210 with the number of data dequeued.

As a method of comparing the number of data enqueued in the message queue 210 with the number of data dequeued, there is a method of measuring the number of data enqueued or dequeued for a unit time. Alternatively, while a specific number of data is enqueued, the number of dequeued data may be measured and compared. That is, the second thread 200 may check the state of the message queue 210 by identifying the number of data so that a relative comparison between the number of enqueued data and the number of dequeued data is possible.

The second thread 200 may transmit the state of the message queue 210 in state data to the first thread 100 according to checking the state of the message queue 210 . In this case, the second thread 200 may transmit the state data to the first thread 100 only when the number of enqueued data is larger, or whenever data is received from the first thread 100 , in response thereto State data can be transmitted.

The state data may include information on the number of enqueued data and the number of dequeued data for a unit time. Alternatively, the information included in the state data may include the number of data dequeued while a specific number of data is enqueued. Alternatively, the second thread 200 compares the number of enqueued data with the number of dequeued data to measure the degree of risk of overflow in stages, and transmits the measured stage in state data to the first thread 100 . there is. For example, when the number of enqueued data is excessive, the second thread 200 puts a 'dangerous stage' in the state data, and when the number of enqueued data is small, the 'good stage' is stored in the state data of the first thread 100 can be sent to

A method of determining the state of the message queue 210 through the number of enqueued data 211 and dequeued data 212 for a unit time will be described in detail with reference to FIG. 3 . In FIG. 3 , the number of data 211 to be enqueued for a unit time is two and the number of data 212 to be dequeued is one. This is a case in which the number of enqueued data 211 is greater than the number of dequeued data 212 and corresponds to a state in which there is a risk of overflow in the message queue 210 . Accordingly, the second thread 200 transmits state data including information on the number of enqueued data 211 and dequeued data 212 to the first thread 100 . Alternatively, the second thread 200 transmits state data including information of a 'dangerous stage'.

4 shows data 213 stored in a message queue 210 . Referring to an embodiment in which the state of the message queue 210 is checked by comparing the number of enqueued data 211 and dequeued data 212 in FIG. 3 , in FIG. 4 , data stored in the message queue 210 ( An embodiment of checking the status of the message queue 210 by the number of 213) will be described.

When the message queue 210 starts to store a number of messages exceeding the limit, an overflow may occur in the message queue 210 . Accordingly, the second thread 200 may check the state of the message queue 210 by determining the number of data stored in the message queue 210 at a specific time.

The method for the second thread 200 to determine the number of data stored in the message queue 210 at a specific point in time is to directly count and determine the number of data stored in the message queue 210 at a specific point in time. It may be a method or a method of identifying through a difference between the number of enqueued data and the number of dequeued data up to a specific time point.

The specific point in time at which the second thread 200 determines the number of data stored in the message queue 210 may be when data is received from the first thread 100 , or from the message queue 210 to the call stack 230 . It may be when data is dequeued, or it may be a time point that is repeated at a specific time interval.

The second thread 200 may transmit a status message including information on the number of identified data according to the checking of the status of the message queue 210 to the first thread 100 . In this case, the second thread 200 may transmit the state data to the first thread 100 only when the number of data stored in the message queue 210 exceeds a predetermined value, or data from the first thread 100 The state data may be transmitted whenever is received. Here, the constant value is a number determined to prevent overflow of the message queue 210 , and may be a value determined by a user or a value determined through learning when an overflow occurs in the message queue 210 .

The state data may include information on the number of data stored in the message queue 210 . Alternatively, the second thread 200 compares the number of data stored in the message queue 210 with a constant value, measures the degree of risk of overflow in stages, and transmits the measured stage in state data to the first thread 100 . can For example, the second thread 200 sets a 'dangerous stage' when the number of stored data very close to or exceeds a predetermined value, and a 'good stage' when the number of stored data falls below a predetermined value. It can be stored in and transmitted to the first thread 100 .

A method of determining the state of the message queue 210 through the number of data stored in the message queue 210 at a specific point in time will be described in more detail with reference to FIG. 4 . In FIG. 4 , there are three pieces of data 213 stored in the message queue 210 . If the constant value is 7, the number of stored data 213 is less than the predetermined value, so the second thread 200 does not send the state data to the first thread 100, or the number of stored data 213 or ' It is possible to send status data containing the information 'good level'.

3 and 4, a method for the second thread 200 to check the status of the message queue 210 and a method for transferring the checked status to the first thread 100 have been described. Hereinafter, a method of controlling the number of data in the first thread 100 to prevent overflow of the message queue 210 and transmitting the adjusted data will be described in FIGS. 5 and 6 .

5 shows a method of transferring data from the first thread 100 to the second thread 200 in a conventional manner. In FIG. 5, data 1, data 2, and data 3 indicate data to be processed. 1, since the postMessage method handles a single variable, the data must be individually transmitted to the second thread 200 through postMessage. Therefore, the number of data delivered through postMessage and enqueued in the message queue 210 is three.

6 illustrates a method of generating intermediate data in the first thread 100 and transmitting the intermediate data to the second thread 200 according to an embodiment of the present invention. In the embodiment of FIG. 6 , the first thread 100 transmits data 1 , data 2 , and data 3 to the second thread 200 as in FIG. 5 , but in order to reduce the number of data enqueued in the message queue 210 . do. To this end, the first thread 100 generates intermediate data by merging data1, data2, and data3. When the first thread 100 transmits intermediate data instead of each data to the second thread 200 through postMessage, one piece of data is enqueued in the message queue 210 .

In the embodiment of FIG. 6 , intermediate data is generated by merging three pieces of data, but this is only a set number for convenience of description and does not limit the number of data to be merged to three. That is, the intermediate data may be generated by merging at least two pieces of data. Also, the number of data generating intermediate data in the first thread 100 is not fixed. The first thread 100 may generate intermediate data by changing the number of data as needed while performing a process.

3 and 4, the second thread 200 checks the status of the message queue 210 and transmits the status data to the first thread 100. In FIGS. 5 and 6, the first thread 100 ), we looked at how to generate intermediate data. Based on the above, a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 will be described in FIGS. 7 to 9 .

7 shows an embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .

The first thread 100 individually transmits data to be processed by the second thread 200 to the second thread 200 . That is, data is transferred to the second thread 200 using the conventional method illustrated in FIG. 5 without generating intermediate data.

The second thread 200 issues a status message when the number of data flowing into the message queue 210 is greater than the number of data flowing out or when the number of data stored in the message queue 210 exceeds a certain value. It is transmitted to the 1st thread (100).

The first thread 100 determines whether state data has been received from the second thread 200 . When the state data is not received, the first thread 100 individually transmits each data to be processed to the second thread 200 .

On the other hand, when receiving the status data, the first thread 100 merges a plurality of data in the manner shown in FIG. 6 instead of individually delivering the data to prevent overflow of the message queue 210 to Data is generated and the intermediate data is transmitted to the second thread 200 .

8 shows another embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .

The first thread 100 individually transmits each data to the second thread 200 without generating intermediate data.

The second thread 200 issues a status message when the number of data flowing into the message queue 210 is greater than the number of data flowing out or when the number of data stored in the message queue 210 exceeds a certain value. It is transmitted to the 1st thread (100).

The first thread 100 determines whether state data has been received from the second thread 200 . When the state data is not received, the first thread 100 individually transmits each data to be processed to the second thread 200 .

On the other hand, when the state data is received, the first thread 100 determines whether the occurrence of an overflow in the message queue 210 is predicted based on the state data. If overflow is not predicted, the first thread 100 individually transfers each data to be processed to the second thread 200 . When overflow is predicted, the first thread 100 generates intermediate data by merging a plurality of data to be individually transmitted, and transmits the intermediate data to the second thread 200 .

Whether the occurrence of overflow is predicted may be determined using information contained in the state data. When the information contained in the current state data is the number of data stored in the message queue 210, it is possible to predict whether an overflow occurs by determining whether the number exceeds a predetermined value. Alternatively, the occurrence of overflow may be predicted by relatively comparing the number of data stored in the message queue 210 contained in the information of the current state data based on the information contained in the past state data. That is, the relatively comparison method predicts whether overflow occurs according to the degree of change in the state of the message queue 210 through the previously received state data and the currently received state data.

9 shows another embodiment of a method in which the first thread 100 adjusts the number of data and transmits the data to the second thread 200 according to the state of the message queue 210 .

In the embodiment of FIG. 9 , unlike the embodiments of FIGS. 5 and 6 , the first thread 100 generates and transmits intermediate data to the second thread 200 from the beginning. This is a case where the intermediate data transfer requirements for preventing the overflow of the message queue 210 from past processes are learned and utilized empirically. When it is desired to perform the same or similar process as the previously performed process, the first thread 100 transfers the intermediate data to the second thread 200 from the beginning with reference to the number of data used to use the intermediate data in the previous process. can be created and delivered.

The second thread 200 issues a status message when the number of data flowing into the message queue 210 is greater than the number of data flowing out or when the number of data stored in the message queue 210 exceeds a certain value. It is transmitted to the 1st thread (100).

The first thread 100 determines whether state data has been received from the second thread 200 . When the first thread 100 does not receive the state data, it generates intermediate data in the same manner as in the previous step and transmits it to the second thread 200 .

On the other hand, when the state data is received, new intermediate data is generated with a larger number of data than the number of data used to generate the intermediate data in the previous step and transferred to the second thread 200 .

When data is transmitted from the first thread 100 to the second thread 200 through the method described above with reference to FIGS. 3 to 9 , an overflow in the message queue 210 can be prevented. Hereinafter, a case in which the above method is applied to image data processing will be described with reference to FIGS. 10 to 14 .

10 shows an apparatus for processing image data in a multi-threaded manner. The image processing apparatus of FIG. 10 processes image data using a first thread 100 and a second thread 200 on a web browser. The image data to be processed by the image processing apparatus is packetized image data received using the RTP protocol. In the image processing apparatus, the first thread 100 includes a websocket client 101 , an RTP client 102 , and a decoder module 103 , and the second thread 200 includes a depacketization module 201 . do.

Looking at the process of processing image data in the image processing apparatus, first, the first thread 100 receives image data transmitted from a server through a network. Modules used to receive image data in the first thread 100 are the WebSocket client 101 and the RTP client 102 .

The websocket client 101 is a module that establishes a websocket connection with the server. The WebSocket client 101 exchanges data with the server through a handshake. When a WebSocket connection is established between the WebSocket client 101 and the server, continuous data transmission/reception between the two is possible thereafter.

The RTP client 102 performs a function of supporting RTP communication with the server. The RTP client 102 provides a function capable of receiving data transmitted through the RTP protocol stably even in a web browser using HTTP.

The image data received by the WebSocket client 101 and the RTP client 102 is packetized data, and is referred to as RTP data. The first thread 100 transfers RTP data to the second thread 200 for depacketization.

The depacketizing module 201 of the second thread 200 depacketizes the received RTP data. The depacketized RTP data is called frame data. The second thread 200 transfers frame data restored from RTP data to the first thread 100 .

The decoder module 103 of the first thread 100 decodes the frame data received from the second thread 200 . In this case, the decoder module 103 may be a script implemented in JavaScript or a decoder embedded in a web browser. Also, decoding in the decoder module 103 may be performed in units of frames or in units of containers composed of a plurality of frames.

Through the above process, the image processing apparatus processes image data in multi-threads on the web browser. However, when the above process is performed using the conventional method of FIG. 5 , an overflow may occur in the message queue 210 due to the characteristics of the RTP protocol. For a detailed look at this, refer to FIG. 11 .

11 shows RTP data and frame data.

In general, the server transmits packetized image data. Packetization refers to dividing video data into appropriate lengths to facilitate transmission over a network, or assigning control information, such as addresses to be received, to each of video data by grouping them into appropriate lengths in case of short video data. RTP data is video data packetized according to the RTP protocol and consists of an RTP header and an RTP payload. A sequence number indicating the transmission order of RTP data, a time stamp for video synchronization, etc. are recorded in the RTP header, and segmented video data is loaded in the RTP payload.

In order to reproduce video data, RTP data must be depacketized. Depacketization is a concept opposite to packetization and refers to accumulating and assembling segmented image data to create a complete frame. For a plurality of RTP data, if payloads are assembled according to a sequence using header information, depacketized frame data can be generated.

Since the size of the packet based on the RTP protocol is limited, the image data is divided into a predetermined size (eg, 512 bytes) and transmitted. Therefore, a plurality of RTP data may be used to restore one frame data. In this case, excessive RTP data is transferred from the first thread 100 to the second thread 200 and an overflow may occur in the message queue 210 . can

Hereinafter, a method of processing image data in a multi-thread in a web browser without overflow will be described in FIGS. 12 to 14 .

12 shows an image processing apparatus according to an embodiment of the present invention. Compared to FIG. 10 , in FIG. 12 , RTPInt data, not RTP data, is transmitted from the first thread 100 to the second thread 200 . Prior to the description of the image processing apparatus of FIG. 12 , reference will be made to FIGS. 13 and 14 for description of RTPInt data.

13 shows RTP data, RTPInt data, and frame data. Since the RTP data and the frame data have been described with reference to FIG. 10, overlapping descriptions will be omitted.

In the present invention, intermediate data is used to prevent an excessive number of data from being transmitted from the first thread 100 to the second thread 200, and the intermediate data generated by assembling a plurality of RTP data is RTPInt data.

Header information of a plurality of RTP data is used to generate RTPInt data. The first thread 100 may generate RTPInt data by merging RTP data according to a sequence using header information of the RTP data.

14 shows message queues 210a and 210b in which RTP data and RTPInt data are enqueued.

When it is said that three RTP data are required to create frame data, when RTP data is individually transmitted from the first thread 100, three data must be enqueued in the message queue 210a to create frame data. On the other hand, when the first thread 100 merges the three RTP data to generate one RTPInt data, only one data is enqueued in the message queue 210b to create frame data.

That is, if the image data is processed using the intermediate data, it is possible to perform the same operation in the second thread 200 while enqueuing a smaller number of data in the message queue 210 . Therefore, according to the present invention, image data divided into a predetermined size according to the RTP protocol and transmitted can be depacketized in the second thread 200 without overflow of the message queue 210 .

Returning to FIG. 12 again, an operation process of the image processing apparatus will be described. In the image processing apparatus of FIG. 10 , if the first thread 100 transmits RTP data to the second thread 200 , in the image processing apparatus of FIG. 12 , the first thread 100 transmits the message queue 210 according to the state of the message queue 210 . RTP data or RTPInt data, which is intermediate data obtained by merging RTP data, is transferred to the second thread 200 . The second thread 200 depacketizes RTP data or RTPInt data to generate frame data and transmits it to the first thread 100 . The first thread 100 decodes the frame data received from the second thread 200 in the decoder module 103 to restore it.

According to the embodiment of FIG. 12 , it is possible to stably process image data in multi-threads on a web browser without overflow of the message queue 210 . Accordingly, the present invention can provide a useful working environment for processing image data requiring real-time performance such as live streaming.

In the embodiment of FIG. 12 , a case in which depacketization is performed in the second thread 200 has been described, but this is only for convenience of description, and the invention is not limited to the case of the embodiment. The system may be configured such that the second thread 200 performs reception functions such as the WebSocket client 101 and the RTP client 102 in addition to performing the depacketization operation.

In the above description, a device for processing data on a web browser includes a computing or processing device suitable for interacting with a server or other computing user devices via a network. For example, a device that processes data on a web browser is a desktop computer, mobile phone or smart phone, personal digital assistant (PDA), laptop computer, set-top box, digital media player, media dongle. ) and tablet computers.

Also, in the description above, web browsers include commonly known browsers installed on desktop computers or mobile phones, such as Google Chrome, Microsoft Explorer, Mozilla's Firefox, and Apple's Safari, as well as access resources or APIs of the web browser. It also includes software applications that have been developed separately. Also, the web browser includes various versions of browsers that support HTML5.

Also, in the above description, the first thread 100 and the second thread 200 in the multi-thread include a case where the first thread 100 is a web worker and the second thread 200 is a sub-worker created from the web worker. do.

Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. do.

100: first thread 101: websocket client
102: RTP client 103: decoder module
200: second thread 201: depacketization module
210, 210a, 210b: message queue 211: data to be enqueued
212: data to be dequeued 213: stored data
220: event loop 230: call stack

Claims (20)

A data processing method performed by at least one processor on a data processing device having a web browser using a thread supported by the web browser,
creating a first thread on the web browser;
generating, by the first thread, a second thread;
receiving, by the first thread, image data from an external server through a web socket connection;
transmitting, by the first thread, the received data to the second thread;
storing, by the second thread, the transferred data in a message queue of the second thread;
depacketizing the stored data by the second thread and transferring the restored frame obtained through the depacketization to the first thread; and
Decoding the transmitted frame by the first thread,
The transmitting of the received data to the second thread may include: the first thread adjusts the number of data to be transferred according to the current state of the message queue of the second thread, and transmits the adjusted data to the second thread. 2 comprising passing to the thread,
The step of transmitting the data whose number is adjusted is,
individually transferring, by the first thread, a plurality of data to the second thread; and
and transmitting, by the first thread, the intermediate data generated by merging the plurality of data after receiving the state data indicating the state of the message queue from the second thread. According to claim 1,
The method further comprising the step of the second thread checking the status of the message queue according to the data received from the first thread. 3. The method of claim 2,
The second thread compares the number of data enqueued in the message queue with the number of data dequeued from the message queue per unit time, and when the number of enqueued data is greater, sending the state data to the first thread Further comprising, a data processing method. 3. The method of claim 2,
The second thread further comprises the step of sending the state data to the first thread when the number of data stored in the message queue exceeds a predetermined value,
The constant value is a number determined so that an overflow does not occur in the message queue. 5. The method of claim 3 or 4, wherein the transmitting of the adjusted number of data comprises:
When the first thread receives the state data, merging a plurality of data instead of individually transmitting each data to generate intermediate data, and transmitting the intermediate data to the second thread , data processing methods. ◈Claim 6 was abandoned when paying the registration fee.◈ 3. The method of claim 2,
The method of claim 1, further comprising, by the second thread, transmitting the state data including the number of data stored in the message queue to the first thread. 3. The method of claim 2,
The method further comprising: transmitting, by the second thread, the state data including the number of data enqueued in the message queue and the number of data dequeued from the message queue per unit time to the first thread. The method of claim 6 or 7, wherein the transmitting of the adjusted number of data comprises:
When the first thread determines the number of messages loaded into the message queue from the status data, and determines that the number of loaded messages exceeds a predetermined value, instead of individually delivering each data, a plurality of data merging to generate intermediate data, and transmitting the intermediate data to the second thread,
The constant value is a number determined so that an overflow does not occur in the message queue. delete ◈Claim 10 was abandoned when paying the registration fee.◈ According to claim 1,
The web browser supports HTML5, data processing method. ◈Claim 11 was abandoned when paying the registration fee.◈ According to claim 1,
The second thread is configured of at least one or more threads. According to claim 1,
The first thread is a main thread that executes a user interface script in a web browser, and the second thread is a web worker thread generated from the main thread. The method of claim 1, wherein the step of delivering the received data and the step of delivering the adjusted data comprises:
The method of claim 1, wherein the first thread transmits data to the second thread as a postMessage. According to claim 1,
The first thread and the second thread execute a script written in JavaScript, a data processing method. According to claim 1,
The first thread is a web worker (web worker) generated from the main thread, the second thread is a sub worker (sub worker) generated from the web worker, data processing method. ◈Claim 16 was abandoned when paying the registration fee.◈ According to claim 1,
wherein the received data is packetized image data. delete delete delete According to claim 1,
The data processing method according to claim 1, wherein the data whose number is adjusted is data generated by merging the plurality of packetized image data using header information of the plurality of packetized image data.

Images