CN110020359B - Data processing method, device and storage medium applied to front end of webpage - Google Patents
Data processing method, device and storage medium applied to front end of webpage Download PDFInfo
- Publication number
- CN110020359B CN110020359B CN201711088531.0A CN201711088531A CN110020359B CN 110020359 B CN110020359 B CN 110020359B CN 201711088531 A CN201711088531 A CN 201711088531A CN 110020359 B CN110020359 B CN 110020359B
- Authority
- CN
- China
- Prior art keywords
- data
- thread
- conversion
- data conversion
- data blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000003860 storage Methods 0.000 title claims abstract description 12
- 238000006243 chemical reaction Methods 0.000 claims abstract description 255
- 238000012545 processing Methods 0.000 claims abstract description 54
- 238000000034 method Methods 0.000 claims abstract description 40
- 230000001174 ascending effect Effects 0.000 claims description 11
- 230000002159 abnormal effect Effects 0.000 claims description 9
- 238000012163 sequencing technique Methods 0.000 claims description 8
- 238000004064 recycling Methods 0.000 claims description 6
- 238000003825 pressing Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 19
- 230000006870 function Effects 0.000 description 30
- 238000010586 diagram Methods 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 239000000872 buffer Substances 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008571 general function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000005316 response function Methods 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/972—Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
Abstract
The application provides a data processing method, a device and a storage medium applied to the front end of a webpage. Wherein the method comprises the following steps: dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks; performing data conversion on the plurality of data blocks in parallel to obtain a plurality of converted data blocks; and sorting the converted data blocks according to the labels to obtain target data, wherein the target data is used for front end view display. Through the method and the device, data conversion can be completed rapidly, the data conversion efficiency is improved, and the problems of low processing efficiency and poor system performance of the prior art that the front end of the webpage serially processes a large amount of data from background service are solved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus applied to a front end of a web page, and a storage medium.
Background
In the prior art, the front end of a webpage (i.e. web front end) is realized based on an Html5 platform, an ember.js frame is adopted as a js object at the web front end, java service is operated at the back end to provide data, data subscription and acquisition are carried out at the two ends based on websocket long connection, a transmission object is Protocol Buffers (PB for short), serialization and reverse sequence of structured data are carried out based on PB, namely, the serialized byte array is received by the web front end, and the byte is required to be converted into the front end js object.
Specifically, as shown in fig. 1, the method for processing data from a background service in a serial manner at the front end of a web page in the prior art includes the following steps:
step S101: the web page invokes a background service interface.
The view of the web page comprises a service data list, and the data subscription is carried out by calling a background service interface through the web page.
Step S103: the data queue Array < byte >.
Optionally, after receiving the subscription request, the background service sends a data queue to the web front end, where the web front end receives a subscribed data queue Array < byte >, where the data queue includes a plurality of Array objects, and the plurality of Array objects are arranged in descending order of time field, for example, 3000 subscribed data queues are obtained in descending order of time field.
Step S105: the method mainly comprises the steps of converting each array object in an obtained data queue into js objects at the front end of a web, and in the conversion process, in order to keep the js data queues after conversion in order, generally adopting serial conversion of the array objects in the data queues one by one in the prior art.
Step S107: outputting the processed JS object data queue.
After the conversion is completed, caching the converted js data queue for view binding display.
Currently, the efficiency of converting data by the web front end is about 400 pieces/s, and if the obtained data queue includes 3000 pieces of data, the time required for performing the operation is more than 7 s.
From this, it can be seen that, in the prior art, the Html5 web front end application running on the common PC configuration machine processes data with an efficiency of about 400 pieces/s, serial conversion of data (i.e. piece-by-piece conversion) when processing a large amount of data (e.g. 3000 pieces) by adopting the above scheme, and a large amount of time is consumed in the conversion process of the data queue, for example, a process of converting 3000 bytes [ ] into 3000 js objects requires about 7s, and a time-consuming process of processing a large amount of data has poor performance and low processing efficiency.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The application provides a data processing method and device, which are used for solving the problems of low processing efficiency and poor system performance of serial processing of a large amount of data from background service at the front end of a webpage in the prior art.
In order to achieve the above objective, an embodiment of the present application discloses a data processing method applied to a front end of a web page, where the method includes: dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks; performing data conversion on the plurality of data blocks in parallel to obtain a plurality of converted data blocks; and sorting the converted data blocks according to the labels to obtain target data, wherein the target data is used for front end view display.
Further, dividing the source data acquired from the background service into a plurality of data blocks includes: dividing array objects in the source data into the plurality of data blocks according to the sequence of the array objects in the source data, wherein the number of the array objects contained in each data block is not more than a preset number, the preset number is determined based on the maximum number of data which can be converted in a unit time by a single data conversion thread, and each data conversion thread is used for carrying out data conversion on one data block.
Further, performing data conversion on the plurality of data blocks in parallel includes: invoking a data conversion thread which is consistent with the number of the data blocks; and carrying out data conversion on the plurality of data blocks through the called data conversion threads, wherein each data conversion thread is used for converting an array object in one data block into a front-end JS object one by one so as to carry out data conversion.
Further, invoking a data conversion thread corresponding to the number of data blocks includes: calling idle data conversion threads which are consistent with the number of the data blocks from a thread pool, wherein the thread pool is at least used for recording thread states of all the data conversion threads, and the thread states comprise: idle, working, and abnormal.
Further, the thread pool is at least further configured to create the data conversion thread and record a thread state of the data conversion thread, where the thread pool is configured to record the thread state of the data conversion thread as work when the data conversion thread is called; and when the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle.
Further, after performing data conversion on the plurality of data blocks by using the called data conversion thread, the method further includes: and if the data conversion thread finishes the data conversion corresponding to the data block, executing a callback function, wherein the callback function is at least used for recycling the data conversion thread, and returning the recycled data conversion thread to a thread pool.
Further, after obtaining the converted plurality of data blocks, the method further includes: pressing the converted data blocks into a data block queue through a callback function; according to the label, sorting the converted data blocks to obtain target data, wherein the target data comprises: setting the serial numbers of the data blocks in the data block queues according to the labels through the callback function; and carrying out ascending sort on the data blocks in the data block queue according to the numbers to obtain the target data.
In order to achieve the above object, an embodiment of the present application discloses a data processing device applied to a front end of a web page, where the device includes: a dividing unit for dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks; the parallel conversion unit is used for carrying out data conversion on the plurality of data blocks in parallel to obtain a plurality of converted data blocks; and the sorting unit is used for sorting the converted data blocks according to the labels to obtain target data, wherein the target data are used for front end view display.
Further, the dividing unit includes: and a dividing module, configured to divide array objects in the source data into the plurality of data blocks according to an order of the array objects in the source data, where each of the data blocks includes no more than a predetermined number of the array objects, where the predetermined number is determined based on a maximum number of data that can be converted in a unit time by a single data conversion thread, and each of the data conversion threads is configured to perform data conversion on one of the data blocks.
Further, the parallel conversion unit includes: the calling module is used for calling the data conversion threads which are consistent with the number of the data blocks; and the conversion module is used for carrying out data conversion on the plurality of data blocks through the called data conversion threads, wherein each data conversion thread is used for converting the array objects in one data block into the front-end JS objects one by one so as to carry out data conversion.
Further, the calling module includes: a calling sub-module, configured to call idle data conversion threads corresponding to the number of the data blocks from a thread pool, where the thread pool is at least used to record thread states of each data conversion thread, and the thread states include: idle, working, and abnormal.
Further, the thread pool is at least further configured to create the data conversion thread and record a thread state of the data conversion thread, where the thread pool is configured to record the thread state of the data conversion thread as work when the data conversion thread is called; and when the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle.
Further, the apparatus further includes: and the execution unit is used for executing a callback function after the data conversion of the plurality of data blocks is carried out through the called data conversion thread, if the data conversion thread finishes the data conversion of the data blocks, the callback function is at least used for recycling the data conversion thread, and the recycled data conversion thread is returned to the thread pool.
Further, the apparatus further includes: the pushing unit is used for pushing the converted data blocks into the data block queue through a callback function after the converted data blocks are obtained; the sorting unit includes: the setting module is used for setting the number of the data block in the data block queue according to the label through the callback function; and the sequencing module is used for sequencing the data blocks in the data block queue in an ascending order according to the numbers to obtain the target data.
In order to achieve the above object, an embodiment of the present application discloses a storage medium storing a program for executing the above-described data processing method applied to a front end of a web page.
By adopting the embodiment, the source data subscribed from the background service is divided into a plurality of data blocks, the data blocks are subjected to data conversion in parallel, and after the conversion, the converted data blocks are sequentially reordered to obtain the target data. In the embodiment, the array objects in the source data do not need to be converted one by one, and the conversion operation can be executed by the array objects in the data blocks at the same time, so that the efficiency of converting the data is improved; meanwhile, after the conversion is completed, the converted data blocks are reordered, so that the data order can be ensured. Through the embodiment, the data conversion can be completed quickly under the condition of ensuring the ordered data, the data conversion efficiency is improved, and the problems of low processing efficiency and poor system performance of the prior art that the front end of the webpage serially processes a large amount of data from background service are solved.
Drawings
The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a flow diagram of a prior art serial processing of data from a background service at the front end of a web page;
FIG. 2 is a schematic diagram of a network environment of a data processing method applied to a front end of a web page according to an embodiment of the present invention;
FIG. 3 is a flowchart of a data processing method applied to a front end of a web page according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the structure of a thread pool created by a data processing method applied at the front end of a web page according to an embodiment of the present invention;
FIG. 5 is a diagram showing the correspondence between the data formats and the processing steps according to an embodiment of the present invention;
FIG. 6 is a second flowchart of a data processing method applied to the front end of a web page according to an embodiment of the present invention; and
fig. 7 is a schematic structural view of a data updating apparatus for a distributed index service engine according to an embodiment of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings.
First, the meaning of terms related to the present application is explained as follows:
ember. Js: is a JavaScript MVC framework. Compared with other JavaScript frames, the JavaScript frame has a more compact modularized frame, is a rich template system, and has a composite view and UI binding. The UI is simply called User Interface (UI).
Protocol Buffers: is a lightweight and efficient structured data storage format that can be used for structured data serialization. It is well suited for use as a data storage or RPC data exchange format. The method can be used for language independent, platform independent and extensible serial structure data formats, namely PB (provider edge) in the fields of communication protocols, data storage and the like.
HTML5: the fifth major modification of the world wide web's core language, a standard generalized markup language, to which hypertext markup language (HTML) is applied, is now widely used in technological platforms for web front-end development.
HTML5 Web workbench (workbench for short in the embodiment of the present application): the HTML5 Web Worker opens up an extra thread for loading and running a specific JavaScript file by using the class of the Worker in the current JavaScript executing thread, and the new thread and the main thread of the JavaScript do not influence each other and block the execution; and providing an interface in the Web workbench for data exchange between the new thread and the JavaScript main thread: postMessage and onmessage events. The postMessage is a common function in the Windows API (application program interface) for placing a message into a message queue, and onmessage is a custom message mapping macro in the MFC, which functions to bind a custom message and a response function of the message.
Data model: the business data model obtained from background service is based on PB generated byte array byte, and each data record is a byte array byte. The Html5 web front-end obtains the data queue Array < byte >, javaScript objects that need to be converted into an email. Js, namely email. Object, targeted for various presentation uses at the web (i.e., web page) front-end.
init: a user-level process initiated by the kernel. After the kernel has been booted (loaded into memory, started running, and all device drivers and data structures have been initialized, etc.), the boot process is completed by starting a user-level program init. Init is always the first process (its process number is always 1).
In the description of the present application, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or an implicit indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. The meaning of "a plurality of" is two or more, unless specifically defined otherwise. The terms "comprising," "including," and similar terms are to be construed as open-ended terms, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment". Related definitions of other terms will be given in the description below.
The data processing method applied to the front end of the webpage can be applied to the network environment shown in fig. 2. As shown in fig. 2, the network environment includes a background service 10 and a web page front end 20, where the background service and the web page front end are connected through a network, and the network may be a computer network, such as a local area network, a metropolitan area network, a wide area network, and another example, a finite network, a wireless network, etc., which is not limited in this application. The background service can provide data for the front end of the webpage, and the front end of the webpage displays the data provided by the background service.
The array object corresponds to service data, each service data may be represented by a character array (also referred to as a data object), each character array may include a plurality of field attributes (for example, a data array object may include 100 field attributes), the service data mainly includes numeric class data and string class data, and the array object is adapted to any service, that is, not limited to a certain service, and is a general function and model.
The embodiment of the application is applied to an Internet environment, is based on an html5 technology framework system, is a front-end web application and operates on a common PC machine; browser-side hardware is popular, and a '4G memory, a CPU 4core i5.5Ghz and a browser Chrome v53+' can be used as a standard. The background service and the front end of the web page may both exist depending on the physical machine, and the background service and the front end of the web page are generally disposed on different machines, alternatively, the background service and the front end of the web page may also be disposed on one machine, which is not limited in this application.
Optionally, the front end of the web page (i.e. the web front end) is implemented based on an Html5 platform, the js object of the web front end adopts an emmber.js frame, the background service operates java service to provide data, two ends perform data subscription acquisition based on websocket long connection (connection for realizing full duplex communication between the browser and the server), the transmission object is Protocol Buffers (PB for short), serialization and reverse sequence of structured data are performed based on the PB, i.e. the web front end receives a serialized byte array byte [ ], and the byte [ ] needs to be converted into the js object of the front end.
Optionally, the data processing method applied to the front end of the web page provided by the application may be implemented through the following steps as shown in fig. 3:
step S301: the method comprises the steps of dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks.
Optionally, the front end of the web page may send a data subscription request to the background service, after the background service receives the data subscription request, acquire data corresponding to the data subscription request, and return the acquired data to the front end of the web page, where the front end of the web page may acquire source data from the background service.
The source data may be a data queue including a plurality of array objects, where the array objects in the data queue are arranged in descending order according to a time sequence. After receiving the subscribed data queue, the front end of the webpage divides the array object in the received data queue into a plurality of data blocks, and each data block at least comprises one array object.
Alternatively, since the array objects in the source data are sequential, in dividing the source data into a plurality of data blocks, the order of each data block may be determined based on the order of the array objects in the data block, and a tag indicating the order of the plurality of data blocks may be set, and the tag may be an alphanumeric or numeric number, such as 1,2,3, … …, for example, a, B, C, … …. The form of the label is not particularly limited in this application.
Step S303: and carrying out data conversion on the plurality of data blocks in parallel to obtain a plurality of converted data blocks.
Optionally, after the front end of the web page divides the received source data into a plurality of data blocks, performing data conversion on the array objects in the plurality of data blocks in parallel to obtain each converted data block. The conversion specifically refers to converting the structured array object into a front-end JS object.
Parallel processing in the computer field refers to a group of programs executing at independent and asynchronous speeds, and in this application, parallel processing refers to data conversion processing performed on a plurality of data blocks simultaneously.
It should be noted that the present invention is not limited to parallel execution of conversion operations for a plurality of data blocks at the same time. Simultaneous may refer herein to simultaneous initiation of data conversion processing for multiple data blocks. For example, if the system is free enough, the conversion operation may begin to be performed on multiple data blocks in parallel at the same time; if the system has a new idle resource, the system may perform parallel conversion operation on a part of data blocks, and in the process of performing parallel conversion operation on a part of data blocks, detect that the system has a new idle resource in real time, and if the system has a new idle resource, perform data conversion on one or more data blocks that have not yet started to perform conversion processing by using the new idle resource.
By executing data conversion operation on a plurality of data blocks in parallel, data conversion can be performed on a plurality of data (such as a plurality of array objects) at the same time, and processing efficiency is improved.
Step S305: and sorting the converted data blocks according to the labels to obtain target data, wherein the target data is used for front end view display.
Optionally, after completing data conversion of the plurality of data blocks, reordering the converted data blocks according to the sequence of the data blocks before conversion, and the sequence of the array objects in the obtained target data is consistent with the sequence of the array objects in the source data, so that the web front end can perform view display on the target data.
Through the steps, after the data conversion is completed, the data in the plurality of data blocks can be reordered to ensure the data order.
By adopting the embodiment, the source data subscribed from the background service is divided into a plurality of data blocks, the data blocks are subjected to data conversion in parallel, and after the conversion, the converted data blocks are sequentially reordered to obtain the target data. In the embodiment, the array objects in the source data do not need to be converted one by one, and the conversion operation can be executed by the array objects in the data blocks at the same time, so that the efficiency of converting the data is improved; meanwhile, after the conversion is completed, the converted data blocks are reordered, so that the data order can be ensured. Through the embodiment, the data conversion can be completed quickly under the condition of ensuring the ordered data, the data conversion efficiency is improved, and the problems of low processing efficiency and poor system performance of the prior art that the front end of the webpage serially processes a large amount of data from background service are solved.
By the aid of the method and the device, the efficiency of processing big data of the web front end in the html5 environment can be improved, and data ordering is guaranteed. In order to ensure consistency of the ordered data before and after processing at the same time, the present application further provides an embodiment, in which dividing the source data acquired from the background service into a plurality of data blocks includes: and dividing the array objects in the source data into the plurality of data blocks according to the sequence of the array objects in the source data, wherein the number of the array objects contained in each data block is not more than a preset number.
Optionally, the predetermined number is determined based on a maximum amount of data that can be converted per unit time by a single data conversion thread, each of the data conversion threads being configured to convert data for one of the data blocks. With this embodiment, the number of objects in each data block can be determined based on the maximum conversion capability (i.e., the maximum amount of data that can be converted in a unit time) of a single data conversion thread, so that the processing speeds of the respective data blocks can be equalized to improve the overall processing efficiency. Wherein the data conversion threads may perform data conversion operations in parallel.
It should be noted that each data conversion thread is loaded with a parallel processing data interface, so that a plurality of data conversion threads may perform conversion operations in parallel.
Alternatively, the number of array objects in each data block may be all or partially the same. For example, the number of array objects in the first (N-1) data blocks in the N data blocks is the same, and the number of array objects in the N data blocks is different from the number of array objects in the first (N-1) data blocks.
For example, the number of the optimal processing array objects of a single data conversion thread is 400, which is used as the basis of slicing, namely every 400 slices are used as a data block, and the last 400 slices are also used as a slice, for example 3000 slices are used as a total 8 blocks, the first 7 blocks are all 400 blocks, and the last block is only 200 blocks. The number of fields of each piece of data is the same, the types of the fields are the same, and the values of the fields can be different. The slicing and blocking actions are serial, are not time-consuming, and can call a slice slicing function of js Array, and source data is divided into a plurality of data blocks by using the slice slicing function. Each data slice is partitioned and handled by a separate thread.
Wherein the slice (start, [ end ]) slice function correlation attribute: the first parameter start represents the starting position, the second parameter end represents the next position of the ending position, and the length of the intercepted character string is the difference between the second parameter and the first parameter; if the parameter value is negative, the value is converted into positive value after the character string length is added; if the first parameter is equal to or greater than the second parameter, then the empty string is returned.
According to an alternative embodiment of the present application, performing data conversion on the plurality of data blocks in parallel includes: invoking a data conversion thread which is consistent with the number of the data blocks; and carrying out data conversion on the plurality of data blocks through called data conversion threads, wherein each data conversion thread is used for carrying out data conversion on one data block. Optionally, each data conversion thread is configured to convert an array object in one data block into a front-end JS object one by one, so as to perform data conversion.
Specifically, a thread pool can be constructed when the front end of the webpage is initialized, and after the front end of the webpage divides the received source data into a plurality of data blocks, a plurality of data conversion threads in the thread pool are called to perform data conversion on the plurality of data blocks in parallel through the thread pool management threads. Multiple data blocks may be processed in parallel by pre-created threads.
In an alternative embodiment, each of the data conversion threads performs data conversion on one of the data blocks includes: and converting the array objects in the corresponding data blocks into front-end JS objects one by one through the data conversion thread.
In the above embodiment, the array objects are serially converted in each data conversion thread according to the sequence of the array objects, and after conversion of the plurality of data blocks is completed, the data can be ensured to be ordered only by ordering the data blocks.
The application also provides an alternative embodiment, wherein the calling the data conversion thread which is consistent with the number of the data blocks comprises the following steps: calling idle data conversion threads which are consistent with the number of the data blocks from a thread pool, wherein the thread pool is at least used for recording thread states of all the data conversion threads, and the thread states comprise: idle, working, and abnormal. The data conversion thread can work orderly through the thread pool management thread, so that resource congestion is avoided.
Alternatively, different identifiers may be used to represent different thread states, e.g., "0" to identify idle, "1" to indicate work, and "2" to indicate exception.
Further optionally, the thread pool is at least further configured to create the data conversion thread, and record a thread state of the data conversion thread, where in a case where the data conversion thread is called, the thread pool is configured to record the thread state of the data conversion thread as work; and when the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle. The thread state is changed in time through the thread pool, so that the threads can be more reasonably distributed, the effective utilization rate of the threads is improved, and the system resources are reasonably utilized.
Optionally, a thread pool workerpool is constructed during initialization and used for managing threads such as production, access, recovery and the like; the number of threads generated is set by the external configuration. Alternatively, a thread pool can be built by initPool, the number of threads created being determined by configs. The number of threads to be arranged is not limited to 10, and may be modified.
Optionally, the created thread pool mainly includes 3 core interfaces: initializing a thread pool (including creation threads) interface, setting a thread to an idle interface and acquiring the idle thread interface.
Alternatively, a thread pool may be created by:
var wpool=new Array(configs.threadCount);
initializing a thread pool, creating threads, thread state status:0 (idle), 1 (working), 2 (abnormal)
*/
initPool: function() {
for(var i = 0; i < configs.threadCount; i++) {
var worker = {};
worker.work = new Worker("com_operator_worker.js");
worker.id = i;
worker.status = 0;
this.wpool.push(worker);
}
},
/**
* Setting the thread completing the work to be idle
*/
rebackWorker: function(id) {
var work=this.getWorkerById(id);
if(work === undefined) {
return false;
}
work.status=0;
return true;
},
/**
* Acquiring free threads from a thread pool
*/
getWorker: function() {
for(var i = 0; i < this.wpool.length; i++) {
if(this.wpool[i].status === 0) {
this.wpool[i].status = 1;
return this.wpool[i];
}
}
return null;
}
The core script in any thread (i.e. data conversion thread) is com_operator_worker.js, and the core script loads a core interface protoAlarmConverter for parallel processing data (i.e. converting data).
As described in detail below with reference to FIG. 4, in an implementation manner in which a thread pool manages data conversion threads, as shown in FIG. 4, the thread pool may include a plurality of data conversion threads, the system acquires a plurality of data conversion threads with idle states, and the acquired data conversion threads execute data conversion processing on a plurality of data blocks in parallel, and return to the thread pool after each data conversion thread completes data conversion. The thread pool records the state of the data conversion thread which calls to execute the data conversion operation as work, and the thread pool records the state of the returned data conversion thread as idle.
In the process of acquiring a plurality of data conversion threads, firstly, sequentially allocating, and returning the processed data conversion threads to a thread pool, wherein the reallocation is randomly allocated, and the random embodiment is that the first idle thread is found and returned.
The application also provides an embodiment, after the data conversion is performed on the plurality of data blocks through the called data conversion thread, the method further comprises: and if the data conversion thread finishes the data conversion corresponding to the data block, executing a callback function, wherein the callback function is at least used for recycling the data conversion thread, and returning the recycled data conversion thread to a thread pool.
The callback function can quickly return to the thread pool after the data conversion thread finishes the data conversion of one data block, so that the subsequent program can be used, and the effective utilization of resources is realized.
Further, after obtaining the converted plurality of data blocks, the method further includes: pressing the converted data blocks into a data block queue through a callback function; according to the label, sorting the converted data blocks to obtain target data, wherein the target data comprises: setting the serial numbers of the data blocks in the data block queues according to the labels through the callback function; and carrying out ascending sort on the data blocks in the data block queue according to the numbers to obtain the target data. In this embodiment, the data blocks are sorted in ascending order according to the numbers of the data blocks, and the sorting speed is high and the error rate is low.
An optional embodiment of the data processing method applied to the front end of the web page according to the present application is described in detail below with reference to fig. 5 and 6, and as shown in fig. 5 and 6, this embodiment may include the following steps:
step S601: a thread pool is created. The specific implementation manner is consistent with that of the corresponding embodiment, and will not be described herein.
Step S602: a thread is created from the configuration file.
Optionally, a data conversion thread corresponding to the thread number is created according to the thread number configured in the configuration file. The specific implementation manner is consistent with that of the corresponding embodiment, and will not be described herein.
Step S603: the front end of the web page obtains a data queue from the background service.
Optionally, the front end of the webpage invokes a background service interface on the webpage to acquire a data queue of the background service. The data queue is the source data in the above embodiment.
Step S604: and slicing the data queue to obtain a plurality of data blocks.
The specific implementation manner is consistent with that of the corresponding embodiment, and will not be described herein.
Step S605: the data conversion threads are obtained from the thread pool as needed. And if the number of the idle data conversion threads in the thread pool is smaller than the number of the data blocks, continuously detecting whether a new idle data conversion thread appears in the thread pool, and calling the new idle data conversion thread until the total number of the called data conversion threads is consistent with the number of the data blocks.
Step S606: the data conversion is performed on the plurality of data blocks asynchronously in parallel. The specific implementation manner is consistent with that of the corresponding embodiment, and will not be described herein.
Step S607: and sequencing the converted data blocks to obtain an ordered data queue.
As shown in fig. 6, the correspondence between the data format and the above steps is shown, and before the source data queue is fragmented/partitioned and numbered, the data in the source data queue is byte [ ] x, where x represents the ordering number of the array object. After the block numbering, a plurality of data blocks 1-M are obtained. After parallel processing of the data after the blocking, the processed data blocks are output unordered, and are ordered according to the blocking numbers, so as to obtain target data after data conversion, wherein the data format in the target data is as follows: jsObject (i.e., JS object).
Step S608: returning the thread to the thread pool.
The above embodiment will be described below taking 3000 pieces of data acquired from a background service by the front end of a web page as an example.
For example, 3000 pieces of data are obtained from a background service, and conversion of byte [ ] to js objects is required. The single web worker data conversion thread optimally processes 400 threads, and the number of threads required to be started is calculated to obtain enough threads from a thread pool to process data conversion, wherein the number of the thread pool is configured, and the number of the threads is fixed after initialization and cannot be changed. The change is the state of the thread itself: work, idle, abnormal. The algorithm is as follows:
/**
* Data conversion and thread allocation
* pbArray source data queues such as 3000 pieces, onceCount: single thread processing is optimal, for example 400 data
* Wiredcount: number of threads finally created
* callbackCount: the number of callbacks processed by the threads is equal to the wiredCount, namely, all threads are processed
*/
protoDataConvert: function(pbArray) {
var n = pbArray.length / this.onceCount;
this.callbackCount = 0;
this.threadCount = n;
for(var k = 0; k < n; k++) {
if(k + 1 > n) {
this.threadCount = k + 1;
} else {
this.threadCount = k;
}
var start = k * this.onceCount;
var end = (k + 1) * this.onceCount;
this.dataConvertWorker(pbArray.slice(start, end), k);
confole.info ('start thread' +k+ ', let it perform data conversion work');
}
}
each time a data conversion thread dataconvertworkbench is started, 400 pieces of data are processed, wherein 400 pieces of data are pbarray (start, end) from 3000 pieces of data, and k is the number of the data block.
For 3000 pieces of data obtained, 8 pieces are divided, the first 7 pieces are 400 pieces, and the last piece is only 200 pieces. The 8 data blocks are serially (sequentially) partitioned from an ordered 3000 queue, and each data block is numbered in turn, i.e., 1/2/3 … 7/8. The single site dataconvertworkbench is started as follows:
/**
* Single thread initiation of a work process
*/
dataConvertWorker: function(dataSliceArray, no) {
Obtaining free worker threads from a thread pool
var convwork = this.wpool.getWorker();
if(convwork === undefined) {
confole. Info (' unable to acquire thread, thread pool work is saturated-;
return;
}
convwork.work.postMessage({
data: dataSliceArray,
no: no,
wid: convwork.id
});
convwork.work.onmessage = function(ndata) {
Callback after processing by the// convwork thread
this.dataConverCallback(ndata);
};
confole.info ('start thread' +no+ ', and start working');
}
after any single thread finishes processing the data, a callback dataconverCallback is executed, and the processed data blocks need to be organized in the callback. When all asynchronous threads do not complete work completely, pushing the processed data blocks to a queue dataBlocks, and returning the recovery threads to a thread pool; when all threads finish working, recovering the last thread, pushing the last data block to a queue dataBlocks, and then carrying out ascending sort on the data block queue dataBlocks according to recorded partition numbers so as to achieve consistency with the data sequence before processing. The algorithm is as follows:
/**
* Callback processing for arbitrary translation threads
*/
dataConverCallback: function(data) {
this.callbackCount++;
if(this.callbackCount === this.threadCount) {
confole. Info ('callback processing number is consistent with the total number of parallel threads, i.e. all threads are asynchronously processed to complete');
this.dataBlocks.push(data);
this.dataBlocks.sort(function(a, b) {
if(a['no'] >= b['no']) {
return 1;
} else {
return -1;
}
});
this.wpool.rebackWorker(data.wid);
this.callbackCount = 0;
conf.info ('regress present worker thread, reset data chunk queue, and return ordered data queue');
return this.dataBlocks;
} else {
this.dataBlocks.push(data);
this.wpool.rebackWorker(data.wid);
confole. Info ('regress to the working thread, add new data block to the block queue; still other threads are working');
return null;
}
}
After all threads finish the work, that means that all the processed data blocks enter the dataBlocks queue, the data blocks in the dataBlocks queue need to be sorted in ascending order according to the number. In this example, 8 data blocks are serially (sequentially) divided from an ordered 3000 queue, each data block is numbered sequentially, i.e. 1/2/3 … 7/8, each thread maintains its own 400 processed sequences consistent with those before processing, then after all threads are processed, the block numbers are sorted in ascending order, and the final 3000 processed sequences in dataBlocks are consistent with 3000 sequences before processing.
The dataBlocks are data block queues, after the data conversion thread finishes data conversion on respective data blocks, the converted data blocks are pushed into (i.e. pushed into) the dataBlocks, all the data blocks in the queues are ordered through the dataBlocks, and after the ordering is finished or initialized, the dataBlocks are emptied.
In the above example, a thread pool is created, and the working, idle and abnormal states of the threads are maintained; obtaining 3000 pieces of data from a background service, partitioning the data, respectively obtaining an idle thread from a thread pool to be responsible for processing of one data partition, and finishing work of all threads, namely finishing conversion from data byte to js objects; returning all the completed threads to a thread pool, and then sequencing the data blocks output by each thread according to the data block numbers, and organizing the data blocks into a data queue with the same sequence as that before parallel processing; the whole process is completed within 1s, so that the big data processing under the environment of the Html5 can be ensured, and the processing efficiency of 3000 strips/s can be met.
In the above embodiment, by performing thread number calculation on big data, acquiring idle threads from a thread pool, partitioning the big data, performing parallel processing, returning threads to the thread pool after thread callback, maintaining a data block queue, sorting data according to partition numbers, and returning processed complete ordered data, a feasible scheme of performing parallel and asynchronous processing on a large amount of ordered data under an Html5 technical system can be adopted, so that the processing efficiency of ordered big data can be greatly improved by running on a common PC machine and using web front-end application of the Html5 technical system.
In order to achieve the above object, an embodiment of the present application discloses a data processing device applied to a front end of a web page, where the device includes: a dividing unit 71 for dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks; a parallel conversion unit 73, configured to perform data conversion on the plurality of data blocks in parallel, so as to obtain a plurality of converted data blocks; and a sorting unit 75, configured to sort the converted plurality of data blocks according to the labels, to obtain target data, where the target data is used for front end view display.
Further, the dividing unit includes: and the dividing module is used for dividing the array objects in the source data into the plurality of data blocks according to the sequence of the array objects in the source data, wherein the number of the array objects contained in each data block is not more than a preset number.
Further, the predetermined number is determined based on a maximum amount of data that can be converted per unit time by a single data conversion thread, each of the data conversion threads being configured to perform data conversion on one of the data blocks.
Further, the parallel conversion unit includes: the calling module is used for calling the data conversion threads which are consistent with the number of the data blocks; and the conversion module is used for carrying out data conversion on the plurality of data blocks through the called data conversion threads, wherein each data conversion thread is used for carrying out data conversion on one data block.
Further, the conversion module includes: and the conversion sub-module is used for converting the array objects in the corresponding data blocks into front-end JS objects one by one through the data conversion thread.
Further, the calling module includes: a calling sub-module, configured to call idle data conversion threads corresponding to the number of the data blocks from a thread pool, where the thread pool is at least used to record thread states of each data conversion thread, and the thread states include: idle, working, and abnormal.
Further, the thread pool is at least further configured to create the data conversion thread and record a thread state of the data conversion thread, where the thread pool is configured to record the thread state of the data conversion thread as work when the data conversion thread is called; and when the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle.
Further, the apparatus further includes: and the execution unit is used for executing a callback function after the data conversion of the plurality of data blocks is carried out through the called data conversion thread, if the data conversion thread finishes the data conversion of the data blocks, the callback function is at least used for recycling the data conversion thread, and the recycled data conversion thread is returned to the thread pool.
Further, the apparatus further includes: the pushing unit is used for pushing the converted data blocks into the data block queue through a callback function after the converted data blocks are obtained; the sorting unit includes: the setting module is used for setting the number of the data block in the data block queue according to the label through the callback function; and the sequencing module is used for sequencing the data blocks in the data block queue in an ascending order according to the numbers to obtain the target data.
In order to achieve the above object, an embodiment of the present application discloses a storage medium storing a program for executing the above-described data processing method applied to a front end of a web page.
By adopting the embodiment, the source data subscribed from the background service is divided into a plurality of data blocks, the data blocks are subjected to data conversion in parallel, and after the conversion, the converted data blocks are sequentially reordered to obtain the target data. In the embodiment, the array objects in the source data do not need to be converted one by one, and the conversion operation can be executed by the array objects in the data blocks at the same time, so that the efficiency of converting the data is improved; meanwhile, after the conversion is completed, the converted data blocks are reordered, so that the data order can be ensured. Through the embodiment, the data conversion can be completed quickly under the condition of ensuring the ordered data, the data conversion efficiency is improved, and the problems of low processing efficiency and poor system performance of the prior art that the front end of the webpage serially processes a large amount of data from background service are solved.
Through the embodiment, the thread pool web worker is established by utilizing the webpage thread pool web worker of the Html5 to manage the thread pool web worker, and the large data can be processed in parallel and asynchronously at the web front end of the Html5 without serial processing of one data, so that the effective utilization rate of the threads can be ensured under the condition of limited resources. By adopting the scheme of dividing the source data into pieces and blocks and then sorting the data blocks after parallel processing according to the blocks so as to orderly reorganize the data, the data processing efficiency can be improved, and the consistency of the data sequence before and after processing can be ensured.
It should be noted that the above-described device embodiments belong to preferred embodiments, and the units and modules involved are not necessarily essential to the present application.
For the foregoing method embodiments, for simplicity of description, all of them are described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently, depending on the application; further, those skilled in the art will also appreciate that the above-described method embodiments are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.
The present application also discloses a storage medium having recorded thereon a program for executing the above method. The storage media includes any mechanism for storing or transmitting information in a form readable by a computer (e.g., a computer). For example, storage media includes read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory media, electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other. For the device embodiments of the present application, the description is relatively simple as it is substantially similar to the method embodiments, with reference to the description of the method embodiments section. The apparatus and apparatus embodiments described above are merely illustrative, in that the modules illustrated as separate components may or may not be physically separate, may be located in one place, or may be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The foregoing has outlined some of the more detailed description of a method and apparatus for a distributed index service engine, wherein specific examples are provided herein to illustrate the principles and implementations of the present application, the description of the above examples being only for the purpose of aiding in the understanding of the method and core concepts of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (11)
1. A data processing method applied to a front end of a web page, comprising:
the method comprises the steps that a webpage front end sends a data subscription request to a background service, after the background service receives the data subscription request, data corresponding to the data subscription request is obtained, the obtained data is returned to the webpage front end, and then the webpage front end obtains source data from the background service; dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks;
performing data conversion on the array objects in the plurality of data blocks in parallel to obtain a plurality of converted data blocks, wherein the conversion specifically refers to converting the structured array objects into front-end JS objects;
pressing the converted data blocks into a data block queue through a callback function;
according to the label, sorting the converted data blocks to obtain target data, including: setting the number of the data block in the data block queue according to the label through the callback function; ascending sort is carried out on the data blocks in the data block queue according to the numbers, and the target data are obtained; wherein the target data is for front end view display;
Dividing source data acquired from a background service into a plurality of data blocks includes:
dividing array objects in the source data into a plurality of data blocks according to the sequence of the array objects in the source data;
wherein each of the data blocks contains no more than a predetermined number of the array objects, the predetermined number being determined based on a maximum amount of data that can be converted per unit time by a single data conversion thread, each of the data conversion threads being configured to perform data conversion on one of the data blocks.
2. The data processing method of claim 1, wherein converting the plurality of data blocks in parallel comprises:
invoking a data conversion thread which is consistent with the number of the data blocks;
and carrying out data conversion on the plurality of data blocks through the called data conversion threads, wherein each data conversion thread is used for converting an array object in one data block into a front-end JS object one by one so as to carry out data conversion.
3. The data processing method of claim 2, wherein invoking a data conversion thread that matches the number of data blocks comprises:
calling idle data conversion threads which are consistent with the number of the data blocks from a thread pool, wherein the thread pool is at least used for recording thread states of all the data conversion threads, and the thread states comprise: idle, working, and abnormal.
4. A data processing method according to claim 3, wherein the thread pool is further used at least for creating the data conversion thread and recording the thread state of the data conversion thread, wherein in case the data conversion thread is invoked, the thread pool is used for recording the thread state of the data conversion thread as work; and under the condition that the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle.
5. The data processing method of claim 2, wherein after data conversion of the plurality of data blocks by the invoked data conversion thread, the method further comprises:
and if the data conversion thread finishes the data conversion corresponding to the data block, executing a callback function, wherein the callback function is at least used for recycling the data conversion thread, and returning the recycled data conversion thread to a thread pool.
6. A data processing apparatus for application at the front end of a web page, comprising:
the front end of the webpage is used for sending a data subscription request to the background service and acquiring source data from the background service;
The background service is used for acquiring data corresponding to the data subscription request after receiving the data subscription request and returning the acquired data to the front end of the webpage;
a dividing unit for dividing source data acquired from a background service into a plurality of data blocks, wherein the plurality of data blocks are provided with labels for marking the sequence of the data blocks;
the parallel conversion unit is used for carrying out data conversion on the array objects in the plurality of data blocks in parallel to obtain a plurality of converted data blocks, wherein the conversion specifically refers to converting the structured array objects into front-end JS objects;
the pushing unit is used for pushing the converted data blocks into the data block queue through a callback function after the converted data blocks are obtained;
the sorting unit is configured to sort the converted multiple data blocks according to the tag, to obtain target data, and includes: setting the number of the data block in the data block queue according to the label through the callback function; ascending sort is carried out on the data blocks in the data block queue according to the numbers, and the target data are obtained; wherein the target data is for front end view display;
A segmentation module for dividing array objects in the source data into a plurality of data blocks according to the sequence of the array objects in the source data,
wherein each of the data blocks contains no more than a predetermined number of the array objects, the predetermined number being determined based on a maximum amount of data that can be converted per unit time by a single data conversion thread, each of the data conversion threads being configured to perform data conversion on one of the data blocks;
and, the sorting unit includes:
the setting module is used for setting the number of the data block in the data block queue according to the label through the callback function;
and the sequencing module is used for sequencing the data blocks in the data block queue in an ascending order according to the numbers to obtain the target data.
7. The data processing apparatus according to claim 6, wherein the parallel conversion unit includes:
the calling module is used for calling the data conversion threads which are consistent with the number of the data blocks;
and the conversion module is used for carrying out data conversion on the plurality of data blocks through the called data conversion threads, wherein each data conversion thread is used for converting the array objects in one data block into the front-end JS objects one by one so as to carry out data conversion.
8. The data processing apparatus of claim 7, wherein the calling module comprises:
a calling sub-module, configured to call idle data conversion threads corresponding to the number of the data blocks from a thread pool, where the thread pool is at least used to record thread states of each data conversion thread, and the thread states include:
idle, working, and abnormal.
9. The data processing apparatus of claim 8, wherein the thread pool is further configured to at least create the data conversion thread and record a thread state of the data conversion thread, wherein the thread pool is configured to record the thread state of the data conversion thread as working if the data conversion thread is invoked; and under the condition that the conversion operation of the data conversion thread is completed and the thread pool is returned, the thread pool is used for recording the thread state of the data conversion thread as idle.
10. The data processing apparatus of claim 7, wherein the apparatus further comprises:
and the execution unit is used for executing a callback function if the data conversion thread finishes the data conversion of the data blocks after the data conversion of the plurality of data blocks is carried out through the called data conversion thread, wherein the callback function is at least used for recycling the data conversion thread, and returning the recycled data conversion thread to a thread pool.
11. A storage medium storing a program for executing the data processing method applied to the front end of a web page according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711088531.0A CN110020359B (en) | 2017-11-08 | 2017-11-08 | Data processing method, device and storage medium applied to front end of webpage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711088531.0A CN110020359B (en) | 2017-11-08 | 2017-11-08 | Data processing method, device and storage medium applied to front end of webpage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110020359A CN110020359A (en) | 2019-07-16 |
| CN110020359B true CN110020359B (en) | 2024-04-05 |
Family
ID=67185946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711088531.0A Active CN110020359B (en) | 2017-11-08 | 2017-11-08 | Data processing method, device and storage medium applied to front end of webpage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110020359B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111158939A (en) * | 2019-12-31 | 2020-05-15 | 中消云（北京）物联网科技研究院有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| CN101621353A (en) * | 2008-07-03 | 2010-01-06 | 华为技术有限公司 | Method, device and system for random linear network encoding |
| CN101860533A (en) * | 2010-05-26 | 2010-10-13 | 卓望数码技术（深圳）有限公司 | Data transmission method based on C/S architecture browser and server |
| CN102736961A (en) * | 2011-03-11 | 2012-10-17 | 微软公司 | Backup and restore strategies for data deduplication |
| CN103167348A (en) * | 2011-12-13 | 2013-06-19 | 深圳长城开发科技股份有限公司 | Method for downloading update data of set top box software in the air |
| CN103218264A (en) * | 2013-03-26 | 2013-07-24 | 广东威创视讯科技股份有限公司 | Multi-thread finite state machine switching method and multi-thread finite state machine switching device based on thread pool |
| CN104166551A (en) * | 2014-08-14 | 2014-11-26 | 华为技术有限公司 | Application interface developing method and developing device |
| CN104216768A (en) * | 2014-09-22 | 2014-12-17 | 北京金山安全软件有限公司 | Data processing method and device |
| CN105224332A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | A kind of application program generates method and apparatus |
| CN106790022A (en) * | 2016-12-14 | 2017-05-31 | 福建天泉教育科技有限公司 | Communication means and its system based on many inquiry threads |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1990004236A1 (en) * | 1988-10-04 | 1990-04-19 | Gems Of Cambridge Limited | Improved data processing |
| JP2004234380A (en) * | 2003-01-30 | 2004-08-19 | Fujitsu Ltd | Table display switching method, text data conversion program, and tagging program |
| JP2005150843A (en) * | 2003-11-11 | 2005-06-09 | Canon Inc | Image data coding apparatus and method, computer program, and computer-readable storage medium |
| WO2007117298A2 (en) * | 2005-12-30 | 2007-10-18 | Public Display, Inc. | Event data translation system |
| US8301768B2 (en) * | 2007-12-20 | 2012-10-30 | Pottenger William M | Peer-to-peer indexing-based marketplace |
| US10747746B2 (en) * | 2013-04-30 | 2020-08-18 | Amazon Technologies, Inc. | Efficient read replicas |
| CN107027068B (en) * | 2016-02-01 | 2021-04-13 | 斑马智行网络(香港)有限公司 | Rendering method, decoding method, and method and device for playing multimedia data stream |
-
2017
- 2017-11-08 CN CN201711088531.0A patent/CN110020359B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| CN101621353A (en) * | 2008-07-03 | 2010-01-06 | 华为技术有限公司 | Method, device and system for random linear network encoding |
| CN101860533A (en) * | 2010-05-26 | 2010-10-13 | 卓望数码技术（深圳）有限公司 | Data transmission method based on C/S architecture browser and server |
| CN102736961A (en) * | 2011-03-11 | 2012-10-17 | 微软公司 | Backup and restore strategies for data deduplication |
| CN103167348A (en) * | 2011-12-13 | 2013-06-19 | 深圳长城开发科技股份有限公司 | Method for downloading update data of set top box software in the air |
| CN103218264A (en) * | 2013-03-26 | 2013-07-24 | 广东威创视讯科技股份有限公司 | Multi-thread finite state machine switching method and multi-thread finite state machine switching device based on thread pool |
| CN104166551A (en) * | 2014-08-14 | 2014-11-26 | 华为技术有限公司 | Application interface developing method and developing device |
| CN104216768A (en) * | 2014-09-22 | 2014-12-17 | 北京金山安全软件有限公司 | Data processing method and device |
| CN105224332A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | A kind of application program generates method and apparatus |
| CN106790022A (en) * | 2016-12-14 | 2017-05-31 | 福建天泉教育科技有限公司 | Communication means and its system based on many inquiry threads |
Non-Patent Citations (3)
| Title |
|---|
| Frame labeling and mapping for non-parallel voice conversion;Minghui Dong等;《2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP)》;361-365 * |
| 基于Wikidata和标签云的搜索算法研究;夏瑀登;《软件导刊》;第15卷(第8期);42-46 * |
| 用于数据交换的XML文档和关系数据库转换;杨甲森等;《计算机工程与设计》;第27卷(第5期);857-859 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110020359A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| CN106802826B (en) | Service processing method and device based on thread pool | |
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| CN106775948B (en) | Cloud task scheduling method and device based on priority | |
| CN111814959A (en) | Model training data processing method, device and system and storage medium | |
| CN113792240A (en) | Page loading method and device and electronic equipment | |
| CN115827250A (en) | Data storage method, device and equipment | |
| CN107204998B (en) | Method and device for processing data | |
| CN110020359B (en) | Data processing method, device and storage medium applied to front end of webpage | |
| CN113660231A (en) | Message parsing method, device, equipment and storage medium | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN111190704A (en) | Task classification processing method based on big data processing framework | |
| CN116069330A (en) | Middle and background template construction method, device and storage medium | |
| CN116126719A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN110502337A (en) | For the optimization system and method for shuffling the stage in Hadoop MapReduce | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN108804155A (en) | Profile acquisition method, device, terminal and computer readable storage medium | |
| CN114637499A (en) | Visualization component processing method, device, equipment and medium | |
| CN113204426A (en) | Task processing method of resource pool and related equipment | |
| CN107180107B (en) | Method and system for uploading selected files by mobile phone attachment under Ionic framework | |
| CN110955461A (en) | Processing method, device and system of computing task, server and storage medium | |
| CN111698109A (en) | Method and device for monitoring log | |
| CN113835852B (en) | Task data scheduling method and device | |
| CN116483841B (en) | Form data management method and device based on compact framework | |
| CN111611479B (en) | Data processing method and related device for network resource recommendation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |