CN104834558A - Method and system for processing data - Google Patents
Method and system for processing data Download PDFInfo
- Publication number
- CN104834558A CN104834558A CN201510254277.1A CN201510254277A CN104834558A CN 104834558 A CN104834558 A CN 104834558A CN 201510254277 A CN201510254277 A CN 201510254277A CN 104834558 A CN104834558 A CN 104834558A
- Authority
- CN
- China
- Prior art keywords
- data
- terminal device
- client terminal
- data processing
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and system for processing data. The data can be processed efficiently and fast. The method comprises the steps that a server side device distributes the grouped data to multiple sets of client side devices; the client side devices receive the grouped data, and store the data; the client side devices read the stored data, then fragment the read data, and carry out single-thread processing on the fragmented data. According to the technical scheme, the problems of data receiving and processing in the process that a single kafka client side fails and updating loss caused by simultaneous updating of multiple threads are solved, data processing tasks can be manually controlled to be suspended and restarted, meanwhile, data loss caused by data overstock or processing failures is avoided, the data processing reliability is improved, and the accuracy and real-time performance of the data are guaranteed.
Description
Technical field
The present invention relates to technical field of the computer network, particularly a kind of method and system of data processing.
Background technology
Along with the widespread use of computer network and the fast development of e-commerce industry, the importance of increasing people's focused data, especially datumization is runed, to process and the analysis of data, while cutting operating costs, can also greatly improve operator to accuracy, the wherein focus paid close attention to especially of real time data.But for the process of magnanimity real time data, be a difficult point always.
In existing technology, the process for magnanimity real time data generally selects Distributed Message Queue, as kafka carries out transmitting-receiving and the management of message.Kafka Distributed Message Queue carries out message management with key word (topic), and each topic is divided into again multiple grouping (partition), using partition as data batchmove, the least unit of storage and reading.
After kafka Distributed Message Queue received server-side to data, the message of each topic is distributed on multiple partition uniformly.Kafka message queue client carries out data receiver according to topic and the partition num specified, therefore often organize the data that Kafka message queue client correspondingly can only receive different partition under a topic, and each partition only can be received by a Kafka message queue client.After Kafka message queue client receives data from data queue, data persistence is stored in database, database processes data according to corresponding service logic by the mode of multi-threading parallel process afterwards, if process successfully, more new data flag position, mark completes data processing; If data processing failure, then read untreatment data from database, re-start process.
The present inventor passes through in industry practical experience for many years, finds that prior art exists following shortcoming:
1, in use, the machine if application kafka client is surprisingly delayed, the data that this client correspondence will be caused to receive overstock, and cannot process, affect the real-time of data;
2, in existing scheme, data processing task can only be dispatched by thread oneself, cannot realize the time-out of data processing and restart;
3, because existing scheme is that multithreading processes data simultaneously, therefore many threads may process data simultaneously, just there will be system resource waste and upgrade the situation of losing, and cannot realize high concurrent, high performance processing messages;
4, in existing scheme, when data processing failure, need from database, read untreated message and re-start process, if it is untreated to overstock too much data, the I/O performance of database will decline greatly, thus causes treatment effeciency to reduce.
Summary of the invention
In view of this, the invention provides a kind of method and system of data processing, solve kafka client surprisingly delay machine time data receiver and process, multithreading upgrade simultaneously and cause upgrading the problem of losing, and the time-out of manual control data Processing tasks can be realized and restart, and it is mutual to reduce data message that is direct and database, thus improve reliability, function-stable and handling property is high.
For achieving the above object, according to an aspect of the present invention, a kind of method of data processing is provided.
The method of a kind of data processing of the present invention comprises: the data after grouping are distributed to many group client terminal devices by service terminal device; Client terminal device receives the data after described grouping, and these data is stored; Client terminal device reads the described data stored, and then to the data fragmentation read, then does single-threaded process respectively to every sheet data.
Alternatively, described service terminal device comprises kafka Message Queuing server end device and task coordinate device, and described client terminal device is provided with kafka message queue client terminal device.
Alternatively, the step that these data carry out storing is comprised: data described in persistent storage, and these data of buffer memory; And the step that client terminal device reads the described data stored comprises: client terminal device reads the described data of buffer memory.
Alternatively, client terminal device reads the described data stored, then to the data fragmentation read, again the step that every sheet data do single-threaded process is respectively comprised: data are read in the file distributing unit timing in client terminal device, and according to the service fields of data, burst is carried out to data, be then distributed to data processing units different in client terminal device; The data that the single-threaded process of each data processing unit receives.
Alternatively; described single-threaded process is done respectively to every sheet data time; the method also comprises: at needs time-out or when restarting data processing, and described file distributing unit will lock or unlock command is distributed to each described data processing unit to suspend or to restart data processing.
According to a further aspect in the invention, a kind of system of data processing is provided.
The system of a kind of data processing of the present invention comprises service terminal device and client terminal device, wherein: described service terminal device is used for the data after by grouping to the distribution of many group client terminal devices; Described client terminal device comprises: receiver module, for receiving the data after described grouping, and these data is stored; Processing module, for reading the described data of storage, then to the data fragmentation read, then does single-threaded process to every sheet data respectively.
Alternatively, described service terminal device comprises kafka Message Queuing server end device and task coordinate device, and is provided with kafka message queue client terminal device in described client terminal device.
Alternatively, described client terminal device is also for data described in persistent storage, and these data of buffer memory; And read the described data of buffer memory.
Alternatively, described processing module comprises file distributing unit and multiple data processing unit, wherein: described file distributing unit is used for timing and reads data, and carries out burst according to the service fields of data to data, is then distributed to different described data processing units; Described data processing unit is used for the data that single-threaded process receives.
Alternatively, described processing module also for: when needs suspend or when restarting data processing, described file distributing unit will lock or unlock command be distributed to each described data processing unit with time-out or restart data processing.
According to technical scheme of the present invention, the data after grouping to the distribution of many group client terminal devices, be doing so avoids the data that one group of client surprisingly delays caused by machine and overstock, thus ensure that real-time and the treatment effeciency of data by service terminal device; Client terminal device receives the data after described grouping, and these data are carried out persistent storage and buffer memory, read from buffer memory during each reading data, can avoid when data processing failure, a large amount of untreated data information exchange is carried out with hard disc data storehouse, database performance is caused to decline, thus release hard drive space, improve handling property; Client terminal device reads the described data stored, then to the data fragmentation read, again single-threaded process is done respectively to every sheet data, both improve processing speed like this, can avoid again occurring that multithreading processes data simultaneously and causes upgrading the problem of losing, thus save memory source, improve handling property; In addition, invention increases a memory database, wherein store lock instruction and unlock command, when needing to carry out the time-out of data processing and restarting, the instruction that user sends by the file distributing unit of processing module is distributed to each data processing unit and carries out corresponding operating, thus achieves the time-out of manual control data Processing tasks and restart.
Accompanying drawing explanation
Accompanying drawing is used for understanding the present invention better, does not form inappropriate limitation of the present invention.Wherein:
Fig. 1 is the schematic diagram of the key step of the method for a kind of data processing according to the embodiment of the present invention.
Fig. 2 is the detailed step process flow diagram of the method for a kind of data processing according to the embodiment of the present invention.
Fig. 3 is the kafka Distributed Message Queue principle of work schematic diagram of the method for a kind of data processing according to the embodiment of the present invention.
Fig. 4 is the main modular schematic diagram of the system of a kind of data processing according to the embodiment of the present invention.
Embodiment
Below in conjunction with accompanying drawing, one exemplary embodiment of the present invention is explained, comprising the various details of the embodiment of the present invention to help understanding, they should be thought it is only exemplary.Therefore, those of ordinary skill in the art will be appreciated that, can make various change and amendment, and can not deviate from scope and spirit of the present invention to the embodiments described herein.Equally, for clarity and conciseness, the description to known function and structure is eliminated in following description.
Fig. 1 is the schematic diagram of the key step of the method for a kind of data processing according to the embodiment of the present invention.As shown in Figure 1, the method for a kind of data processing of the embodiment of the present invention mainly comprises following step S11 to step S13.
Step S11: the data after grouping are distributed to many group client terminal devices by service terminal device.In conjunction with the preferred embodiments of the present invention, service terminal device is server-side device and the task coordinate device of kafka message queue, and the client terminal device that client terminal device is provided with kafka message queue stores with the reception carrying out data.
As everyone knows, can improve the efficiency of data processing to the classification reception of data and storage greatly, so, introducing Distributed Message Queue at data receiver with when storing is exactly a kind of good processing mode.At present, have at great majority in the system of task process and all can relate to message queue, separate each system of coupling by message queue, reduce the dependence between each system, improve stability, such as activemq, kafka etc.
Be described for kafka in the preferred embodiments of the present invention.Such as, when there being data request information, the real time data received can be classified according to different key words (topic) by kafka server end, sorted data can, by multiple groupings (partition) of being evenly distributed under this topic, take partition as distribution and the reading that minimum data unit carries out data.So, the mass data simultaneously received can be carried out detailed grouping according to himself different feature.Afterwards, often organize data be distributed to different clients carry out receptions storage.The reception of data and storing process will describe in detail later, repeat no more herein.
Service terminal device also includes task coordinate device, when occur client terminal device add or leave situation time, task coordinate device can starting load equalization algorithm, and the data balancing after kafka server-side device being divided into groups is assigned to corresponding client terminal device.Herein, task coordinate device can adopt Zookeeper or other task coordinate devices to realize.
In a preferred embodiment of the invention, problem is overstock in order to solve one group of kafka client data that surprisingly machine of delaying causes, take the structure of distributed multiple stage machine, introduce many group kafka client terminal devices, realize data receiver by the Group ID and introducing " duplicate removal " mechanism on-the-fly modifying kafka client terminal device.Due to when kafka client receive data time, need to specify topic, and the data of each topic can only one group of kafka client of a corresponding Group ID, therefore, the present invention is by introducing many group kafka clients, on-the-fly modify the Group ID often organizing kafka client, kafka client can be organized to make the data of each topic more and receive simultaneously.Often organizing kafka client to check data when receiving data, judging whether it was received by other kafka client, if other clients existing carry out receiving, discard processing is carried out to corresponding data, otherwise carry out reception storage.
After adopting scheme as above, even if the one group of kafka client surprisingly machine of delaying cannot receive data, the real-time reception of data is not affected yet, this scheme can solve the data that the single client machine of delaying causes and overstock, the problem that when can solve again service end propelling data in enormous quantities, client receiving efficiency is low.
Step S12: client terminal device receives the data after described grouping, and these data are stored.Wherein, the step that these data carry out storing is comprised: data described in persistent storage, and these data of buffer memory.
In a preferred embodiment of the invention, client terminal device receives after data, by described data respectively persistent storage to database and be cached to the very high database of a reading performance.After increasing this buffer memory, when carrying out data processing, directly from this buffer memory, read information, reading speed improves greatly, after data processing completes, data corresponding in this buffer memory is deleted, and result is fed back to database to carry out data mode change.When data processing failure, not feedback information, reads untreatment data and proceeds from buffer memory.So, need when data processing will greatly reduce with the data interaction amount of database, large quantity space can be discharged, thus ensure that the I/O performance of database, improve treatment effeciency, be conducive to the real-time ensureing data.
Step S13: client terminal device reads the described data stored, then to the data fragmentation read, then does single-threaded process respectively to every sheet data.
In a preferred embodiment of the invention, step S13 can carry out in accordance with the following steps:
Step S131: data are read in the file distributing unit timing in client terminal device, and carry out burst according to the service fields of data to data, are then distributed to data processing units different in client terminal device;
Step S132: the data that the single-threaded process of each data processing unit receives.
As can be seen from step S131 and step S132, when pending data are stored into after in buffer memory, file distributing unit in client terminal device reads data from buffer memory, and carries out burst according to the service fields of data to data, is then distributed to different data processing units.Herein, the service fields of data can be configured as required.
There is independent data queue each data processing unit inside, processed by independent thread, material is thus formed the mechanism of single-threaded sequence processing data successively, thus both ensure that handling property, turn avoid multithreading and upgrade the renewal loss and memory source waste problem that cause simultaneously.
After each data processing unit processes data, if data processing success, data mode corresponding for database can be changed to processed, delete the data in buffer memory simultaneously; If data processing failure, does not change state, continue next data in process data queue.File distributing unit regularly can read untreated data from buffer memory, and according to the service fields of data, burst is carried out to data, then carry out distributing again processing, thus ensure that the accuracy of data processing, avoid the loss of data that data overstock or data processing unsuccessfully causes.
In addition in embodiment of the present invention, also can increase a memory database and lock and unlock command for storage, to realize the time-out of manual control data process and to restart.At needs time-out or when restarting data processing, described file distributing unit will lock or unlock command is distributed to each described data processing unit to suspend or to restart data processing.Such as, user needs to suspend data processing to carry out other operation at the fixed time of every day, now just can set a timer module in this memory database and control.
When needs suspend data processing, the timer in memory database can send the instruction that locks to file distributing unit.Instruction can be distributed to all data processing units after file distributing unit receives the instruction that locks.After data processing unit receives instruction, first empty its memory queue, then wait for processing current data, its data queue becomes blocked state afterwards, waits for and continues to perform after unlocking.
When needs restart data processing, the timer in memory database can send a unlock command to file distributing unit.After file distributing unit receives unlock command, instruction is distributed to all data processing units.After data processing unit receives instruction, the data queue waking obstruction up continues data processing, thus reaches the effect restarted.
Fig. 2 is the detailed step process flow diagram of the method for a kind of data processing according to the embodiment of the present invention.
Process flow diagram as shown in Figure 2, after service terminal device receives real time data, divides into groups to data according to the message management of kafka message queue rule, and is distributed under the coordination of task conditioning unit and organizes client terminal device (step S21) more.Client terminal device receives data according to reception rule and is saved in database and buffer memory LevelDB, and buffer memory also can select the database (step S22) that other readwrite performances are high herein.Store successfully, file distributing unit timing reads data from buffer memory, and is distributed to different data processing units (step S23) after carrying out burst according to the service fields of setting to data.There is independent data queue each data processing unit inside, is undertaken processing (step S24) by independent thread.When data processing success, be processed by corresponding data Status Change in database, delete the data in buffer memory LevelDB simultaneously; If process unsuccessfully, do not change the data mode in database.File distributing unit regularly can read untreated data and again process from buffer memory LevelDB, ensure that the accuracy of data processing.
When needing to carry out the time-out of data processing and restarting, store in memory database Redis lock and unlock command can be distributed to each data processing unit by file distributing unit, to process accordingly.Here memory database also can select other high performance buffer memory or databases.
Fig. 3 is the kafka Distributed Message Queue principle of work schematic diagram of the method for a kind of data processing according to the embodiment of the present invention.
When there being data request information, the real time data received can be classified according to different key words (topic) by kafka server end, and sorted data can be evenly distributed in this type of multiple groupings (partition).
Kafka client carries out data receiver according to the topic specified, and often organizes the Group ID that kafka client has oneself, and the different kafka client correspondences often in group receive the data of corresponding partition.
As shown in Figure 3, the data of one group of same keyword (topic) are divided into multiple grouping (partition): grouping 1, grouping 2, grouping 3 ...Corresponding client terminal device has 3 groups, kafka customer end A, kafka customer end B and kafka client C.Under the scheduling of task conditioning unit Zookeeper, 3 groups of kafka clients receive the data of multiple grouping partition of this topic respectively simultaneously and check with duplicate removal data.When the kafka customer end A machine of delaying leaves, the load-balancing algorithm of Zookeeper starts, and the data of remaining grouping partition is re-started to distribute to kafka customer end B and kafka client C receives.
This scheme can solve the data that single group client machine of delaying causes and overstock, the problem that when can solve again service end propelling data in enormous quantities, client receiving efficiency is low.
Fig. 4 is the main modular schematic diagram of the system of a kind of data processing according to the embodiment of the present invention.As shown in Figure 4, the data handling system 4 in the embodiment of the present invention mainly comprises service terminal device 41 and client terminal device 42, and wherein, client terminal device 42 comprises again receiver module 421 and processing module 422.
Service terminal device 41 is for distributing the data after grouping to many group client terminal devices; Client terminal device 42 comprises: receiver module 421, for receiving the data after described grouping, and these data is stored; Processing module 422, for reading the described data of storage, then to the data fragmentation read, then does single-threaded process to every sheet data respectively.
Be introduced below in conjunction with the preferred embodiments of the present invention.Described service terminal device 41 comprises kafka Message Queuing server end device and task coordinate device, and described client terminal device is provided with kafka message queue client terminal device to carry out data receiver and storage.
Data persistence be stored into database after client terminal device receiver module 421 receives data and be cached in LevelDB.Afterwards, the file distributing unit of client terminal device processing module 422 and LevelDB carry out data communication, read the information in LevelDB and carry out burst according to the service fields of data to data, being then distributed in different data processing units and processing.In each data processing unit, there is independent queue, processed by independent thread.After data processing success, data mode corresponding for hard disc data storehouse can be changed to processed, delete the data in LevelDB simultaneously, if process unsuccessfully, do not change data mode.
Client terminal device processing module 422 can also be used for when needs suspend or restart data processing, and file distributing unit will lock or unlock command is distributed to each data processing unit to carry out corresponding operating.Wherein, lock and unlock command be kept in the memory database Redis of increase, it is read, access speed is all very fast.
When needs suspend data processing, the instruction that locks in memory database Redis is distributed to all data processing units by file distributing unit.After data processing unit receives instruction, first empty its memory queue, then wait for processing current data, its data queue becomes blocked state afterwards, waits for and continues to perform after unlocking.
When needs restart data processing, the unlock command in memory database Redis is distributed to all data processing units by file distributing unit.After data processing unit receives instruction, the data queue waking obstruction up continues data processing, thus reaches the effect restarted.
According to the technical scheme of the embodiment of the present invention, the data after grouping to the distribution of many group client terminal devices, be doing so avoids the data that one group of client surprisingly delays caused by machine and overstock, thus ensure that real-time and the treatment effeciency of data by service terminal device; Client terminal device receives the data after described grouping, and these data are carried out persistent storage and buffer memory, read from buffer memory during each reading data, can avoid when data processing failure, a large amount of untreated data information exchange is carried out with hard disc data storehouse, database performance is caused to decline, thus release hard drive space, improve handling property; Client terminal device reads the described data stored, then to the data fragmentation read, again single-threaded process is done respectively to every sheet data, both improve processing speed like this, can avoid again occurring that multithreading processes data simultaneously and causes upgrading the problem of losing, thus save memory source, improve handling property; In addition, invention increases a memory database, wherein store lock instruction and unlock command, when needing to carry out the time-out of data processing and restarting, the instruction that user sends by the file distributing unit of processing module is distributed to each data processing unit and carries out corresponding operating, thus achieves the time-out of manual control data Processing tasks and restart.
Above-mentioned embodiment, does not form limiting the scope of the invention.It is to be understood that depend on designing requirement and other factors, various amendment, combination, sub-portfolio can be there is and substitute in those skilled in the art.Any amendment done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within scope.
Claims (10)
1. a method for data processing, is characterized in that, comprising:
Data after grouping are distributed to many group client terminal devices by service terminal device;
Client terminal device receives the data after described grouping, and these data is stored;
Client terminal device reads the described data stored, and then to the data fragmentation read, then does single-threaded process respectively to every sheet data.
2. method according to claim 1, is characterized in that, described service terminal device comprises kafka Message Queuing server end device and task coordinate device, and described client terminal device is provided with kafka message queue client terminal device.
3. method according to claim 1, is characterized in that,
The step that these data carry out storing is comprised: data described in persistent storage, and these data of buffer memory;
And the step that client terminal device reads the described data stored comprises: client terminal device reads the described data of buffer memory.
4. method according to claim 1, is characterized in that, client terminal device reads the described data stored, and then to the data fragmentation read, then comprises the step that every sheet data do single-threaded process respectively:
Data are read in file distributing unit timing in client terminal device, and carry out burst according to the service fields of data to data, are then distributed to data processing units different in client terminal device;
The data that the single-threaded process of each data processing unit receives.
5. method according to claim 4, is characterized in that, described single-threaded process is done respectively to every sheet data time, the method also comprises:
At needs time-out or when restarting data processing, described file distributing unit will lock or unlock command is distributed to each described data processing unit to suspend or to restart data processing.
6. a system for data processing, is characterized in that, comprises service terminal device and client terminal device, wherein:
Described service terminal device is used for the data after by grouping to the distribution of many group client terminal devices;
Described client terminal device comprises:
These data for receiving the data after described grouping, and are stored by receiver module;
Processing module, for reading the described data of storage, then to the data fragmentation read, then does single-threaded process to every sheet data respectively.
7. system according to claim 6, is characterized in that, described service terminal device comprises kafka Message Queuing server end device and task coordinate device, and is provided with kafka message queue client terminal device in described client terminal device.
8. system according to claim 6, is characterized in that, described client terminal device is also for data described in persistent storage, and these data of buffer memory; And read the described data of buffer memory.
9. system according to claim 6, is characterized in that, described processing module comprises file distributing unit and multiple data processing unit, wherein:
Described file distributing unit is used for timing and reads data, and carries out burst according to the service fields of data to data, is then distributed to different described data processing units;
Described data processing unit is used for the data that single-threaded process receives.
10. system according to claim 9, is characterized in that, described processing module also for:
At needs time-out or when restarting data processing, described file distributing unit will lock or unlock command is distributed to each described data processing unit to suspend or to restart data processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510254277.1A CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510254277.1A CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104834558A true CN104834558A (en) | 2015-08-12 |
| CN104834558B CN104834558B (en) | 2018-06-01 |
Family
ID=53812466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510254277.1A Active CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104834558B (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183858A (en) * | 2015-09-10 | 2015-12-23 | 国家计算机网络与信息安全管理中心 | Distributed type data real-time deduplication method based on information arrays |
| CN106302385A (en) * | 2016-07-26 | 2017-01-04 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106775989A (en) * | 2016-12-31 | 2017-05-31 | 北京神州绿盟信息安全科技股份有限公司 | A kind of job control method and device |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN106844682A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN107273542A (en) * | 2017-07-06 | 2017-10-20 | 华泰证券股份有限公司 | High concurrent method of data synchronization and system |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN108984285A (en) * | 2018-06-28 | 2018-12-11 | 上海数据交易中心有限公司 | A kind of analysis of data collision flow point method and device, storage medium, terminal |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110647477A (en) * | 2018-06-27 | 2020-01-03 | 广州神马移动信息科技有限公司 | Data caching method, device, terminal and computer readable storage medium |
| CN110706019A (en) * | 2019-09-03 | 2020-01-17 | 苏宁云计算有限公司 | Effective price tag pushing method and device, computer equipment and storage medium |
| CN110765143A (en) * | 2019-10-10 | 2020-02-07 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and storage medium |
| CN111010403A (en) * | 2019-12-26 | 2020-04-14 | 紫光云(南京)数字技术有限公司 | Method and device for automatically generating SASL authentication file and computer storage medium |
| CN111104412A (en) * | 2018-10-25 | 2020-05-05 | 阿里巴巴集团控股有限公司 | Single-thread-based concurrency control method, device and system |
| CN111158610A (en) * | 2019-12-31 | 2020-05-15 | 苏州浪潮智能科技有限公司 | Method, device and equipment for synchronously setting cache acceleration and readable medium |
| CN112560660A (en) * | 2020-12-10 | 2021-03-26 | 杭州宇泛智能科技有限公司 | Face recognition system and preset method thereof |
| CN113760782A (en) * | 2021-08-23 | 2021-12-07 | 南京森根科技股份有限公司 | Dynamically adjustable annular cache system and control method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6134594A (en) * | 1997-10-28 | 2000-10-17 | Microsoft Corporation | Multi-user, multiple tier distributed application architecture with single-user access control of middle tier objects |
| CN102833336A (en) * | 2012-08-31 | 2012-12-19 | 河海大学 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
| CN103502943A (en) * | 2011-12-31 | 2014-01-08 | 华为技术有限公司 | Distributed task processing method, device and system based on message queue |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
| CN104506950A (en) * | 2014-12-29 | 2015-04-08 | 珠海全志科技股份有限公司 | Multithread download method and download device in network streaming media play, and download equipment |
-
2015
- 2015-05-19 CN CN201510254277.1A patent/CN104834558B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6134594A (en) * | 1997-10-28 | 2000-10-17 | Microsoft Corporation | Multi-user, multiple tier distributed application architecture with single-user access control of middle tier objects |
| CN103502943A (en) * | 2011-12-31 | 2014-01-08 | 华为技术有限公司 | Distributed task processing method, device and system based on message queue |
| CN102833336A (en) * | 2012-08-31 | 2012-12-19 | 河海大学 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
| CN104506950A (en) * | 2014-12-29 | 2015-04-08 | 珠海全志科技股份有限公司 | Multithread download method and download device in network streaming media play, and download equipment |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183858A (en) * | 2015-09-10 | 2015-12-23 | 国家计算机网络与信息安全管理中心 | Distributed type data real-time deduplication method based on information arrays |
| CN105183858B (en) * | 2015-09-10 | 2018-12-21 | 国家计算机网络与信息安全管理中心 | A kind of distributed data real-time repetition removal method based on message queue |
| CN106302385A (en) * | 2016-07-26 | 2017-01-04 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106302385B (en) * | 2016-07-26 | 2019-11-15 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN106775989A (en) * | 2016-12-31 | 2017-05-31 | 北京神州绿盟信息安全科技股份有限公司 | A kind of job control method and device |
| CN106844682B (en) * | 2017-01-25 | 2019-08-16 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN106844682A (en) * | 2017-01-25 | 2017-06-13 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN107273542A (en) * | 2017-07-06 | 2017-10-20 | 华泰证券股份有限公司 | High concurrent method of data synchronization and system |
| CN107273542B (en) * | 2017-07-06 | 2020-11-27 | 华泰证券股份有限公司 | High-concurrency data synchronization method and system |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110647477A (en) * | 2018-06-27 | 2020-01-03 | 广州神马移动信息科技有限公司 | Data caching method, device, terminal and computer readable storage medium |
| CN110647477B (en) * | 2018-06-27 | 2022-02-11 | 阿里巴巴（中国）有限公司 | Data caching method, device, terminal and computer readable storage medium |
| CN108984285B (en) * | 2018-06-28 | 2021-10-15 | 上海数据交易中心有限公司 | Data collision flow analysis method and device, storage medium and terminal |
| CN108984285A (en) * | 2018-06-28 | 2018-12-11 | 上海数据交易中心有限公司 | A kind of analysis of data collision flow point method and device, storage medium, terminal |
| CN111104412B (en) * | 2018-10-25 | 2023-05-30 | 阿里巴巴集团控股有限公司 | Concurrent control method, device and system based on single thread |
| CN111104412A (en) * | 2018-10-25 | 2020-05-05 | 阿里巴巴集团控股有限公司 | Single-thread-based concurrency control method, device and system |
| CN110706019A (en) * | 2019-09-03 | 2020-01-17 | 苏宁云计算有限公司 | Effective price tag pushing method and device, computer equipment and storage medium |
| CN110765143A (en) * | 2019-10-10 | 2020-02-07 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and storage medium |
| CN110765143B (en) * | 2019-10-10 | 2022-08-02 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and storage medium |
| CN111010403A (en) * | 2019-12-26 | 2020-04-14 | 紫光云(南京)数字技术有限公司 | Method and device for automatically generating SASL authentication file and computer storage medium |
| CN111158610A (en) * | 2019-12-31 | 2020-05-15 | 苏州浪潮智能科技有限公司 | Method, device and equipment for synchronously setting cache acceleration and readable medium |
| CN111158610B (en) * | 2019-12-31 | 2022-02-22 | 苏州浪潮智能科技有限公司 | Method, device and equipment for synchronously setting cache acceleration and readable medium |
| CN112560660A (en) * | 2020-12-10 | 2021-03-26 | 杭州宇泛智能科技有限公司 | Face recognition system and preset method thereof |
| CN113760782A (en) * | 2021-08-23 | 2021-12-07 | 南京森根科技股份有限公司 | Dynamically adjustable annular cache system and control method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104834558B (en) | 2018-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104834558A (en) | Method and system for processing data | |
| US20200327113A1 (en) | Data storage application programming interface | |
| US7680848B2 (en) | Reliable and scalable multi-tenant asynchronous processing | |
| US9563681B1 (en) | Archival data flow management | |
| CN107370667B (en) | Multithreading parallel processing method and device, readable medium and storage controller | |
| Xu et al. | Blending on-demand and spot instances to lower costs for in-memory storage | |
| US11157324B2 (en) | Partitioning for delayed queues in a distributed network | |
| US9250811B1 (en) | Data write caching for sequentially written media | |
| US8190564B2 (en) | Temporary session data storage | |
| US10075549B2 (en) | Optimizer module in high load client/server systems | |
| EP3163446B1 (en) | Data storage method and data storage management server | |
| CN106375241B (en) | Batch data processing method, front-end system, host and batch data processing system | |
| US9690813B2 (en) | Tunable hardware sort engine for performing composite sorting algorithms | |
| US20060259485A1 (en) | System and method for intelligent data caching | |
| EP1330907A2 (en) | Method and apparatus for real-time parallel delivery of segments of a large payload file | |
| US7895264B2 (en) | Storage cluster server network | |
| CN113391890A (en) | Task processing method, device and equipment and computer storage medium | |
| CN105119997A (en) | Data processing method of cloud computing system | |
| CN111597033A (en) | Task scheduling method and device | |
| US9578120B1 (en) | Messaging with key-value persistence | |
| CN114553959A (en) | Situation awareness-based cloud native service grid configuration on-demand issuing method and application | |
| Silva et al. | Controlling network latency in mixed hadoop clusters: Do we need active queue management? | |
| US9110823B2 (en) | Adaptive and prioritized replication scheduling in storage clusters | |
| Emerson et al. | An atomic-multicast service for scalable in-memory transaction systems | |
| US12032995B1 (en) | Asynchronous task queue configuration in a database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |