WO2016110263A1 - 高并发数据存储方法及装置 - Google Patents

高并发数据存储方法及装置 Download PDF

Info

Publication number
WO2016110263A1
WO2016110263A1 PCT/CN2016/070397 CN2016070397W WO2016110263A1 WO 2016110263 A1 WO2016110263 A1 WO 2016110263A1 CN 2016070397 W CN2016070397 W CN 2016070397W WO 2016110263 A1 WO2016110263 A1 WO 2016110263A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
high concurrent
concurrent data
queue
local
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/070397
Other languages
English (en)
French (fr)
Inventor
兰健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to RU2017128080A priority Critical patent/RU2666245C1/ru
Priority to JP2017554633A priority patent/JP6464284B2/ja
Priority to US15/542,088 priority patent/US11327945B2/en
Publication of WO2016110263A1 publication Critical patent/WO2016110263A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2308—Concurrency control
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2457—Query processing with adaptation to user needs
    • G06F16/24578—Query processing with adaptation to user needs using ranking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2471—Distributed queries
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/461—Saving or restoring of program or task context
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Definitions

  • Embodiments of the present invention relate to data processing technologies, and in particular, to a high concurrent data storage method and apparatus.
  • embodiments of the present invention provide a high concurrent data storage method and apparatus to improve storage efficiency of high concurrent data.
  • an embodiment of the present invention provides a high concurrent data storage method, where the method includes:
  • the high concurrent data is sliced according to the number of local queues
  • the embodiment of the present invention further provides a high concurrent data storage device, where the device includes:
  • a receiving module configured to receive high concurrent data sent by multiple clients
  • a first pushing module configured to push the high concurrent data to a primary data queue, and respond to the corresponding client
  • a consumption module configured to consume high concurrent data in the primary data queue by using multiple threads
  • a fragmentation module configured to slice the high concurrent data according to the number of local queues
  • a second pushing module configured to push the high concurrent data after the fragmentation to the local queue
  • a storage module that consumes high-concurrency data on a local queue and stores it in a database.
  • the high concurrent data in the primary data queue is consumed by using multiple threads, and the high concurrent data is fragmented according to the number of local queues, and the fragmentation is performed.
  • the high concurrent data on the local queue is consumed and stored in the database
  • the high concurrent data is cached through the main data queue and the local queue, and the asynchronous storage method is used for data storage, which alleviates The pressure to directly store high-concurrency data to the database, avoiding the problem of database downtime in high concurrent storage, and improving the storage efficiency of high concurrent data.
  • FIG. 1 is a flowchart of a high concurrent data storage method according to Embodiment 1 of the present invention.
  • FIG. 2 is a schematic structural diagram of a high concurrent data storage method according to Embodiment 2 of the present invention.
  • FIG. 3 is a flowchart of a high concurrent data storage method according to Embodiment 2 of the present invention.
  • FIG. 4 is a schematic diagram of a high concurrent data storage device according to Embodiment 3 of the present invention.
  • Embodiment 1 is a flowchart of a high concurrent data storage method according to Embodiment 1 of the present invention.
  • the present embodiment is applicable to the storage of high concurrent data.
  • the method may be performed by a server, and specifically includes the following steps:
  • Step 110 Receive high concurrent data sent by multiple clients.
  • high concurrency means that the amount of visits is relatively large at a certain moment.
  • the server receives requests from multiple clients, and many clients simultaneously request high concurrent data.
  • Step 120 Push the high concurrent data to the primary data queue and respond to the corresponding client.
  • the data queue adopts distributed MQ (Message Queue), which can support distributed expansion, and can also make the framework highly available, and can still have more objective performance in big data processing.
  • MQ Message Queue
  • ActiveMQ is the most popular and powerful open source message bus produced by Apache.
  • the server pushes the high concurrent data to the primary data queue and responds to the client that sends the high concurrent data.
  • the main data queue is the main ingress queue of all data. When the data is pushed to the queue, the result is returned immediately, and the response speed is improved.
  • Step 130 using high threads to consume high concurrent data in the primary data queue.
  • the multi-thread is allocated by the thread pool, and the thread pool provides multi-thread processing data and pushes the data to the corresponding local queue to improve the consumption speed of the message. You can also use the characteristics of MQ to persist data and prevent data loss.
  • the server quickly consumes high concurrent data in the primary data queue using multithreading allocated by the thread pool. The consumption of the high concurrent data in the primary data queue refers to moving the high concurrent data in the primary data queue out of the primary data queue.
  • Step 140 Fragment the high concurrent data according to the number of local queues.
  • the local queue is used to store high-concurrency data after fragmentation to prevent data loss after fragmentation.
  • the local queue can be scaled according to the amount of data of the high concurrent data, and the number of local queues can be expanded when the amount of data is large.
  • the server Based on the number of local queues, the server divides the high concurrent data that is removed from the primary data queue into the same number of segments as the local queue to store the fragmented high concurrent data in the local queue. Fragment processing of high concurrent data ensures that data processing is not repeated and the processing speed is improved.
  • the primary key feature of the data can be sliced, and the primary key of the high concurrent data is subjected to modulo calculation on the number of local queues, and the result is several.
  • the data is allocated to the local queue numbered a few; it can also be fragmented according to the time range, for example, the first 1 million data can be allocated to the first local queue, and the second 1 million data can be allocated. Go to the second local queue; of course, there are other sharding strategies.
  • step 150 the high concurrency data after the slice is pushed to the local queue.
  • the server pushes the high concurrency data after fragmentation to the corresponding local queue according to the fragmentation strategy of fragmenting the high concurrent data, and the local queue caches the high concurrent data.
  • step 160 the high concurrent data on the local queue is consumed and stored in the database.
  • the server consumes high concurrent data on the local queue, moving the high concurrent data on the local queue out of the local queue, and then storing the high concurrent data that was moved out of the local queue into the database.
  • the high concurrent data in this embodiment can be transmitted in a high-performance JSON format during the delivery process (for example, pushing high concurrent data to the main data queue, consumption data, etc.) to facilitate serialization and deserialization of data.
  • JSON JavaScript Object Notation
  • Serialization is a mechanism for processing object streams.
  • the object stream is to stream the contents of the object, and can read and write the streamed object.
  • Serialization is to solve the problem caused by the object stream reading and writing. of.
  • the high concurrent data in the primary data queue is consumed by using multiple threads, and the high concurrent data is fragmented according to the number of local queues, and the fragmentation is performed after fragmentation.
  • the high concurrent data is pushed to the local queue, consumes the high concurrent data on the local queue and stores it in the database, caches the high concurrent data through the main data queue and the local queue, and uses the asynchronous storage method for data storage, which alleviates the direct
  • the pressure on the database when storing high concurrent data to the database avoids the problem of database downtime in high concurrent storage and improves the storage efficiency of high concurrent data.
  • the high concurrent data is fragmented according to the number of local queues, and preferably includes:
  • the primary key of the high concurrent data is modulo the number of the local queues
  • the corresponding high concurrent data is divided into one piece according to the same modulo calculation result.
  • the primary key generated in the high concurrent data is subjected to modulo calculation on the number of the local queues, and the high concurrent data obtained by the same modulo calculation result is divided into one segment.
  • the high concurrent data after the fragmentation is pushed to the local queue, which specifically includes:
  • the high concurrency data after fragmentation is pushed to the corresponding numbered local queue according to the modulo calculation result.
  • each local queue has its own number.
  • the high concurrency data after the slice is pushed to the local queue with the same number as the modulo calculation result.
  • the high concurrent data on the local queue is consumed and stored in the database, preferably including:
  • the server uses the timing scheduling to periodically/immediate the high concurrent data on the local queue from the local queue.
  • the user can control the number of the captured data (moving out of the local queue) and the time interval for capturing the data according to the demand, and move the timing/immediately out of the local queue.
  • High concurrent data is stored in the database.
  • Timing scheduling can be used to configure the function of timing/instant processing of data, alleviating the storage pressure of the database.
  • the timing scheduling is implemented by Quartz.
  • Quartz is an open source job scheduling framework written entirely in Java.
  • FIG. 2 is a structural diagram of an architecture of a high concurrent data storage method according to Embodiment 2 of the present invention.
  • the main data queue is the main ingress queue of all data.
  • the Thread Pool provides multi-thread processing data and fragments the data. Push to the corresponding local queue to improve the consumption speed of the message. At the same time, it can also use the characteristics of MQ to persist data and prevent data loss.
  • the local queue is used to store data after data fragmentation to prevent data loss after fragmentation.
  • Work 1, Work 2, Work 3 Process data and store it in the database through timed/instant consumption data. You can control the amount of data captured and the time interval for capturing data.
  • FIG. 3 is a flowchart of a high concurrent data storage method according to Embodiment 2 of the present invention. As shown in FIG. 3, the high concurrent data storage method provided in this embodiment specifically includes the following steps:
  • Step 310 Push the high concurrent data to be stored to the primary data queue.
  • the high concurrent data to be stored is pushed to the main data queue.
  • the result is returned and the response client is responded.
  • step 320 the multi-thread consumes high concurrent data in the main data queue.
  • step 330 the high concurrent data is sliced and pushed to the corresponding local queue.
  • the primary key of the high concurrent data is subjected to modulo calculation, and the high concurrent data after the fragmentation is pushed to the local queue numbered as the modulo calculation result according to the modulo calculation result.
  • step 340 the high concurrent data in the local queue is periodically consumed.
  • Step 350 storing data to the database.
  • This embodiment is directed to a traditional relational database, which uses a primary data queue to cache high concurrent data, and uses a local queue to perform secondary data buffering on high concurrent data, thereby preventing data loss and improving the consumption speed of the primary data queue.
  • the storage efficiency of high concurrent data can be configured to process data in a timely/instantaneous manner, alleviating the pressure on the database and avoiding the problem of database downtime in the case of high concurrent storage.
  • the high concurrent data storage device provided in this embodiment is used to implement the high concurrent data storage method provided in Embodiment 1.
  • the high concurrent data storage device provided in this embodiment includes: The receiving module 410, the first pushing module 420, the consumption module 430, the fragmentation module 440, the second pushing module 450, and the storage module 460.
  • the receiving module 410 is configured to receive high concurrent data sent by multiple clients; the first pushing module 420 is configured to push the high concurrent data to the primary data queue, and respond to the corresponding client; the consumption module 430 is configured to: The high-concurrency data in the primary data queue is consumed by the multi-thread; the fragmentation module 440 is configured to slice the high concurrent data according to the number of local queues; and the second push module 450 is configured to use the high-concurrency after fragmentation.
  • the data is pushed to the local queue; the storage module 460 is configured to consume high concurrent data on the local queue and store it in the database.
  • the fragmentation module includes: a modulo submodule, configured to modulo the number of the local queues by using the primary key of the high concurrent data; and a fragmentation submodule for corresponding to the same modulo calculation result
  • the high concurrency data is divided into one piece.
  • the second pushing module is specifically configured to push the high concurrency data after the fragmentation into the local queue of the corresponding number according to the modulo calculation result.
  • the storage module includes: a consumption sub-module for consuming high-concurrency data on the local queue by using timing scheduling; and a storage sub-module for storing the consumed high concurrent data in the database.
  • the timing schedule is implemented by Quartz.
  • the receiving module receives the high concurrent data sent by the multiple clients; the first pushing module pushes the high concurrent data to the primary data queue, and responds to the corresponding client; the consumption module consumes the primary by using multiple threads.
  • High concurrency data is stored in the database.
  • the high-concurrency data is cached through the main data queue and the local queue, and the asynchronous storage method is used for data storage, which relieves the pressure on the database when the high concurrent data is directly stored in the database, and avoids the database in the case of high concurrent storage.
  • the problem of the machine improves the storage efficiency of high concurrent data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种高并发数据存储方法及装置。所述方法包括：接收多个客户端发送来的高并发数据（110）；将所述高并发数据推送到主数据队列，并响应相应的客户端（120）；利用多线程消费所述主数据队列中的高并发数据（130）；根据本地队列的数量将所述高并发数据进行分片（140）；将分片后的高并发数据推送到本地队列中（150）；消费本地队列上的高并发数据并存储到数据库中（160）。通过主数据队列和本地队列对高并发数据进行缓存，采用了异步存储方式进行数据存储，缓解了数据库的压力，避免了数据库在高并发存储情况下宕机的问题，提升了高并发数据的存储效率。

Description

高并发数据存储方法及装置 技术领域
本发明实施例涉及数据处理技术，尤其涉及一种高并发数据存储方法及装置。
背景技术
随着互联网信息的爆炸式增长，越来越多的互联网服务器将遭遇高并发、海量数据的环境，在这种环境下会面临高并发数据存储的问题。为了解决这个问题，Hadoop、MongoDB、Hbase等一些NoSql(Not Only SQL，非关系型数据库)语言得到了广泛的应用。
但是，针对一些老业务系统，更换新的存储方式所消耗的成本会比较大，因此仍然使用着传统的关系型数据库存储数据，如MySql，Oracle等。当系统面临高并发数据存储的时候，会导致数据库的链接太多，占用资源过多，使得存储效率下降，进而导致服务器宕机等问题。
发明内容
有鉴于此，本发明实施例提供一种高并发数据存储方法及装置，以提升高并发数据的存储效率。
第一方面，本发明实施例提供了一种高并发数据存储方法，所述方法包括：
接收多个客户端发送来的高并发数据；
将所述高并发数据推送到主数据队列，并响应相应的客户端；
利用多线程消费所述主数据队列中的高并发数据；
根据本地队列的数量将所述高并发数据进行分片；
将分片后的高并发数据推送到本地队列中；
消费本地队列上的高并发数据并存储到数据库中。
第二方面，本发明实施例还提供了一种高并发数据存储装置，所述装置包括：
接收模块，用于接收多个客户端发送来的高并发数据；
第一推送模块，用于将所述高并发数据推送到主数据队列，并响应相应的客户端；
消费模块，用于利用多线程消费所述主数据队列中的高并发数据；
分片模块，用于根据本地队列的数量将所述高并发数据进行分片；
第二推送模块，用于将分片后的高并发数据推送到本地队列中；
存储模块，用于消费本地队列上的高并发数据并存储到数据库中。
本发明实施例通过将所述高并发数据推送到主数据队列，利用多线程消费所述主数据队列中的高并发数据，根据本地队列的数量将所述高并发数据进行分片，将分片后的高并发数据推送到本地队列中，消费本地队列上的高并发数据并存储到数据库中，通过主数据队列和本地队列对高并发数据进行缓存，采用了异步存储方式进行数据存储，缓解了直接将高并发数据存储到数据库时带给数据库的压力，避免了数据库在高并发存储情况下宕机的问题，提升了高并发数据的存储效率。
附图说明
图1是本发明实施例一提供的高并发数据存储方法的流程图；
图2是本发明实施例二提供的高并发数据存储方法的架构设计图；
图3是本发明实施例二提供的高并发数据存储方法的流程图；
图4是本发明实施例三提供的高并发数据存储装置的示意图。
具体实施方式
下面结合附图和实施例对本发明作进一步的详细说明。可以理解 的是，此处所描述的具体实施例仅仅用于解释本发明，而非对本发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本发明相关的部分而非全部内容。
实施例一
图1是本发明实施例一提供的高并发数据存储方法的流程图，本实施例可适用于对高并发数据的存储，该方法可以由服务器来执行，具体包括如下步骤：
步骤110，接收多个客户端发送来的高并发数据。
其中，高并发是指在某一时刻访问量比较大。服务器接收多个客户端的请求，许多客户端同时请求形成高并发数据。
步骤120，将所述高并发数据推送到主数据队列，并响应相应的客户端。
其中，数据队列采用分布式MQ(Message Queue，消息队列)，可以支持分布式扩展，也可以使得框架具有高可用性，在大数据处理的时候仍可具有较为客观的性能。其中，ActiveMQ是Apache出品的，最流行的、能力强劲的开源消息总线。
服务器接收到高并发数据后，将所述高并发数据推送到主数据队列，并响应发送该高并发数据的客户端。其中，主数据队列是所有数据的主入口队列，当数据推送到该队列之后，立刻返回结果，提升响应速度。
步骤130，利用多线程消费所述主数据队列中的高并发数据。
其中，多线程由线程池进行分配，线程池提供多线程处理数据并将数据进行分片推送到对应的本地队列，提升消息的消费速度，同时 还能利用MQ的特性，持久化数据，防止数据的丢失。服务器利用线程池分配的多线程快速消费所述主数据队列中的高并发数据。其中，消费所述主数据队列中的高并发数据是指将主数据队列中的高并发数据移出该主数据队列。
步骤140，根据本地队列的数量将所述高并发数据进行分片。
其中，本地队列用于存放分片后的高并发数据，防止分片后的数据丢失。本地队列可以根据所述高并发数据的数据量进行横向扩展，当数据量大的时候可以扩展本地队列的数量。服务器根据本地队列的数量，将从主数据队列中移出的高并发数据分成与本地队列相同数量的片段，以将分片后的高并发数据存储到本地队列中。对高并发数据进行分片处理，保证数据处理不重复，且提升处理速度。本领域技术人员可以了解，对高并发数据进行分片有多种策略：可以根据数据的主键特征进行分片，将高并发数据的主键对本地队列的数量进行取模计算，得到的结果是几就将该数据分配到编号为几的本地队列中；也可以根据时间范围进行分片，例如：可以将前100万个数据分配到第一个本地队列中，将第二个100万个数据分配到第二个本地队列中；当然，还有其他分片策略。
步骤150，将分片后的高并发数据推送到本地队列中。
服务器根据对高并发数据进行分片的分片策略，将分片后的高并发数据推送到相应的本地队列中，由本地队列对高并发数据进行缓存。
步骤160，消费本地队列上的高并发数据并存储到数据库中。
服务器消费本地队列上的高并发数据，即将本地队列上的高并发数据移出本地队列，然后将移出本地队列的高并发数据存储到数据库中。
本实施例中的高并发数据在传递过程(例如：将高并发数据推送到主数据队列、消费数据等)中，可以采用高性能的JSON格式进行传递，方便数据的序列化和反序列化，使得数据的传递速度更快。其中，JSON(JavaScript Object Notation)是一种轻量级的数据交换格式，是基于JavaScript语法的子集，即数组和对象表示。序列化是一种用来处理对象流的机制，对象流是将对象的内容进行流化，可以对流化后的对象进行读写操作，序列化是为了解决对象流进行读写操作时引发的问题的。
本实施例通过将所述高并发数据推送到主数据队列，利用多线程消费所述主数据队列中的高并发数据，根据本地队列的数量将所述高并发数据进行分片，将分片后的高并发数据推送到本地队列中，消费本地队列上的高并发数据并存储到数据库中，通过主数据队列和本地队列对高并发数据进行缓存，采用了异步存储方式进行数据存储，缓解了直接将高并发数据存储到数据库时带给数据库的压力，避免了数据库在高并发存储情况下宕机的问题，提升了高并发数据的存储效率。
在上述技术方案的基础上，根据本地队列的数量将所述高并发数据进行分片，优选包括：
将所述高并发数据的主键对所述本地队列的数量进行取模；
根据相同的取模计算结果将对应的高并发数据分成一片。
首先将高并发数据中生成的主键对所述本地队列的数量进行取模计算，将得到相同取模计算结果的高并发数据分成一个片段。
在上述技术方案的基础上，将分片后的高并发数据推送到本地队列中，具体包括：
根据取模计算结果将分片后的高并发数据推送到相应编号的本地队列中。
其中，每个本地队列有自己的编号。将分片后的高并发数据推送到与取模计算结果相同编号的本地队列中。
在上述任一技术方案的基础上，消费本地队列上的高并发数据并存储到数据库中，优选包括：
利用定时调度消费本地队列上的高并发数据；
将消费的高并发数据存储到数据库中。
服务器利用定时调度将本地队列上的高并发数据定时/即时移出本地队列，用户可以根据需求控制抓取数据(移出本地队列)的数量和抓取数据的时间间隔，将定时/即时移出本地队列的高并发数据存储到数据库中。通过定时调度可以配置定时/即时处理数据的功能，缓解数据库的存储压力。优选的，所述定时调度由Quartz实现。其中，Quartz是一个完全由Java编写的开源作业调度框架。
实施例二
图2是本发明实施例二提供的高并发数据存储方法的架构设计图。如图2所示，主数据队列是所有数据的主入口队列，当数据推送到该队列之后，立刻返回结果，提升响应速度；线程池(Thread Pool)提供多线程处理数据并将数据进行分片推送到对应的本地队列，提升消息的消费速度，同时还能利用MQ的特性，持久化数据，防止数据的丢失；本地队列用于存放数据分片后的数据，防止分片后的数据丢失；工作(Work)1、工作2、工作3通过定时/即时消费数据，处理数据并存储到数据库中，可以自主控制抓取数据的数量和抓取数据的时间间隔。
图3是本发明实施例二提供的高并发数据存储方法的流程图。如图3所示，本实施例提供的高并发数据存储方法具体包括如下步骤：
步骤310，将待存储的高并发数据推送到主数据队列。
首先将待存储的高并发数据推送到主数据队列，当高并发数据推送到主数据队列后，返回结果，响应客户端。
步骤320，多线程消费主数据队列中的高并发数据。
利用线程池提供的多线程快速消费主数据队列中的高并发数据。
步骤330，将高并发数据进行分片并推送到对应的本地队列中。
根据本地队列的数量，将高并发数据的主键对本地队列的数量进行取模计算，按照取模计算结果将分片后的高并发数据推送到编号为取模计算结果的本地队列中。
步骤340，定时消费本地队列中的高并发数据。
利用Quartz实现定时/即时消费本地队列中的高并发数据。
步骤350，存储数据到数据库。
本实施例针对传统的关系型数据库，利用主数据队列对高并发数据进行缓存，利用本地队列对高并发数据进行二次数据缓存，可以防止数据丢失，也能够提升主数据队列的消费速度，提升了高并发数据的存储效率，可以配置定时/即时处理数据的功能，缓解了数据库的压力，避免了数据库在高并发存储情况下宕机的问题。
实施例三
图4是本发明实施例三提供的高并发数据存储装置的示意图。本实施例提供的高并发数据存储装置用于实现实施例一提供的高并发数据存储方法。如图4所示，本实施例提供的高并发数据存储装置包括： 接收模块410、第一推送模块420、消费模块430、分片模块440、第二推送模块450和存储模块460。
其中，接收模块410用于接收多个客户端发送来的高并发数据；第一推送模块420用于将所述高并发数据推送到主数据队列，并响应相应的客户端；消费模块430用于利用多线程消费所述主数据队列中的高并发数据；分片模块440用于根据本地队列的数量将所述高并发数据进行分片；第二推送模块450用于将分片后的高并发数据推送到本地队列中；存储模块460用于消费本地队列上的高并发数据并存储到数据库中。
优选的，所述分片模块包括：取模子模块，用于将所述高并发数据的主键对所述本地队列的数量进行取模；分片子模块，用于根据相同的取模计算结果将对应的高并发数据分成一片。优选的，所述第二推送模块具体用于根据取模计算结果将分片后的高并发数据推送到相应编号的本地队列中。
优选的，所述存储模块包括：消费子模块，用于利用定时调度消费本地队列上的高并发数据；存储子模块，用于将消费的高并发数据存储到数据库中。有选的，所述定时调度由Quartz实现。
本实施例通过接收模块接收多个客户端发送来的高并发数据；第一推送模块将所述高并发数据推送到主数据队列，并响应相应的客户端；消费模块利用多线程消费所述主数据队列中的高并发数据；分片模块根据本地队列的数量将所述高并发数据进行分片；第二推送模块将分片后的高并发数据推送到本地队列中；存储模块消费本地队列上的高并发数据并存储到数据库中。通过主数据队列和本地队列对高并发数据进行缓存，采用了异步存储方式进行数据存储，缓解了直接将高并发数据存储到数据库时带给数据库的压力，避免了数据库在高并发存储情况下宕机的问题，提升了高并发数据的存储效率。
注意，上述仅为本发明的较佳实施例及所运用技术原理。本领域技术人员会理解，本发明不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代而不会脱离本发明的保护范围。因此，虽然通过以上实施例对本发明进行了较为详细的说明，但是本发明不仅仅限于以上实施例，在不脱离本发明构思的情况下，还可以包括更多其他等效实施例，而本发明的范围由所附的权利要求范围决定。

Claims (10)

  1. 一种高并发数据存储方法，其特征在于，所述方法包括：
    接收多个客户端发送来的高并发数据；
    将所述高并发数据推送到主数据队列，并响应相应的客户端；
    利用多线程消费所述主数据队列中的高并发数据；
    根据本地队列的数量将所述高并发数据进行分片；
    将分片后的高并发数据推送到本地队列中；
    消费本地队列上的高并发数据并存储到数据库中。
  2. 根据权利要求1所述的方法，其特征在于，根据本地队列的数量将所述高并发数据进行分片，包括；
    将所述高并发数据的主键对所述本地队列的数量进行取模；
    根据相同的取模计算结果将对应的高并发数据分成一片。
  3. 根据权利要求2所述的方法，其特征在于，将分片后的高并发数据推送到本地队列中，具体包括：
    根据取模计算结果将分片后的高并发数据推送到相应编号的本地队列中。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，消费本地队列上的高并发数据并存储到数据库中，包括：
    利用定时调度消费本地队列上的高并发数据；
    将消费的高并发数据存储到数据库中。
  5. 根据权利要求4所述的方法，其特征在于，所述定时调度由Quartz实现。
  6. 一种高并发数据存储装置，其特征在于，所述装置包括：
    接收模块，用于接收多个客户端发送来的高并发数据；
    第一推送模块，用于将所述高并发数据推送到主数据队列，并响应相应的客户端；
    消费模块，用于利用多线程消费所述主数据队列中的高并发数据；
    分片模块，用于根据本地队列的数量将所述高并发数据进行分片；
    第二推送模块，用于将分片后的高并发数据推送到本地队列中；
    存储模块，用于消费本地队列上的高并发数据并存储到数据库中。
  7. 根据权利要求6所述的装置，其特征在于，所述分片模块包括：
    取模子模块，用于将所述高并发数据的主键对所述本地队列的数量进行取模；
    分片子模块，用于根据相同的取模计算结果将对应的高并发数据分成一片。
  8. 根据权利要求7所述的装置，其特征在于，所述第二推送模块具体用于根据取模计算结果将分片后的高并发数据推送到相应编号的本地队列中。
  9. 根据权利要求6-8任一项所述的装置，其特征在于，所述存储模块包括：
    消费子模块，用于利用定时调度消费本地队列上的高并发数据；
    存储子模块，用于将消费的高并发数据存储到数据库中。
  10. 根据权利要求9所述的装置，其特征在于，所述定时调度由Quartz实现。
PCT/CN2016/070397 2015-01-08 2016-01-07 高并发数据存储方法及装置 Ceased WO2016110263A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
RU2017128080A RU2666245C1 (ru) 2015-01-08 2016-01-07 Способ и устройство для сохранения данных с высокой степенью многопоточности
JP2017554633A JP6464284B2 (ja) 2015-01-08 2016-01-07 高同時実行性データの記憶方法および装置
US15/542,088 US11327945B2 (en) 2015-01-08 2016-01-07 Method and device for storing high-concurrency data

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510009667.2A CN104598563B (zh) 2015-01-08 2015-01-08 高并发数据存储方法及装置
CN201510009667.2 2015-01-08

Publications (1)

Publication Number Publication Date
WO2016110263A1 true WO2016110263A1 (zh) 2016-07-14

Family

ID=53124348

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/070397 Ceased WO2016110263A1 (zh) 2015-01-08 2016-01-07 高并发数据存储方法及装置

Country Status (5)

Country Link
US (1) US11327945B2 (zh)
JP (1) JP6464284B2 (zh)
CN (1) CN104598563B (zh)
RU (1) RU2666245C1 (zh)
WO (1) WO2016110263A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111327452A (zh) * 2018-12-17 2020-06-23 瞻博网络公司 使用消息总线的网络设备配置
CN112749174A (zh) * 2019-10-30 2021-05-04 中国移动通信集团安徽有限公司 高并发处理方法、装置、处理设备及计算机存储介质

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598563B (zh) * 2015-01-08 2018-09-04 北京京东尚科信息技术有限公司 高并发数据存储方法及装置
CN105468710A (zh) * 2015-11-19 2016-04-06 中国建设银行股份有限公司 应用于高并发场景的图片处理方法及系统
CN105786629A (zh) * 2016-02-02 2016-07-20 四川长虹电器股份有限公司 基于消息队列的数据处理方法
CN105760452B (zh) * 2016-02-04 2020-03-10 深圳市嘉力达实业有限公司 一种高并发海量数据的采集、处理和存储的方法及系统
CN108073444A (zh) * 2016-11-18 2018-05-25 北京京东尚科信息技术有限公司 向客户端推送用户数据的方法及系统
CN108664315A (zh) * 2017-03-30 2018-10-16 北京京东尚科信息技术有限公司 多线程动态处理业务的方法、电子设备和可读存储介质
CN109150929B (zh) * 2017-06-15 2021-11-12 北京京东尚科信息技术有限公司 高并发场景下的数据请求处理方法和装置
CN107241279A (zh) * 2017-06-22 2017-10-10 北京天德科技有限公司 一种基于多级缓冲队列的区块链交易限流方法
CN110019008A (zh) * 2017-11-03 2019-07-16 北京金山安全软件有限公司 一种数据存储方法及装置
CN107957903B (zh) * 2017-11-13 2020-11-10 中国平安财产保险股份有限公司 异步任务调度方法、服务器及存储介质
CN108595121B (zh) * 2018-04-17 2021-03-02 湖北文理学院 数据存储方法和装置
CN110515741A (zh) * 2018-05-21 2019-11-29 北京京东尚科信息技术有限公司 一种基于本地任务队列的降级处理方法和装置
CN109284335A (zh) * 2018-09-10 2019-01-29 郑州云海信息技术有限公司 一种跨数据库批量传导数据的方法与装置
CN109379605B (zh) * 2018-09-29 2020-10-16 武汉斗鱼网络科技有限公司 基于弹幕顺序的弹幕分发方法、装置、设备和存储介质
CN109874027A (zh) * 2019-03-11 2019-06-11 宸瑞普惠(广州)科技有限公司 一种低延时手术示教直播方法及其系统
CN110162401A (zh) * 2019-05-24 2019-08-23 广州中望龙腾软件股份有限公司 Dwg文件并行读取方法、电子设备和存储介质
CN112019689B (zh) * 2019-05-29 2025-03-25 北京奇虎科技有限公司 来电秀业务处理系统及方法
CN111629019B (zh) * 2019-08-13 2022-11-18 广州凡科互联网科技股份有限公司 一种异步处理大数据和高并发的方法
CN110704110B (zh) * 2019-09-30 2021-09-14 浪潮软件股份有限公司 一种提高系统高并发情况下的响应速度的方法和装置
CN111831458B (zh) * 2020-06-11 2024-04-26 武汉烽火技术服务有限公司 一种高并发高解耦数据处理方法及数据中台系统
CN112131286B (zh) * 2020-11-26 2021-03-02 畅捷通信息技术股份有限公司 一种基于时间序列的数据处理方法、装置及存储介质
CN113312386B (zh) * 2021-05-10 2022-06-24 四川新网银行股份有限公司 一种基于分布式消息的批量入库方法
CN115510020A (zh) * 2021-06-23 2022-12-23 比亚迪股份有限公司 一种数据归档方法、电子设备及可读存储介质
CN114238037B (zh) * 2022-02-28 2022-04-29 北京金堤科技有限公司 高并发服务请求结果存储方法和装置、以及电子设备和存储介质
CN119201457B (zh) * 2024-09-18 2025-04-04 湖南北斗赛格导航科技有限公司 基于自适应熵分割与交互式差分学习的多线程并发处理方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101739296A (zh) * 2009-12-28 2010-06-16 广州菲音信息科技有限公司 数据处理系统和方法
US20110302583A1 (en) * 2010-06-04 2011-12-08 Yale University Systems and methods for processing data
CN103024014A (zh) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 通过消息队列的海量数据分发处理的方法和系统
CN104102693A (zh) * 2014-06-19 2014-10-15 广州华多网络科技有限公司 对象处理方法和装置
CN104598563A (zh) * 2015-01-08 2015-05-06 北京京东尚科信息技术有限公司 高并发数据存储方法及装置

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6205122B1 (en) * 1998-07-21 2001-03-20 Mercury Interactive Corporation Automatic network topology analysis
US6321265B1 (en) * 1999-11-02 2001-11-20 Altavista Company System and method for enforcing politeness while scheduling downloads in a web crawler
US6922724B1 (en) * 2000-05-08 2005-07-26 Citrix Systems, Inc. Method and apparatus for managing server load
US7171613B1 (en) * 2000-10-30 2007-01-30 International Business Machines Corporation Web-based application for inbound message synchronization
US7234139B1 (en) * 2000-11-24 2007-06-19 Catharon Productions, Inc. Computer multi-tasking via virtual threading using an interpreter
US6892199B2 (en) * 2001-06-29 2005-05-10 Trendium, Inc. Saving burst data by using semi-merge sorting module
US6886041B2 (en) * 2001-10-05 2005-04-26 Bea Systems, Inc. System for application server messaging with multiple dispatch pools
US8266327B2 (en) * 2005-06-21 2012-09-11 Cisco Technology, Inc. Identity brokering in a network element
CN100407203C (zh) * 2005-12-30 2008-07-30 华为技术有限公司 一种海量数据的处理方法
US20070192431A1 (en) * 2006-02-10 2007-08-16 Sun Microsystems, Inc. Method and apparatus for service oriented architecture infrastructure switch
US7764678B2 (en) * 2006-10-10 2010-07-27 Oracle America, Inc. Routing based on dynamic classification rules
US8621573B2 (en) * 2007-08-28 2013-12-31 Cisco Technology, Inc. Highly scalable application network appliances with virtualized services
US20100332593A1 (en) * 2009-06-29 2010-12-30 Igor Barash Systems and methods for operating an anti-malware network on a cloud computing platform
US8880524B2 (en) * 2009-07-17 2014-11-04 Apple Inc. Scalable real time event stream processing
WO2012153376A1 (ja) * 2011-05-06 2012-11-15 株式会社日立製作所 計算機システム、及び、情報処理方法
US9436532B1 (en) * 2011-12-20 2016-09-06 Emc Corporation Method and system for implementing independent message queues by specific applications
CN102622426A (zh) * 2012-02-27 2012-08-01 杭州闪亮科技有限公司 数据库写入系统及方法
CN103699660B (zh) * 2013-12-26 2016-10-12 清华大学 一种大规模网络流式数据缓存写入的方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101739296A (zh) * 2009-12-28 2010-06-16 广州菲音信息科技有限公司 数据处理系统和方法
US20110302583A1 (en) * 2010-06-04 2011-12-08 Yale University Systems and methods for processing data
CN103024014A (zh) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 通过消息队列的海量数据分发处理的方法和系统
CN104102693A (zh) * 2014-06-19 2014-10-15 广州华多网络科技有限公司 对象处理方法和装置
CN104598563A (zh) * 2015-01-08 2015-05-06 北京京东尚科信息技术有限公司 高并发数据存储方法及装置

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111327452A (zh) * 2018-12-17 2020-06-23 瞻博网络公司 使用消息总线的网络设备配置
CN111327452B (zh) * 2018-12-17 2022-11-01 瞻博网络公司 使用消息总线的网络设备配置
CN112749174A (zh) * 2019-10-30 2021-05-04 中国移动通信集团安徽有限公司 高并发处理方法、装置、处理设备及计算机存储介质
CN112749174B (zh) * 2019-10-30 2024-05-10 中国移动通信集团安徽有限公司 高并发处理方法、装置、处理设备及计算机存储介质

Also Published As

Publication number Publication date
US20180276265A1 (en) 2018-09-27
CN104598563B (zh) 2018-09-04
CN104598563A (zh) 2015-05-06
US11327945B2 (en) 2022-05-10
JP2018507496A (ja) 2018-03-15
RU2666245C1 (ru) 2018-09-06
JP6464284B2 (ja) 2019-02-06

Similar Documents

Publication Publication Date Title
CN104598563B (zh) 高并发数据存储方法及装置
US10565112B2 (en) Relay consistent memory management in a multiple processor system
CN110502583B (zh) 分布式数据同步方法、装置、设备及可读存储介质
WO2019033605A1 (zh) 订单异步处理的方法、装置、存储介质及终端
CN104834558A (zh) 一种数据处理的方法及系统
TW202021313A (zh) 訊息佇列發佈與訂閱之同步方法及其系統
CN103198097A (zh) 一种基于分布式文件系统的海量地学数据并行处理方法
WO2018120705A1 (zh) 一种数据处理方法、装置及系统
CN103942101A (zh) 一种基于多核网络处理器的实时任务调度方法及系统
CN107025142A (zh) 一种线程间消息传递方法、装置及电子设备
CN104572505A (zh) 一种保证海量数据缓存最终一致性的系统及方法
CN105843933A (zh) 分布式内存列式数据库的索引建立方法
CN105786629A (zh) 基于消息队列的数据处理方法
US20160253216A1 (en) Ordering schemes for network and storage i/o requests for minimizing workload idle time and inter-workload interference
CN104408047A (zh) 一种基于nfs文件服务器的文本文件多机并行上传到hdfs方法
CN104376096B (zh) 基于缓冲区的异步更新的方法
CN103123575A (zh) 一种数据写入方法
WO2016202113A1 (zh) 一种队列管理方法、装置及存储介质
Geyer et al. Pipeline Group Optimization on Disaggregated Systems.
CN106201918B (zh) 一种基于大数据量和大规模缓存快速释放的方法和系统
Kim et al. Large scale image processing in real-time environments with Kafka
Wang et al. Vifusion: In-network tensor fusion for scalable video feature indexing
CN107025064B (zh) 一种低延迟高iops的数据访问方法
CN105162684A (zh) 群组通信系统以及方法
CN102881050B (zh) 票务处理方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16734923

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2017554633

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 15542088

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017128080

Country of ref document: RU

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 16734923

Country of ref document: EP

Kind code of ref document: A1