WO2017173919A1 - 一种并发数据的缓存方法及结构 - Google Patents

一种并发数据的缓存方法及结构 Download PDF

Info

Publication number
WO2017173919A1
WO2017173919A1 PCT/CN2017/077486 CN2017077486W WO2017173919A1 WO 2017173919 A1 WO2017173919 A1 WO 2017173919A1 CN 2017077486 W CN2017077486 W CN 2017077486W WO 2017173919 A1 WO2017173919 A1 WO 2017173919A1
Authority
WO
WIPO (PCT)
Prior art keywords
cache
data
partition
write
state
Prior art date
Application number
PCT/CN2017/077486
Other languages
English (en)
French (fr)
Inventor
徐驰
Original Assignee
武汉数字派特科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 武汉数字派特科技有限公司 filed Critical 武汉数字派特科技有限公司
Publication of WO2017173919A1 publication Critical patent/WO2017173919A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0877—Cache access modes
    • G06F12/0884—Parallel mode, e.g. in parallel with main memory or CPU
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the invention relates to a data acquisition technology in the field of radiation detection, in particular to a cache method and device for high speed data acquisition.
  • each data channel In the high-speed data acquisition system of the prior art, each data channel generally has a separate FIFO (First Input First Ouput) for buffering channel data.
  • FIFO First Input First Ouput
  • FIG. 1 is a schematic diagram of data acquisition and processing flow in an all-digital PET device.
  • Each channel of the detector samples and encodes the received signal, and sends the data packet to the network in a specific format.
  • the effective events required for PET imaging are distributed across different detector channels, and the matching relationship between the two can be calibrated by the event sampling time.
  • the effective events and noise data from each detector channel are mixed together and serially distributed in the network transmission link.
  • the collection server receives the data, the effective event is filtered out based on the matching algorithm, and time and energy correction are performed.
  • the screening results are then converted to PET images according to a reconstruction algorithm.
  • data collection should be performed concurrently with compliance processing, real-time data screening, time correction, and energy correction to reduce the resource requirements for pre-reconstruction data storage.
  • compliance processing real-time data screening, time correction, and energy correction to reduce the resource requirements for pre-reconstruction data storage.
  • energy correction is reduced, image reconstruction cannot be achieved or, although it can be achieved, it seriously affects accuracy.
  • the object of the present invention is to provide a buffering method and structure for concurrent data, which can effectively solve the problems of high data acquisition rate, speed block, and long data processing time and slow speed, and is especially suitable for detector data collection in all-digital PET. .
  • the solution of the present invention is:
  • the invention discloses a method for caching concurrent data, comprising the steps of: setting a working state of a cache partition to any one of an idle state, a write lock state, a write full state, and a read lock state,
  • Step (B) When the data is read, (B1) real-time monitoring whether there is a cache partition in the full state: if there is a cache partition in the full state, (B11) the cache partition selected in the full state of the group is set to Reading the lock state; (B12) reading the cache data in the read lock state cache partition, and when determining that the read lock state cache partition is read, setting the cache to an idle state, and proceeding to step (B1) Continue reading the next set of cache partitions; if there is no cache partition full of status, go to step (B1) and continue reading the data.
  • the set of cache partitions is set to a full state for data reading.
  • Each cache partition is sequentially accessed in order to monitor whether there is a write lock or an idle state cache partition;
  • any method of traversing, the same sequence as the data write access, and the cache partition write completion active reporting is used to monitor whether there is a cache partition in the full state in real time;
  • the cache partition internally performs a data write operation by using a hierarchical management manner.
  • the cache partition includes a plurality of cache sectors of the same size, and the cache sectors are encoded one by one, and include multiple cache pages of the same size.
  • the size of each set of write data is the same as the size of the cache page.
  • determining whether the data is successfully written includes the following steps: each group of data is written into the cache partition.
  • the internal count of the cache sector is determined, the current cache sector code is determined, and the cache sector code is compared with the maximum code number of the cache sector: if the cache sector code is smaller than the cache partition The maximum number of encodings, to determine the next set of data can be successfully written in this cache partition, If the cache sector code is equal to the maximum number of codes of the cache partition, it is determined that the cache partition is full, and the process proceeds to step (A1), and the next set of data is written into the next cache partition;
  • the data is written into the cache sector according to an encoding sequence, and the cache pages are encoded one by one, and when the data is written to a cache sector, the data is sequentially written according to an encoding order of the cache page.
  • the data writing success determination includes the following steps: writing an externally sent data group to the write lock status cache partition: when determining that the write lock status cache partition is not full When the write succeeds, the data writing of the group of cache partitions is continued; when it is judged that the write lock state buffer partition is full or the external data is written, the group cache partition is set to the full state, and the process proceeds to the step ( A1) continue writing of the next set of external data;
  • the cache partition write full judgment of the write lock state includes the following steps: comparing the external data amount with the remaining space of the cache partition of the write lock state when writing externally sent data to the write lock state cache partition a size, if the amount of external data is greater than the remaining space of the cache partition of the write lock state, determining that the cache partition of the write lock state is full; or writing externally sent data to the cache partition of the write lock state, If the system reports an error, it is determined that the cache partition of the write lock state is full.
  • the method further includes the step of monitoring whether there is a read lock status cache partition in real time, and if so, directly transferring to the step (B12) to read the cache data in the read lock status cache partition; if not, Transfer to step (B1);
  • the data reading speed is greater than a data writing speed
  • the invention discloses a cache structure of concurrent data, which simultaneously writes and reads data, and includes a set of data writing threads, a set of data reading threads, a set of data cache modules and a partition control module;
  • the data cache module includes a plurality of sets of cache partitions
  • the partition control module is in communication with the data cache module to control an operating state of the plurality of cache partitions and an access sequence, and the partition control module is configured to set an operating state of each of the cache partitions to an idle state. Any one of a write lock state, a write full state, and a read lock state, support write lock only when the cache partition is in an idle state, and support data write when the cache partition is in a write lock state, only The read buffer is supported when the cache partition is in the full state, and the data read is supported when the cache partition is in the read lock state;
  • the data write thread and the data read thread are respectively connected to each cache partition via the partition control module, and the partition control module controls the data write thread and the data read thread to access the cache partition.
  • the ordering is such that the data writing thread and the data reading thread perform writing and reading of data according to the working state of the accessed cache partition.
  • Each set of cache partitions includes a control unit and a plurality of sets of cache sectors, the cache sectors are encoded one by one, and the control unit is communicatively coupled with the each set of cache sectors to control each set of cache partitions. The order in which data is written to the cache sectors;
  • each set of cache sectors includes a control component and a plurality of cache pages
  • the cache pages are coded one by one and the size of each set of write data is the same as the cache page
  • the control component and the cache The pages are respectively connected to the connection to control the order in which data is written to the cache page number in each set of cache sectors.
  • the partition control module includes a write lock determination unit and a read lock determination unit;
  • the write lock determination unit is communicatively coupled to the data cache module to control an access sequence, a write lock and a write lock lock of the cache partition in the data cache module, and the data write thread is communicatively coupled to the write lock determination unit to perform Describe a cache partition accessed by the access determination unit, and perform a data write operation according to an operation state of the currently accessed cache partition;
  • the read lock determination unit is communicatively coupled to the corresponding data cache module to control an access sequence, a read lock and a read lock lock of the cache partition in the data cache module, and the data read thread is communicably connected to the read lock determination unit Whether to read the cache data is determined according to the current state of the cache partition accessed.
  • the data caching module is a producer and consumer model.
  • the present invention also discloses a cache model of concurrent data, and includes at least two sets of cache structures of concurrent data according to claim 8.
  • the present invention has the beneficial effects that the simple concurrent data caching method and structure shown in the present invention can support higher network data rate by optimizing strict read and write locks for access to the cache partition. Data processing speed.
  • Figure 1 is a schematic diagram of the data acquisition and processing flow of all-digital PET
  • FIG. 2 is a schematic diagram of steps of a concurrent data caching method according to an embodiment of the present invention.
  • FIG. 3 is a flow chart of the data writing method of the concurrent data buffering method in the embodiment shown in FIG. 2;
  • FIG. 4 is a flow chart of the data read by the concurrent data buffer method in the embodiment shown in FIG. 2;
  • FIG. 5 is a schematic diagram of a concurrent data cache structure according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a cache sector in the embodiment shown in FIG. 5.
  • the present invention provides a method for caching concurrent data, comprising the steps of: setting a working state of a cache partition to any one of an idle state, a write lock state, a write full state, and a read lock state, wherein when a certain cache partition is all When the write or the external data is written in a certain period of time, the cache partition is set to the full state when no new data is written.
  • each group of data is written.
  • the process is as follows, (A) a set of data writes: (A1) real-time monitoring whether there is a write lock state cache partition: when it is determined that there is a write lock state cache partition, directly go to step (A2-2), write the set of data Enter the write lock status cache partition.
  • step (A2) monitors whether there is a cache partition in the idle state in real time: (A21) If there is a cache partition in the idle state, (A21-1) select a group of idle The cache partition of the state is set to the write lock state; (A21-2) writing the set of data to the cache partition of the write lock state: (A21-3) determining whether the data is successfully written: if the data is successfully written, Then, the data of the group is completed, and the writing of the group of data is ended. At the same time, the writing of the next group of data can be continued in step (A); if the writing of the data fails, the buffer currently in the state of the write locking state is indicated. If the partition is full and cannot be written to new data, the working state of the current cache partition is set to full state, and the process proceeds to step (A1) to continue writing the external data of the group. External data is written to other cache partitions;
  • step (A21-3) whether the data is successfully written can be determined by determining whether the selected write lock state cache partition is full.
  • the write lock state buffer partition When writing a data group sent externally to the cache partition of the write lock state, if it is judged that the write lock state buffer partition is not full, it indicates that there is enough space to write the current data, and when the set data is written, It is judged that the writing is successful; when it is judged that the selected write lock state buffer partition is full, there is not enough space to continue to write new data, indicating that the group data cannot be written to the above write lock state cache partition, and it is judged The write fails. At this time, the group cache partition needs to be set to full state, and the process proceeds to step (A1) to write the data of the group to other suitable cache partitions.
  • the cache partition write full judgment of the write lock state further includes the following steps: when writing externally sent data to the write lock state cache partition, first comparing the set data and the write lock state of the cache partition remaining space size, if the external data If the remaining space of the cache partition is greater than the write lock state, it is determined that the cache partition of the write lock state is full; if the external data is not greater than the remaining space of the cache partition of the write lock state, it is determined that the cache partition of the write lock state is not full, the data of the group Can continue Continue to write; can also take the initiative to report whether the write lock state of the cache partition is full, that is, when writing externally sent data to the write lock state of the cache partition, if the system actively reports an error, then determine the write lock state The cache partition is full.
  • the data reading includes the following steps: (B) when the data is read, based on a single cache partition, (B1) real-time monitoring whether there is a cache partition in the full state: if there is a cache partition in the full state (B11) selecting one of the cached partitions of the full state to be set to the read lock state; (B12) reading the cached data in the read lock state cache partition, when judging that the read lock state cache partition is read Setting the cache to the idle state, and proceeding to step (B1) to continue reading of the next group of cache partitions; if there is no cache partition of the full state, then proceeding to step (B1) to determine whether the next time is There is a full state cache partition to continue reading the data.
  • the set of cache partitions is placed in a full state for data reading.
  • the received external data may not be able to fill a set of data caches at the end of the data writing process in a certain period of time, or the last piece of data cannot be filled with a set of data caches.
  • the part of the data is read, and the cache partition that writes the part of the data is also placed in a full state.
  • the method further includes the step of monitoring whether there is a read lock state cache partition in real time, and if so, directly transferring to the step (B12) to read the cache data in the selected read lock state cache partition; if not, Go to step (B1) to determine if there is a cache partition in the full state to continue reading the data.
  • the cache method shown in the present invention sets the access order of the cache partition to better realize the reasonable management of data writing and reading.
  • each cache partition is sequentially accessed sequentially to monitor whether there is a write lock or an idle state cache partition, that is, the write thread arbitrarily sets a cache partition as a starting point when looking for a write lock or an idle state cache partition, and then According to the sequential loop access, in particular, after each cache partition is constructed and initialized, it is generally idle. Since there is only one set of write lock state cache partitions at the same time, when data is written, the write thread follows Start from the beginning The first order sequentially accesses each cache partition. First, the idle state cache of the starting point is set to the write lock state for data writing. When the cache partition of the starting point is full, the next cache partition of the starting point is set to write. In the locked state, the write thread directly transfers to the next cache partition to continue writing data until the cache partition is full, and the data is sequentially written in a loop.
  • the cache partition includes a plurality of cache sectors of the same size, and a cache fan.
  • the area is encoded one by one and includes a plurality of cache pages of the same size, and the size of each set of write data is set to be the same as the size of the cache page.
  • the next set of data is written into the second cache page of the first set of cache sectors, and is sequentially performed until the set of cache sectors is full, and the cache sector internally counts itself to determine the current cache sector. Encode and automatically write unwritten data or the next set of data to the first cache page of the second set of cache partitions (next cache sector) until all cached pages in the cache partition are written full.
  • the corresponding data write time buffer partition full judgment is as follows: compare the cache sector code and the maximum number of encoding sectors of the cache sector: if the cache sector code is smaller than the maximum number of buffer partitions, determine the next set of data can If the cache sector code is equal to the maximum number of encodings of the cache partition, it is judged that the cache partition is full, and the process proceeds to step (A1), and the next set of data is written into the next cache partition.
  • the data When data is read, the data can be read in the same order as the data write access, so that when the cache partition is full and set to the full state, the data read thread can access the first time.
  • the cache partition is further set to the read lock state and the data is read. This setting can save time and improve data reading efficiency.
  • the data read speed is greater than the data write speed setting.
  • the following describes the working process by applying the caching method shown in the present invention to the basic cache model (i.e., the scenario of a single producer and a single consumer).
  • the entire cache is divided into several cache partitions, and the access lock protection of the cache partition is realized by the producer and the consumer through a simple state lock.
  • Each cache partition is idle after it is built and initialized. Before writing data to the cache partition, first determine whether there is a cache partition in the idle state in real time, if there is a data write operation, set the cache partition state to the write lock state, and support writing only when the cache partition is idle. Locks and writes data to the set of cache partitions through the data write thread.
  • the cache partition When it is judged that the cache partition is full, the cache partition is set to a readable state (ie, full and unlocked), and on the other hand, other free partitions in the cache are searched.
  • the loop continues the data write operation. While the data write operation is being performed, the data read thread also accesses the cache partition for data read operation, and the data read thread accesses each cache partition in real time to determine whether there is a readable state cache partition, when the data read thread judges the current
  • the accessed cache partition becomes readable, before the data read thread reads data from the cache partition, it first needs to set the partition state to read lock state, and only supports read lock when the cache partition is readable. If an attempt to add a read or write lock to the cache partition fails, it means that the network receive thread and the data processing thread access location meet, and if the write lock related operation fails, the cache is full.
  • the cache method of the concurrent data shown in the present invention can also be used in a cache composed of a plurality of basic cache models. As described above, each basic cache model adopts a cache method of concurrent data to write and read data. , will not repeat them here.
  • the NETTY network framework defines an independent data receiving thread to act as a producer.
  • the custom worker thread acts as a consumer and is responsible for network packet parsing, event compliance calculations, and so on.
  • the access lock of the cache partition is implemented based on the CAS mechanism provided by the JAVA development environment.
  • the basic principle is to implement the state setting of the cache partition as an atomic operation. Specifically, accessing an access lock object is an atomic operation. An operation is atomic, and the higher layers of this operation cannot discover its internal implementation and structure.
  • the atomic operation can be a step or a multiple step, the order of which can not be disturbed, and can not be divided into only the execution part.
  • CAS refers to Compare and Set, which is the value of the check object. If a certain condition is met, the object is set to the new value. Using the CAS mechanism to access and update the value of an object without using a thread lock, the performance is better than the scheme using a mutex.
  • a single cache partition is used as the basic collection unit of the event screening sample, and its capacity can be flexibly configured to adjust the event loss rate due to data fragmentation.
  • the network packet parsing and conforming filtering links all introduce the Fork/Join parallel computing model provided by the JAVA development environment.
  • the capacity of each cache partition is the same, and its capacity can be flexibly configured. In theory, the larger the cache partition, the better, which can ensure less data loss, but the larger the cache partition, the higher the physical memory requirements. Therefore, the size of the cache partition size can be determined according to the size of the physical memory.
  • the principle of the Fork/Join parallel computing model is the Map/Reduce model, which divides and conquers a complex task and improves computational efficiency by using the current computer multi-core multi-threading technology. For example, the sum of N numbers is calculated, the N numbers are divided into M parts, and the tasks of the calculation are performed synchronously using M threads, and the M results are combined into one result.
  • the invention also discloses a cache structure of concurrent data, simultaneously writing and reading data, comprising a set of data writing threads 100, a set of data reading threads 200, a set of data cache modules 300 and a partition control Module 400.
  • the data cache module 300 includes a plurality of sets of cache partitions 310.
  • the partition control module 400 is in communication with the data cache module 300 for controlling the working state of each cache partition 310 in the data cache module 300, that is, the partition control module 400 divides each cache.
  • the operational state of the zone 310 is set to any one of an idle state, a write lock state, a full state, and a read lock state.
  • Write lock is supported only when the cache partition 310 is in an idle state, and data write thread 100 is allowed to write data when the work state of the cache partition 310 is write lock; read lock support is only supported when the cache partition 310 is full.
  • the partition control module 400 is further configured to control the accessed order of the respective cache partitions 310, the data write thread 100 and The data reading thread 200 is communicably connected to each of the cache partitions 310 via the partition control module 400, and the data write thread 100 and the data read thread 200 respectively access the plurality of sets of caches in the data cache module 300 according to the access order determined by the partition control module 400.
  • the partition is then written or read based on the state of the cache partition it is accessing.
  • the partition control module 400 performs the read and write operations separately, and includes a write lock judging unit 410 and a read lock judging unit 420, and only one set is set in all the cache partitions at the same time.
  • Write lock state cache partition and a set of read lock state cache partitions are included in the partition control module 400.
  • the write lock determination unit 410 is in communication with the data cache module 300 for controlling the access sequence of the cache partition 310 in the data cache module 300, the write lock (putting the cache partition in the idle state to the write lock state), and the write lock (write The full write lock state cache partition is set to full state).
  • the data write thread 100 is communicatively coupled to the write lock determination unit 410, which accesses each of the cache partitions 310 according to the access order determined by the write lock determination unit 410, and performs a data write operation according to the operating state of the currently accessed cache partition 310.
  • the write lock determination unit 410 adopts a custom traversal strategy to adjust the distribution of data in the cache module, that is, sequential traversal, and one of the cache partitions can be arbitrarily selected as Start points and control the cache partitions in the encoding order.
  • each cache partition is in an idle state, and the write lock determination unit 410 sets the cache partition in the idle state of the start point to the write lock state, and the data write thread 100 determines according to the write lock determination unit 410.
  • the access partition accesses the cache partition of the starting point. Since it is in the write lock state, data writing can be started.
  • the write lock judging unit 410 sets the cache partition of the starting point to the full state.
  • the idle state cache partition at the next code of the start point is set to the write lock state, and the data write thread 100 continues to access the next code buffer partition as determined by the write lock determination unit 410, and then performs a data write operation.
  • the read lock determination unit 420 is also in communication with the corresponding data cache module 300 for controlling the access sequence of the cache partition in the data cache module 300, the read lock (putting the cache partition in the full state to the read lock state), and the read lock lock. (The read cache partition is placed in an idle state), and the data read thread 200 is communicatively coupled to the read lock determination unit 420 to determine whether to read the cache data based on the current state of the accessed cache partition. While the data is being written, the data read thread 200 is also performing a data read operation, and the access order of the data read may be in the same order as the data write access.
  • the readout of the data controls the data read thread 200 to access each cache partition according to the access order determined by the write lock determination unit 410, so as to be set, when the cache partition is full and set to the full state. After that, the data reading thread 200 can access the cache partition in the first time, and the buffer partition is set to the read lock state and the data is read, which can further improve the data reading efficiency.
  • the read lock determination unit 420 may also take the traversal order, or fill the active report by each cache partition, and determine the access order of the data read thread 200 in the order of the read lock determination unit 420 and the cache partition report.
  • each group of caches The partition 310 is hierarchically managed, and includes a control unit 311 and a plurality of sets of cache sectors 312.
  • the cache sectors 312 are coded, and the control unit 311 is communicably connected to all the cache sectors 312 of the set of cache partitions 310. Used to control the order in which external data is written to the cache sector 312 in each set of cache partitions.
  • each set of cache sectors 312 includes a control component 312-1 and a plurality of cache pages 312-2.
  • the cache pages 312-2 are encoded one by one and the size of each set of write data is the same as that of the cache page 312-2.
  • Control component 312-1 and cache page 312-2 respectively lead to a connection to control the order in which data is written to the cache page number in each set of cache sectors.
  • the control unit 311 controls the internal processing of the cache partition to be sequentially written.
  • the set of data is sequentially written into the cache page 312-2 of each cache sector. That is, the group of data is first written into the first cache page 312-2 of the first group of cache sectors, and the next set of data is written into the second cache page 312-2 of the first group of cache sectors, in sequence.
  • the cache sector internally counts itself, determines the current cache sector code, and automatically writes the unwritten data or the next set of data to the second set of cache sectors (next In the first cache page 312-2 of the cache sector, until all cache pages 312-2 in the cache partition are filled.
  • the corresponding data write time buffer partition full judgment is as follows: in each group of cache partitions, the control unit 311 compares the cache sector code with the maximum number of encoding sectors of the cache sector: if the cache sector code is smaller than the maximum size of the cache partition The number of codes determines that the next set of data can be successfully written in the cache partition. If the cache sector code is equal to the maximum number of encodings of the cache partition, it is determined that the cache partition is full, and the next set of data is written into the next cache partition. .
  • the data caching module 300 can be a producer and consumer model.
  • the present invention also provides a cache model of concurrent data, which includes two or more cache structures of the above concurrent data. Each cache structure works the same, so I won’t go into details here.
  • a plurality of data write threads 100 are simultaneously communicatively connected to an external data source to perform a data write operation, and simultaneously communicate with an external processing device via a plurality of data read threads 200, and each cache internal The data is read out in time and sent to an external processing device for the next step of data processing, thereby effectively enhancing the data caching capability.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

一种并发数据的缓存方法，A每组数据写入，A1监测是否有写锁定状态缓存分区：若有转入步骤A2-2；若无A2监测是否存在空闲状态缓存分区：A21若有A21-1选定一组空闲状态缓存分区置为写锁定状态；A21-2向缓存分区写入数据：A21-3判断数据是否写入成功：若成功，则该组数据写入完成；若失败，则将该缓存分区置为写满状态，转入步骤A1；A22若无空闲状态缓存分区，结束数据写入；步骤B数据读取，B1实时监测是否存在写满状态下缓存分区：若有B11选定该组缓存分区置为读锁定状态；B12读取该缓存分区内的缓存数据,当判断缓存分区读取完毕时，将其设为空闲状态，转入步骤B1；若没有，转入步骤B1，继续数据读取，其对缓存分区访问施加严格读写锁，可有效优化数据处理速率。

Description

一种并发数据的缓存方法及结构 技术领域
本发明涉及辐射检测领域的数据采集技术，具体涉及一种高速数据采集的缓存方法和设备。
背景技术
现有技术中的高速数据采集系统，其每个数据通道一般都对应设有一个独立的用于缓存通道数据的FIFO(First Input First Ouput，先入先出队列)，然而在实际的数据采集中，经常遇到数据采集率高、速度快，而数据处理时间长，速度较慢的矛盾，当缓存容量大时，这种原始数据的速率较高，对网络接收和数据处理提出了较高的要求。
以全数字PET为例，图1为全数字PET设备中数据采集与处理流程示意图，探测器各通道对接收到信号进行采样和编码，以特定格式的数据包发送到网络。PET成像所需的有效事件分布于不同的探测器通道，两者之间的匹配关系可通过事件采样时间来标定。来自各探测器通道的有效事件以及噪声数据混合在一起，串行分布于网络传输链路中，当采集服务器接收到这些数据后，基于符合算法从中筛选出有效事件，并进行时间、能量校正，然后根据重建算法将筛选结果转换为PET图像。理想情况下数据采集应与符合处理并发执行，实时完成数据筛选、时间校正、能量校正，从而降低重建前数据存储所需的资源要求。但是由于通道数量多，且每个通道中数据量十分巨大，每秒需要同时处理1.5GB至3GB的数据，若无法及时进行处理，会导致有效事件的丢包，进而导致数据筛选、时间校正、能量校正的有效性降低，图像重建无法实现或者虽然能实现但是严重影响精准性。
发明内容
本发明的目的在于提供一种并发数据的缓存方法及结构，能够有效解决数据采集率高、速度块，而数据处理时间长、速度慢的问题，尤其适用于全数字PET中的探测器数据采集。
为达到上述目的，本发明的解决方案是：
本发明公开一种并发数据的缓存方法，包括以下步骤：缓存分区的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种，
(A)每组数据写入时，(A1)实时监测是否存在写锁定状态缓存分区：若有写锁定状态缓存分区直接转入步骤(A2-2)进行该组数据写入；若没有写锁定状态缓存分区，(A2)实时监测是否存在空闲状态的缓存分区：
(A21)若有处于空闲状态的缓存分区，(A21-1)选定一组空闲状态的缓存分区置为写锁定状态；(A21-2)向所述写锁定状态的缓存分区写入该组数据：(A21-3)判断所述数据是否写入成功：若写入成功，则该组数据写入完成，结束本次数据写入；若写入失败，则将当前缓存分区置为写满状态，并转入步骤(A1)继续该组数据的写入；
(A22)若没有处于空闲状态的缓存分区，则结束该组数据写入；
步骤(B)数据读取时，(B1)实时监测是否存在写满状态下的缓存分区：若有处于写满状态下的缓存分区，(B11)选定该组写满状态的缓存分区置为读锁定状态；(B12)读取该读锁定状态缓存分区内的缓存数据,当判断所述读锁定状态缓存分区读取完毕时，将所述缓存设置为空闲状态，同时转入步骤(B1)继续下一组缓存分区的读取；若没有写满状态的缓存分区，转入步骤(B1)，继续数据的读取。
所述缓存分区中，同时只存在一组写锁定状态缓存分区与一组读锁定状态缓存分区；
优选的，全部数据写入进程结束时，若存在一组未写满的缓存分区，则将该组缓存分区置为写满状态以进行数据的读取。
依次顺序访问各缓存分区以监测是否存在写锁定或空闲状态的缓存分区；
优选的，采取遍历、与数据写入访问相同的顺序、缓存分区写入完成主动上报中的任一方式实时监测是否存在写满状态的缓存分区；
所述缓存分区内部采用分级管理的方式进行数据写入操作，所述缓存分区中包括多个大小相同的缓存扇区，且所述缓存扇区一一编码，且包括多个大小相同的缓存页面，所述每组写入数据的大小与所述缓存页面大小相同设置，则所述步骤(A21-3)中，判断所述数据是否写入成功包括以下步骤：每组数据写入缓存分区的每一缓存扇区后，缓存扇区内部计数，确定当前缓存扇区编码，比较所述缓存扇区编码与所述缓存扇区的最大编码数大小：若该缓存扇区编码小于所述缓存分区的最大编码数，判断下一组数据能够在本缓存分区中写入成功， 若缓存扇区编码等于所述缓存分区的最大编码数，则判断所述缓存分区已满，转入步骤(A1)，将下一组数据写入下一个缓存分区中；
优选的，所述数据按照编码顺序写入至所述缓存扇区中，且所述缓存页面一一编码，所述数据写入至缓存扇区时，按照缓存页面的编码顺序依次写入。
所述步骤(A21-3)中，所述数据写入成功判断包括以下步骤：向所述写锁定状态的缓存分区写入外部发送的数据组：当判断所述写锁定状态缓存分区未写满时，写入成功，继续该组缓存分区的数据写入；当判断所述写锁定状态缓存分区写满或外部数据写完毕时，将该组缓存分区置为写满状态，并转入步骤(A1)继续下一组外部数据的写入；
进一步的，所述写锁定状态的缓存分区写满判断包括以下步骤：向所述写锁定状态的缓存分区写入外部发送的数据时，比较外部数据量与所述写锁定状态的缓存分区剩余空间大小，若外部数据的数量大于所述写锁定状态的缓存分区剩余空间，判断所述写锁定状态的缓存分区写满；或，向所述写锁定状态的缓存分区写入外部发送的数据时，系统报错，则判断所述写锁定状态的缓存分区写满。
所述步骤(B1)之前,还包括实时监测是否存在读锁定状态缓存分区的步骤，若有，直接转入步骤(B12)读取所述读锁定状态缓存分区内的缓存数据；若无,则转入步骤(B1)；
优选的，所述数据读取速度大于数据写入速度；
优选的，其应有于生产者与消费者模型。
本发明公开了一种并发数据的缓存结构，其同时进行数据的写入与读取，包括一组数据写入线程、一组数据读取线程、一组数据缓存模块以及一分区控制模块；
所述数据缓存模块包括多组缓存分区；
所述分区控制模块与所述数据缓存模块通信连接以控制所述多个缓存分区的工作状态以及被访问顺序，所述分区控制模块用于将所述每个缓存分区的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种，仅当所述缓存分区为空闲状态时支持写锁定，且所述缓存分区处于写锁定状态时支持数据写入，仅当所述缓存分区为写满状态时支持读锁定，且所述缓存分区处于读锁定状态时支持数据读出；
所述数据写入线程与所述数据读取线程分别经由所述分区控制模块与各缓存分区通信连接，所述分区控制模块控制所述数据写入线程与所述数据读取线程访问缓存分区的顺序，以使得所述数据写入线程与所述数据读取线程根据所访问缓存分区的工作状态进行数据的写入与读取。
所述每组缓存分区包括一控制单元以及多组缓存扇区，所述缓存扇区一一编码设置，所述控制单元与所述每组缓存扇区通信连接，以控制每组缓存分区中，数据写入缓存扇区的顺序；
进一步优选的，所述每组缓存扇区包括一控制组件以及多个缓存页面，所述缓存页面一一编码且每组写入数据的大小与缓存页面相同设置，所述控制组件与所述缓存页面分别通向连接，以控制每组缓存扇区中，数据写入缓存页码的顺序。
所述分区控制模块包括一写锁定判断单元以及一读锁定判断单元；
所述写锁定判断单元与数据缓存模块通信连接，以控制数据缓存模块中缓存分区的访问顺序、写锁定与去写锁定，所述数据写入线程与所述写锁定判断单元通信连接以根据所述写锁定判断单元确定的访问顺序访问的缓存分区，并根据当前访问缓存分区的工作状态进行数据的写入操作；
所述读锁定判断单元与相应的数据缓存模块通信连接，以控制数据缓存模块中缓存分区的访问顺序、读锁定与去读锁定，所述数据读取线程与所述读锁定判断单元通信连接以根据所访问的缓存分区的当前状态判断是否读取缓存数据。
优选的，同时只存在一组写锁定状态缓存分区与一组读锁定状态缓存分区。
优选的，所述数据缓存模块为生产者与消费者模型。
此外，本发明还公开了一种并发数据的缓存模型，同时包括至少两组以上如权利要求8所述并发数据的缓存结构。
由于采用上述方案，本发明的有益效果是：本发明所示的简单的并发数据缓存方法及结构，通过为对缓存分区的访问施加严格的读写锁，可支持更高的网络数据速率，优化数据处理速度。
附图说明
图1为全数字PET的数据采集与处理流程示意图；
图2为本发明一实施例中并发数据缓存方法步骤示意图；
图3为图2所示实施例中并发数据缓存方法数据写入时的流程图；
图4为图2所示实施例中并发数据缓存方法数据读取时的流程图；
图5为本法一实施例中并发数据缓存结构示意图；
图6为图5所示实施例中缓存扇区的结构示意图。
具体实施方式
以下结合附图所示实施例对本发明作进一步的说明。
本发明提供了一种并发数据的缓存方法，包括以下步骤，缓存分区的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种，其中当某一缓存分区全部写满或某一时间段内外部数据写入完毕，没有新的数据写入时，该缓存分区被置为写满状态。
由于无法确定外部数据的发送时间以及数量的大小，为了确保每组外部发送的数据能够及时有效的写入至缓存中，尽量避免出现丢包的问题，如图3所示，每组数据写入过程如下，(A)一组数据写入：(A1)实时监测是否存在写锁定状态缓存分区：当确定存在写锁定状态缓存分区时，直接转入步骤(A2-2)，将该组数据写入至写锁定状态缓存分区中。若当前缓存分区中没有处于写锁定的工作状态时，(A2)则实时监测是否存在空闲状态的缓存分区：(A21)若有处于空闲状态的缓存分区，(A21-1)选定一组空闲状态的缓存分区置为写锁定状态；(A21-2)向所述写锁定状态的缓存分区写入该组数据：(A21-3)判断所述数据是否写入成功：若数据写入成功，则该组数据写入完成，结束该组数据的写入，同时可转入步骤(A)中继续下一组数据的写入；若数据写入失败，则说明当前处于写锁定工作状态的缓存分区已经写满了，无法在进行新的数据写入，则将当前缓存分区的工作状态置为写满状态，同时转入步骤(A1)中，继续该组外部数据的写入，将该组外部数据写入至其他缓存分区中；
其中，上述步骤(A21-3)中，可通过判断选定的写锁定状态缓存分区是否写满的步骤来确定数据是否写入成功。在向写锁定状态的缓存分区写入外部发送的数据组时，若判断写锁定状态缓存分区还未写满时，则说明有足够的空间写入当前数据，则当该组数据写入时可判断其写入成功；当判断选定的写锁定状态缓存分区已写满时，没有足够的空间继续写入新的数据，则说明无法将该组数据写入至上述写锁定状态缓存分区，判断其写入失败，此时，需将该组缓存分区置为写满状态，并转入步骤(A1)，以将本组数据写入至其他合适的缓存分区中。
上述写锁定状态的缓存分区写满判断具体又包括以下步骤：向写锁定状态的缓存分区写入外部发送的数据时，首先比较该组数据与写锁定状态的缓存分区剩余空间大小，若外部数据大于写锁定状态的缓存分区剩余空间，则判断写锁定状态的缓存分区写满；若外部数据不大于写锁定状态的缓存分区剩余空间，则判断写锁定状态的缓存分区未写满，本组数据可继 续写入；也可采取主动上报的方式判断写锁定状态的缓存分区是否写满，即当向写锁定状态的缓存分区写入外部发送的数据时，若系统主动报错，则判断写锁定状态的缓存分区写满。
(A22)若没有处于空闲状态的缓存分区，说明当前缓存已经全部写满，则结束该组数据的写入进程。
数据读取包括以下步骤，(B)数据读取时，以单个缓存分区为基础来进行的，(B1)实时监测是否存在写满状态下的缓存分区：若有处于写满状态下的缓存分区，(B11)选定其中一组写满状态的缓存分区置为读锁定状态；(B12)读取该读锁定状态缓存分区内的缓存数据,当判断所述读锁定状态缓存分区读取完毕时，将所述缓存设置为空闲状态，同时转入步骤(B1)继续下一组缓存分区的读取；若没有写满状态的缓存分区，则转入步骤(B1)中，判断下一时刻是否存在写满状态缓存分区以继续进行数据的读取。
此外，全部数据写入进程结束时，若存在一组未写满的缓存分区，则将该组缓存分区置为写满状态以进行数据的读取。考虑到外部数据的大小无法确定，某一时间段内数据写入进程结束时，所接收的外部数据可能无法写满一组数据缓存，或者最后一段数据无法写满一组数据缓存，此时为了读取该部分数据，将写入该部分数据的缓存分区也置为写满状态。
所述步骤(B1)之前,还包括实时监测是否存在读锁定状态缓存分区的步骤，若有，直接转入步骤(B12)读取选定读锁定状态缓存分区内的缓存数据；若无,则转入步骤(B1)，判断是否存在写满状态下的缓存分区以继续数据的读取。
为了便于数据写入与读取的管理，在所有的缓存分区中，同时只设置一组写锁定状态缓存分区与一组读锁定状态缓存分区，即缓存中同时只进行一组缓存分区的数据写入和/或一组缓存分区的数据读取，这样设置，一方面可使得数据的写入与读取更为有序，只有一组缓存分区写完或读完之后，才能对下一分区进行读写操作，另一方面，也可充分利用缓存分区的空间，保证每组缓存分区都得到有效利用，而不会出现多组缓存分区只有一部分空间被利用的情形。
在上述基础上，本发明所示的缓存方法对缓存分区的访问顺序进行设置，以更好的实现数据写入与读取的合理管理。
数据写入过程中，依次顺序访问各缓存分区以监测是否存在写锁定或空闲状态的缓存分区，即写入线程在寻找写锁定或空闲状态的缓存分区时，任意设置一缓存分区作为起点，然后按照顺序循环依次访问，具体而言，各缓存分区构建和初始化完成后一般即为空闲状态，由于同时只存在一组写锁定状态的缓存分区，在进行数据写入时，写入线程按照从起始点开 始的顺序依次访问各缓存分区，首先起始点的空闲状态缓存分别被置为写锁定状态以进行数据写入，当起始点的缓存分区写满后，起始点的下一缓存分区被置为写锁定状态，写入线程直接转入下一缓存分区继续数据的写入，直至该缓存分区写满，依次循环实现数据的写入。
由于数据写入的时间以及写入量是人为无法控制的，为了更好的应对各种突发的数据写入情况，更进一步的，缓存分区中包括多个大小相同的缓存扇区，缓存扇区一一编码且包括多个大小相同的缓存页面，将每组写入数据的大小与缓存页面大小相同设置。当确定一组写锁定状态下的缓存分区进行数据写入时，将本组数据按照顺序写入各缓存扇区的缓存页面中，即首先本组数据写入第一组缓存扇区的第一个缓存页面中，下一组数据写入第一组缓存扇区的第二个缓存页面中，依次进行，直至该组缓存扇区写满，则缓存扇区内部自己计数，确定当前缓存扇区编码，并自动将未写完的数据或者下一组数据写入至第二组缓存分区(下一缓存扇区)的第一个缓存页面中，直至该缓存分区中所有的缓存页面均被写满。相应的数据写入时缓存分区写满判断的步骤如下：比较缓存扇区编码与缓存扇区的最大编码数大小：若该缓存扇区编码小于缓存分区的最大编码数，判断下一组数据能够在本缓存分区中写入成功，若缓存扇区编码等于缓存分区的最大编码数，则判断缓存分区已满，转入步骤(A1)，将下一组数据写入下一个缓存分区中。
在进行数据读取时，可采用与数据写入访问相同的顺序的进行数据的读出，从而当缓存分区写满并置为写满状态后，数据读取线程能在第一时间内访问该缓存分区，进而该缓存分区被置为读锁定状态并进行数据的读取，这样设置，可尽量节约时间，提高数据的读出效率。同时，也可采取遍历、缓存分区写满主动上报等方式实时监测是否存在写满状态的缓存分区，从进行数据的读出。
此外，考虑到若出现外部数据量较大，缓存空间存储能力有限的，可能导致数据丢失的情形，故本发明所示的缓存方法中，数据读取速度大于数据写入速度设置。
以下以本发明所示的缓存方法应用于基本缓存模型(即单个生产者和单个消费者的场景)对其工作过程进行进一步的说明。将整个缓存分为若干个缓存分区，通过简单的状态锁来实现生产者与消费者对缓存分区的访问保护。各缓存分区构建和初始化完成后即为空闲状态。在将数据写入缓存分区之前，首先实时判断是否存在空闲状态的缓存分区，如果存在则进行数据写入操作，将该缓存分区状态置为写锁定状态，仅当缓存分区为空闲状态时支持写锁定，并通过数据写入线程进行该组缓存分区的数据写入。当判断该缓存分区写满后，一方面将该缓存分区置为可读状态(即写满且未锁定状态)，另一方面寻找缓存内其他的空闲分区，循 环继续数据写入操作。在进行数据写入操作的同时，数据读取线程也在访问缓存分区进行数据读取操作，数据读取线程实时访问各缓存分区以判断是否存在可读状态缓存分区，当数据读取线程判断当前访问的缓存分区变为可读状态时，在数据读取线程从缓存分区读取数据之前，首先需要将该分区状态置为读锁定状态，仅当缓存分区为可读状态时支持读锁定。若试图对缓存分区添加读锁定或写锁定而操作失败，意味着网络接收线程和数据处理线程访问位置发生交汇，若为写锁定相关操作失败则表示缓存已满。
本发明所示的并发数据的缓存方法也可用于由多个基本缓存模型组成的缓存中，具体步骤如前所述，每基本缓存模型均采取并发数据的缓存方法进行数据的写入与读出，此处不再赘述。
当该方法应用于PET设备中的数据采集实现时，NETTY网络框架定义了独立的数据接收线程充当生产者。由定制的工作线程充当消费者，负责网络数据包解析、事件符合计算等处理。缓存分区的访问锁基于JAVA开发环境提供的CAS机制实现，其基本原理是将缓存分区的状态设定实现为原子性操作。具体来说，访问锁对象的访问是一个原子性操作。一个操作是原子的，则这个操作的更高层不能发现其内部实现与结构。原子操作可以是一个步骤，也可以是多个步骤，其顺序是不可以被打乱的，也不能被分割只执行部分。CAS是指Compare and Set，即检查对象的值，如果满足某种条件，则将此对象设置为新值。使用CAS机制可实现不使用线程锁的方式对对象的值进行访问和更新，性能优于使用互斥锁的方案。
单个缓存分区作为事件筛选样本的基本集合单元，其容量可灵活配置，以便于调整因数据分片而引起的符合事件丢失率。为优化数据处理速度，网络数据包解析和符合筛选环节均引入了JAVA开发环境提供的Fork/Join并行计算模型。每个缓存分区的容量是相同，其容量大小可灵活配置。理论上缓存分区越大越好，这样可以保证数据丢失少，但缓存分区越大对物理内存的要求也就越高。故这个缓存分区容量大小值可根据物理内存的大小来确定。Fork/Join并行计算模型的原理是Map/Reduce模型，对一个复杂的任务进行分而治之，通过使用现在计算机多核多线程技术提高计算效率。例如计算N个数的和，将N个数分为M份，并使用M个线程同步执行计算的任务，再将M个结果合并成一个结果。
本发明还公开了一种并发数据的缓存结构，同时进行数据的写入与读取，包括一组数据写入线程100、一组数据读取线程200、一组数据缓存模块300以及一分区控制模块400。
数据缓存模块300包括多组缓存分区310，分区控制模块400与数据缓存模块300通信连接，用于控制数据缓存模块300中各个缓存分区310的工作状态，即分区控制模块400将每个缓存分 区310的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种。仅当缓存分区310为空闲状态时支持写锁定，且当缓存分区310的工作状态为写锁定时支持数据写入线程100进行数据的写入；仅当缓存分区310为写满状态时支持读锁定，且当缓存分区310的工作状态为读锁定状态时支持数据读取线程200进行数据的读出，同时分区控制模块400还用于控制各个缓存分区310的被访问顺序，数据写入线程100与数据读取线程200分别经由分区控制模块400与各缓存分区310通信连接，数据写入线程100与数据读取线程200按照分区控制模块400确定的访问顺序分别访问数据缓存模块300中的多组缓存分区，然后根据其所访问缓存分区的状态来进行数据的写入或者读取操作。
为了便于数据的写入与读取，分区控制模块400将读写操作分开进行，其包括一写锁定判断单元410以及一读锁定判断单元420，且在所有的缓存分区中，同时只设置一组写锁定状态缓存分区与一组读锁定状态缓存分区。
写锁定判断单元410与数据缓存模块300通信连接，用于控制数据缓存模块300中缓存分区310的访问顺序、写锁定(将空闲状态的缓存分区置为写锁定状态)与去写锁定(将写满的写锁定状态缓存分区置为写满状态)。数据写入线程100与写锁定判断单元410通信连接，其根据写锁定判断单元410确定的访问顺序来访问各缓存分区310，并根据当前所访问缓存分区310的工作状态进行数据的写入操作。在其中一实施例中，为了使得各缓存分区的数据输入相对均衡，写锁定判断单元410采用定制遍历策略来调节数据在缓存模块中的分布，即顺序遍历，可任意选取其中一组缓存分区作为起始点并按照编码顺序对缓存分区进行控制。缓存构建完成后，一般情况下，各缓存分区均为空闲状态，写锁定判断单元410将处于起始点的空闲状态的缓存分区置为写锁定状态，数据写入线程100按照写锁定判断单元410确定的访问顺序访问起始点的缓存分区，由于其为写锁定状态，则可开始进行数据写入，待该缓存分区写满以后，写锁定判断单元410将起始点的缓存分区置为写满状态，并将起始点下一编码处的空闲状态缓存分区置为写锁定状态，数据写入线程100按照写锁定判断单元410确定的继续访问下一编码缓存分区，然后进行数据写入操作。
读锁定判断单元420与相应的数据缓存模块300也通信连接，用于控制数据缓存模块300中缓存分区的访问顺序、读锁定(将写满状态的缓存分区置为读锁定状态)与去读锁定(将读取完毕的缓存分区置为空闲状态)，数据读取线程200与读锁定判断单元420通信连接以根据所访问的缓存分区的当前状态判断是否读取缓存数据。在数据写入的同时，数据读取线程200也在进行数据的读取操作，数据读取的访问顺序可采用与数据写入访问相同的顺序的 进行数据的读出，即读锁定判断单元420控制数据读取线程200按照写锁定判断单元410所确定的访问顺序，来访问各缓存分区，这样设置，当缓存分区写满并置为写满状态后，数据读取线程200能在第一时间内访问该缓存分区，进而该缓存分区被置为读锁定状态并进行数据的读取，能够更进一步的提高数据的读出效率。此外，读锁定判断单元420也可采取遍历顺序，或者由各缓存分区写满主动上报，读锁定判断单元420以及缓存分区上报的顺序确定数据读取线程200的访问顺序。
由于数据写入的过程是人为无法控制的过程，可能出现在上述设置的基础上，为了有效的进行数据的写入管理，减少数据丢包事件的发生，同时提高缓存的利用率，每组缓存分区310分层级管理设置，其包括一控制单元311以及多组缓存扇区312，缓存扇区312一一编码设置，控制单元311与该组缓存分区310中的所有缓存扇区312通信连接，用于控制每组缓存分区中，外部数据写入缓存扇区312的顺序。进一步的，每组缓存扇区312包括一控制组件312-1以及多个缓存页面312-2，缓存页面312-2一一编码且每组写入数据的大小与缓存页面312-2相同设置，控制组件312-1与缓存页面312-2分别通向连接，以控制每组缓存扇区中，数据写入缓存页码的顺序。
当该组缓存分区设置为写锁定状态且与数据写入线程100通信成功进行数据写入操作时，缓存分区内部对数据写入的过程二次导向。在其中一实施例中，控制单元311控制缓存分区内部也采用顺序写入的方式进行，一组数据写入时，将本组数据按照顺序写入各缓存扇区的缓存页面312-2中，即首先将该组数据写入第一组缓存扇区的第一个缓存页面312-2中，下一组数据写入第一组缓存扇区的第二个缓存页面312-2中，依次进行，直至该组缓存扇区写满，则缓存扇区内部自己计数，确定当前缓存扇区编码，并自动将未写完的数据或者下一组数据写入至第二组缓存扇区(下一缓存扇区)的第一个缓存页面312-2中，直至该缓存分区中所有的缓存页面312-2均被写满。相应的数据写入时缓存分区写满判断的步骤如下：每组缓存分区中，控制单元311比较缓存扇区编码与缓存扇区的最大编码数大小：若该缓存扇区编码小于缓存分区的最大编码数，判断下一组数据能够在本缓存分区中写入成功，若缓存扇区编码等于缓存分区的最大编码数，则判断缓存分区已满，将下一组数据写入下一个缓存分区中。
作为一个优选方案，数据缓存模块300可为生产者与消费者模型。
理论上缓存分区越大越好，这样可以保证数据丢失少，但缓存分区越大对物理内存的要求也就越高。由于缓存容量有限，故本发明还提供了一种并发数据的缓存模型，其同时包括两组以上的上述并发数据的缓存结构。每个缓存结构的工作原理相同，此处不再赘述，上述 并发数据的缓存模型中，同时经由多个数据写入线程100与外部数据源通信连接，进行数据的写入操作，同时经由多个数据读取线程200与外部处理设备通信连接，将各缓存内部的数据及时读出并发送至外部处理设备中进行下一步的数据处理，从而有效增强数据的缓存能力。
上述的对实施例的描述是为便于该技术领域的普通技术人员能理解和使用本发明。熟悉本领域技术的人员显然可以容易地对这些实施例做出各种修改，并把在此说明的一般原理应用到其他实施例中而不必经过创造性的劳动。因此，本发明不限于上述实施例，本领域技术人员根据本发明的揭示，不脱离本发明范畴所做出的改进和修改都应该在本发明的保护范围之内。

Claims (11)

  1. 一种并发数据的缓存方法，其特征在于：包括以下步骤：缓存分区的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种，
    (A)每组数据写入时，(A1)实时监测是否存在写锁定状态缓存分区：若有写锁定状态缓存分区直接转入步骤(A2-2)进行该组数据写入；若没有写锁定状态缓存分区，(A2)实时监测是否存在空闲状态的缓存分区：
    (A21)若有处于空闲状态的缓存分区，(A21-1)选定一组空闲状态的缓存分区置为写锁定状态；(A21-2)向所述写锁定状态的缓存分区写入该组数据：(A21-3)判断所述数据是否写入成功：若写入成功，则该组数据写入完成，结束本次数据写入；若写入失败，则将当前缓存分区置为写满状态，并转入步骤(A1)继续该组数据的写入；
    (A22)若没有处于空闲状态的缓存分区，则结束该组数据写入；
    步骤(B)数据读取时，(B1)实时监测是否存在写满状态下的缓存分区：若有处于写满状态下的缓存分区，(B11)选定该组写满状态的缓存分区置为读锁定状态；(B12)读取该读锁定状态缓存分区内的缓存数据,当判断所述读锁定状态缓存分区读取完毕时，将所述缓存设置为空闲状态，同时转入步骤(B1)继续下一组缓存分区的读取；若没有写满状态的缓存分区，转入步骤(B1)，继续数据的读取。
  2. 根据权利要求1所述的并发数据的缓存方法，其特征在于：所述缓存分区中，同时只存在一组写锁定状态缓存分区与一组读锁定状态缓存分区；
    优选的，全部数据写入进程结束时，若存在一组未写满的缓存分区，则将该组缓存分区置为写满状态以进行数据的读取。
  3. 根据权利要求1或2所述的并发数据的缓存方法，其特征在于：依次顺序访问各缓存分区以监测是否存在写锁定或空闲状态的缓存分区；
    优选的，采取遍历、与数据写入访问相同的顺序、缓存分区写入完成主动上报中的任一方式实时监测是否存在写满状态的缓存分区；
  4. 根据权利要求1所述的并发数据的缓存方法，其特征在于：所述缓存分区内部采用分级管理的方式进行数据写入操作，所述缓存分区中包括多个大小相同的缓存扇区，且所述缓存扇区一一编码，且包括多个大小相同的缓存页面，所述每组写入数据的大小与所述缓存页面大小相同设置，则所述步骤(A21-3)中，判断所述数据是否写入成功包括以下步骤：每组数据写入缓存分区的每一缓存扇区后，缓存扇区内部计数，确定当前缓存扇区编码，比较所述缓存 扇区编码与所述缓存扇区的最大编码数大小：若该缓存扇区编码小于所述缓存分区的最大编码数，判断下一组数据能够在本缓存分区中写入成功，若缓存扇区编码等于所述缓存分区的最大编码数，则判断所述缓存分区已满，转入步骤(A1)，将下一组数据写入下一个缓存分区中；
    优选的，所述数据按照编码顺序写入至所述缓存扇区中，且所述缓存页面一一编码，所述数据写入至缓存扇区时，按照缓存页面的编码顺序依次写入。
  5. 根据权利要求1所述的并发数据的缓存方法，其特征在于：所述步骤(A21-3)中，所述数据写入成功判断包括以下步骤：向所述写锁定状态的缓存分区写入外部发送的数据组：当判断所述写锁定状态缓存分区未写满时，写入成功，继续该组缓存分区的数据写入；当判断所述写锁定状态缓存分区写满或外部数据写完毕时，将该组缓存分区置为写满状态，并转入步骤(A1)继续下一组外部数据的写入；
    进一步的，所述写锁定状态的缓存分区写满判断包括以下步骤：向所述写锁定状态的缓存分区写入外部发送的数据时，比较外部数据量与所述写锁定状态的缓存分区剩余空间大小，若外部数据的数量大于所述写锁定状态的缓存分区剩余空间，判断所述写锁定状态的缓存分区写满；或，向所述写锁定状态的缓存分区写入外部发送的数据时，系统报错，则判断所述写锁定状态的缓存分区写满。
  6. 根据权利要求1所述的并发数据的缓存方法，其特征在于：所述步骤(B1)之前,还包括实时监测是否存在读锁定状态缓存分区的步骤，若有，直接转入步骤(B12)读取所述读锁定状态缓存分区内的缓存数据；若无,则转入步骤(B1)；
    优选的，所述数据读取速度大于数据写入速度；
    优选的，其应有于生产者与消费者模型。
  7. 一种并发数据的缓存结构，其特征在于：其同时进行数据的写入与读取，包括一组数据写入线程、一组数据读取线程、一组数据缓存模块以及一分区控制模块；
    所述数据缓存模块包括多组缓存分区；
    所述分区控制模块与所述数据缓存模块通信连接以控制所述多个缓存分区的工作状态以及被访问顺序，所述分区控制模块用于将所述每个缓存分区的工作状态设置为空闲状态、写锁定状态、写满状态以及读锁定状态中的任意一种，仅当所述缓存分区为空闲状态时支持写锁定，且所述缓存分区处于写锁定状态时支持数据写入，仅当所述缓存分区为写满状态时支持读锁定，且所述缓存分区处于读锁定状态时支持数据读出；
    所述数据写入线程与所述数据读取线程分别经由所述分区控制模块与各缓存分区通信连接，所述分区控制模块控制所述数据写入线程与所述数据读取线程访问缓存分区的顺序，以使得所述数据写入线程与所述数据读取线程根据所访问缓存分区的工作状态进行数据的写入与 读取。
  8. 根据权利要求7所述的缓存结构，其特征在于：所述每组缓存分区包括一控制单元以及多组缓存扇区，所述缓存扇区一一编码设置，所述控制单元与所述每组缓存扇区通信连接，以控制每组缓存分区中，数据写入缓存扇区的顺序；
    进一步优选的，所述每组缓存扇区包括一控制组件以及多个缓存页面，所述缓存页面一一编码且每组写入数据的大小与缓存页面相同设置，所述控制组件与所述缓存页面分别通向连接，以控制每组缓存扇区中，数据写入缓存页码的顺序。
  9. 根据权利要求7所述的缓存结构，其特征在于：所述分区控制模块包括一写锁定判断单元以及一读锁定判断单元；
    所述写锁定判断单元与数据缓存模块通信连接，以控制数据缓存模块中缓存分区的访问顺序、写锁定与去写锁定，所述数据写入线程与所述写锁定判断单元通信连接以根据所述写锁定判断单元确定的访问顺序访问的缓存分区，并根据当前访问缓存分区的工作状态进行数据的写入操作；
    所述读锁定判断单元与相应的数据缓存模块通信连接，以控制数据缓存模块中缓存分区的访问顺序、读锁定与去读锁定，所述数据读取线程与所述读锁定判断单元通信连接以根据所访问的缓存分区的当前状态判断是否读取缓存数据。
  10. 根据权利要求7所述的缓存结构，其特征在于：同时只存在一组写锁定状态缓存分区与一组读锁定状态缓存分区。
    优选的，所述数据缓存模块为生产者与消费者模型。
  11. 一种并发数据的缓存模型，其特征在于：同时包括至少两组以上如权利要求7所述的并发数据的缓存结构。
PCT/CN2017/077486 2016-04-07 2017-03-21 一种并发数据的缓存方法及结构 WO2017173919A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610210432.4 2016-04-07
CN201610210432.4A CN105912479B (zh) 2016-04-07 2016-04-07 一种并发数据的缓存方法及装置

Publications (1)

Publication Number Publication Date
WO2017173919A1 true WO2017173919A1 (zh) 2017-10-12

Family

ID=56745376

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/077486 WO2017173919A1 (zh) 2016-04-07 2017-03-21 一种并发数据的缓存方法及结构

Country Status (2)

Country Link
CN (1) CN105912479B (zh)
WO (1) WO2017173919A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111367666A (zh) * 2020-02-28 2020-07-03 深圳壹账通智能科技有限公司 数据读写方法及系统
CN111913657A (zh) * 2020-07-10 2020-11-10 长沙景嘉微电子股份有限公司 块数据读写方法、装置、系统及存储介质
CN112416816A (zh) * 2020-12-08 2021-02-26 上证所信息网络有限公司 一种一写多读高并发无锁环形缓存及其实现方法
CN115357199A (zh) * 2022-10-19 2022-11-18 安超云软件有限公司 分布式存储系统中的数据同步方法、系统及存储介质

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105912479B (zh) * 2016-04-07 2023-05-05 合肥锐世数字科技有限公司 一种并发数据的缓存方法及装置
CN109213691B (zh) * 2017-06-30 2023-09-01 伊姆西Ip控股有限责任公司 用于缓存管理的方法和设备
CN110647477B (zh) * 2018-06-27 2022-02-11 阿里巴巴（中国）有限公司 数据缓存方法、装置、终端以及计算机可读存储介质
CN110874273B (zh) * 2018-08-31 2023-06-13 阿里巴巴集团控股有限公司 一种数据处理方法及装置
CN109617825B (zh) * 2018-11-30 2022-03-25 京信网络系统股份有限公司 消息处理装置、方法及通信系统
CN109711323B (zh) * 2018-12-25 2021-06-15 武汉烽火众智数字技术有限责任公司 一种实时视频流分析加速方法、装置和设备
CN112804003A (zh) * 2021-02-19 2021-05-14 上海剑桥科技股份有限公司 一种基于光模块通信的存储方法、系统及终端
CN114035746B (zh) * 2021-10-28 2023-06-16 中国科学院声学研究所 一种高采样率数据实时采集存储方法及采集存储系统
CN114217587A (zh) * 2021-12-15 2022-03-22 之江实验室 一种拟态执行体多类型数据实时比较和汇聚方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103218176A (zh) * 2013-04-02 2013-07-24 中国科学院信息工程研究所 数据处理方法及装置
US20140089539A1 (en) * 2012-09-24 2014-03-27 Sap Ag Lockless Spin Buffer
CN104881258A (zh) * 2015-06-10 2015-09-02 北京金山安全软件有限公司 缓冲区并发访问方法及装置
CN105868123A (zh) * 2016-04-07 2016-08-17 武汉数字派特科技有限公司 一种并发数据的缓存结构及方法
CN105912479A (zh) * 2016-04-07 2016-08-31 武汉数字派特科技有限公司 一种并发数据的缓存方法及结构

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5872980A (en) * 1996-01-25 1999-02-16 International Business Machines Corporation Semaphore access control buffer and method for accelerated semaphore operations
US8386714B2 (en) * 2010-06-29 2013-02-26 International Business Machines Corporation Reducing write amplification in a cache with flash memory used as a write cache
CN102298561B (zh) * 2011-08-10 2016-04-27 北京百度网讯科技有限公司 一种对存储设备进行多通道数据处理的方法、系统和装置
CN102325010A (zh) * 2011-09-13 2012-01-18 浪潮(北京)电子信息产业有限公司 一种避免数据粘包的处理装置及方法
CN103257888A (zh) * 2012-02-16 2013-08-21 阿里巴巴集团控股有限公司 一种对缓冲队列并发执行读、写访问的方法和设备
CN102799537B (zh) * 2012-06-18 2015-07-08 北京空间飞行器总体设计部 一种航天器aos系统双口ram缓冲区管理方法
CN103780506B (zh) * 2012-10-26 2017-08-08 中兴通讯股份有限公司 一种用于以太网设备的数据缓存系统及方法
US9223710B2 (en) * 2013-03-16 2015-12-29 Intel Corporation Read-write partitioning of cache memory
CN103150149B (zh) * 2013-03-26 2015-11-25 华为技术有限公司 处理数据库重做数据的方法和装置
US9003131B1 (en) * 2013-03-27 2015-04-07 Parallels IP Holdings GmbH Method and system for maintaining context event logs without locking in virtual machine
US10684986B2 (en) * 2013-08-28 2020-06-16 Biosense Webster (Israel) Ltd. Double buffering with atomic transactions for the persistent storage of real-time data flows
CN103412786B (zh) * 2013-08-29 2017-04-12 苏州科达科技股份有限公司 一种高性能服务器架构系统及数据处理方法
CN103914565B (zh) * 2014-04-21 2017-05-24 北京搜狐新媒体信息技术有限公司 一种向数据库插入数据的方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140089539A1 (en) * 2012-09-24 2014-03-27 Sap Ag Lockless Spin Buffer
CN103218176A (zh) * 2013-04-02 2013-07-24 中国科学院信息工程研究所 数据处理方法及装置
CN104881258A (zh) * 2015-06-10 2015-09-02 北京金山安全软件有限公司 缓冲区并发访问方法及装置
CN105868123A (zh) * 2016-04-07 2016-08-17 武汉数字派特科技有限公司 一种并发数据的缓存结构及方法
CN105912479A (zh) * 2016-04-07 2016-08-31 武汉数字派特科技有限公司 一种并发数据的缓存方法及结构

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111367666A (zh) * 2020-02-28 2020-07-03 深圳壹账通智能科技有限公司 数据读写方法及系统
CN111913657A (zh) * 2020-07-10 2020-11-10 长沙景嘉微电子股份有限公司 块数据读写方法、装置、系统及存储介质
CN112416816A (zh) * 2020-12-08 2021-02-26 上证所信息网络有限公司 一种一写多读高并发无锁环形缓存及其实现方法
CN115357199A (zh) * 2022-10-19 2022-11-18 安超云软件有限公司 分布式存储系统中的数据同步方法、系统及存储介质
CN115357199B (zh) * 2022-10-19 2023-02-10 安超云软件有限公司 分布式存储系统中的数据同步方法、系统及存储介质

Also Published As

Publication number Publication date
CN105912479A (zh) 2016-08-31
CN105912479B (zh) 2023-05-05

Similar Documents

Publication Publication Date Title
WO2017173919A1 (zh) 一种并发数据的缓存方法及结构
US11882054B2 (en) Terminating data server nodes
Gu et al. Biscuit: A framework for near-data processing of big data workloads
US10296498B2 (en) Coordinated hash table indexes to facilitate reducing database reconfiguration time
CN105868123B (zh) 一种并发数据的缓存装置及方法
CN105144160B (zh) 利用闪存高速缓存中动态生成的替代数据格式加速查询的方法
US20160328331A1 (en) Fast Data Initialization
US8352517B2 (en) Infrastructure for spilling pages to a persistent store
US11487435B1 (en) System and method for non-volatile memory-based optimized, versioned, log-structured metadata storage with efficient data retrieval
CN101989236B (zh) 一种指令缓存锁实现方法
CN103577158B (zh) 数据处理方法与装置
CN107870732B (zh) 从固态存储设备冲刷页面的方法和设备
CN111831330B (zh) 用于联邦学习的异构计算系统设备交互方案
US20110161608A1 (en) Method to customize function behavior based on cache and scheduling parameters of a memory argument
US20170068675A1 (en) Method and system for adapting a database kernel using machine learning
CN105094751B (zh) 一种用于流式数据并行处理的内存管理方法
US9858191B2 (en) Electronic system with data management mechanism and method of operation thereof
CN105138679A (zh) 一种基于分布式缓存的数据处理系统及处理方法
US20190042427A1 (en) Reconfigurable cache architecture and methods for cache coherency
CN117176811B (zh) 阻塞监听多客户并控制多硬件的服务端架构、系统及方法
CN104679688B (zh) 数据访问方法、装置及系统
US11294885B2 (en) Transactional integrity in a segmented database architecture
KR20210103376A (ko) 적응 회귀를 이용한 네트워크 키-값 저장 장치에서 객체 그룹화 스키마의 성능을 최적화하는 방법, 및 이를 이용한 비 일시적 컴퓨터 판독 가능 매체와 시스템
CN111860788A (zh) 一种基于数据流架构的神经网络计算系统及方法
CN112988885A (zh) 一种基于改进的布谷鸟过滤器的智能合约

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17778583

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17778583

Country of ref document: EP

Kind code of ref document: A1