CN110287044A - Without lock shared drive processing method, device, electronic equipment and readable storage medium storing program for executing - Google Patents
Without lock shared drive processing method, device, electronic equipment and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN110287044A CN110287044A CN201910591481.0A CN201910591481A CN110287044A CN 110287044 A CN110287044 A CN 110287044A CN 201910591481 A CN201910591481 A CN 201910591481A CN 110287044 A CN110287044 A CN 110287044A
- Authority
- CN
- China
- Prior art keywords
- data
- block
- information
- shared drive
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application provides a kind of without lock shared drive processing method, device, electronic equipment and readable storage medium storing program for executing, shared memory file is mapped in shared drive region in such a way that memory maps, and the head pointer address of corresponding mapping area in obtained shared drive region is recorded in the memory of the corresponding application program of data service with record pointer address is deviated according to the mapping return value after the completion of mapping.On this basis, with the atomic data structure of atomic update, and by data be divided into index area, Hash array data area, extract pond and storage data area multiple atom integers method, without setting privately owned retrieval data for each process or thread, in the processing of data high concurrent, committed memory is smaller, so that in live broadcast service the reading and writing data service of high concurrent can be provided for high speed development business, and provide the high performance read-write of high concurrent and storage scheme for memory database.
Description
Technical field
This application involves field of computer technology, in particular to it is a kind of without lock shared drive processing method, device,
Electronic equipment and readable storage medium storing program for executing.
Background technique
Live broadcast service concurrency is big, and data are updated and read frequently, and there are bottlenecks for the concurrent reading and writing under traditional lock competition.
In addition, the reading and writing data mode that tradition is single, has no longer met live broadcast service and has been skyrocketed through, reads and writes the huge need of concurrent request
It asks.
Summary of the invention
In view of this, being designed to provide for the embodiment of the present application is a kind of without lock shared drive processing method, device, electronics
Equipment and readable storage medium storing program for executing, to solve or improve the above problem.
According to the one aspect of the embodiment of the present application, a kind of electronic equipment is provided, may include that one or more storages are situated between
Matter and one or more processors communicated with storage medium.One or more storage mediums are stored with the executable machine of processor
Device executable instruction.When electronic equipment operation, processor executes the machine-executable instruction, following total without lock to execute
Enjoy internal memory processing method.
It according to the another aspect of the embodiment of the present application, provides a kind of without lock shared drive processing method, is set applied to electronics
It is standby, which comprises
For each data service, the memory configurations data of the data service are configured, the memory configurations data include institute
Data block size, data number of blocks and the shared memory file path needed；
Shared memory space needed for calculating the data service according to the data block size and data number of blocks；
Shared memory file is opened according to the shared memory file path, and according to the shared memory space, is passed through
The shared memory file is mapped in shared drive region by preset memory mapping mode, and obtains reflecting after the completion of mapping
Penetrate return value；
The head pointer address of corresponding mapping area in the shared drive region is obtained according to the mapping return value
With offset record pointer address, and by the head pointer address and offset record pointer address the data service pair is recorded
In the memory for the application program answered；
It is shared drive region distribution index after the atom map for carrying out auxiliary data to the shared drive region
Area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the total without lock of the data service
Enjoy memory processing.
It according to the another aspect of the embodiment of the present application, provides a kind of without lock shared drive processing unit, is set applied to electronics
Standby, described device includes:
Data configuration module, it is described interior for configuring the memory configurations data of the data service for each data service
Depositing configuration data includes required data block size, data number of blocks and shared memory file path；
Computing module, in being shared needed for calculating the data service according to the data block size and data number of blocks
Deposit space；
Memory mapping block, for opening shared memory file according to the shared memory file path, and according to described
The shared memory file is mapped in shared drive region by shared memory space by preset memory mapping mode, and
Obtain the mapping return value after the completion of mapping；
Address logging modle, for obtaining corresponding map section in the shared drive region according to the mapping return value
The head pointer address and offset record pointer address in domain, and the head pointer address and offset record pointer address are recorded
Into the memory of the corresponding application program of the data service；
Distribution module is described shared interior after the atom map to shared drive region progress auxiliary data
Region distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area are deposited, to complete the number
According to business without lock shared drive processing.
According to the another aspect of the embodiment of the present application, a kind of readable storage medium storing program for executing is provided, is stored on the readable storage medium storing program for executing
There is machine-executable instruction, can be executed when which is run by processor above-mentioned without lock shared drive processing method
The step of.
Based on any of the above-described aspect, shared memory file is mapped to altogether by the embodiment of the present application in such a way that memory maps
It enjoys in region of memory, and according to the mapping return value after the completion of mapping by corresponding mapping area in obtained shared drive region
Head pointer address be recorded in the memory of the corresponding application program of data service with offset record pointer address.It is basic herein
On, with the atomic data structure of atomic update, and by data be divided into index area, Hash array data area, extract pond with
And the method for multiple atom integers of storage data area, without setting privately owned retrieval data for each process or thread,
Committed memory is smaller when the processing of data high concurrent, so that can provide high concurrent in live broadcast service for high speed development business
Reading and writing data service, and the high performance read-write of high concurrent and storage scheme are provided for memory database.
Detailed description of the invention
Technical solution in ord to more clearly illustrate embodiments of the present application, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only some embodiments of the application, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 shows one of the flow diagram without lock shared drive processing method provided by the embodiment of the present application；
Fig. 2 shows the flow diagrams of each sub-steps for including of step S150 shown in Fig. 1；
Fig. 3 shows two of the flow diagram without lock shared drive processing method provided by the embodiment of the present application；
Fig. 4 is shown provided by the embodiment of the present application for executing the above-mentioned electronics without lock shared drive processing method
The structural schematic block diagram of equipment.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it should be understood that attached drawing in the application
The purpose of illustration and description is only played, is not used to limit the protection scope of the application.In addition, it will be appreciated that schematical attached
Figure does not press scale.Process used herein shows real according to some embodiments of the embodiment of the present application
Existing operation.It should be understood that the operation of flow chart can be realized out of order, the step of context relation of logic can be with
Reversal order is implemented simultaneously.In addition, those skilled in the art under the guide of teachings herein, can add to flow chart
Other one or more operations, can also remove one or more operations from flow chart.
In addition, described embodiments are only a part of embodiments of the present application, instead of all the embodiments.Usually exist
The component of the embodiment of the present application described and illustrated in attached drawing can be arranged and be designed with a variety of different configurations herein.Cause
This, is not intended to limit claimed the application's to the detailed description of the embodiments herein provided in the accompanying drawings below
Range, but it is merely representative of the selected embodiment of the application.Based on embodiments herein, those skilled in the art are not being done
Every other embodiment obtained under the premise of creative work out, shall fall in the protection scope of this application.
Server is a multicore processing equipment mostly, and the most important mode of Performance tuning for server is parallel
Programming, so that server can be with some shared datas of parallel processing.For example, server can using liquidate impact lock by the way of,
I.e. multiple Parallel Objects (thread or process) access (read-write) to same part data.Wherein, in order to protect the number of server
According to area, when operating there is currently an object to certain part of data block, another object, which needs to block, to be waited.
And then withouted waiting for without lock data structure, it can concurrently carry out, thus by reducing obstruction and waiting, to improve
The concurrency and scalability of server.The deadlock that the data structure of locking may result in server in abnormal cases is asked
Topic also results in all threads or process blocking in addition, if the dynamics of locking is too big, leads to money when subsequent unlock and unlock
Source consumption and context switching consumption, to Priority Inversion occur and lock the phenomenon that escorting.Data structure without lock, is compared
In the above-mentioned data structure for having lock, it is possible to reduce resource consumption (namely time loss in some sense), and can eliminate
Condition competition, obstruction, deadlock, composability deficiency bring potential problems.However, inventor has found under study for action, current nothing
The data structure of lock needs to set privately owned retrieval data for each process or thread, in the processing of data high concurrent, committed memory
It is larger.
For this purpose, the discovery based on above-mentioned technical problem, inventor proposes that following technical proposals are above-mentioned to solve or improve
Problem.It is inventor by practice and careful it should be noted that defect present in the above scheme in the prior art
It is being obtained after research as a result, therefore, the discovery procedure of the above problem and hereinafter the embodiment of the present application institute regarding to the issue above
The solution of proposition all should be the contribution that inventor makes the application during innovation and creation, without should understand that
Technology contents known to those skilled in the art.
Fig. 1 shows the flow diagram provided by the embodiments of the present application without lock shared drive processing method, it should be understood that
In other embodiments, the sequence without lock shared drive processing method part step of the present embodiment can not with Fig. 1 and
The sequence of following specific embodiments is limitation, such as can be exchanged with each other according to actual needs or part steps therein
It can be omitted or delete.The detailed step without lock shared drive processing method is described below.
Step S110 configures the memory configurations data of the data service for each data service.
In the present embodiment, by taking live scene as an example, data service may include live video business, live streaming voice service,
The multiple business such as order business are broadcast live, corresponding memory configurations data can be configured for different data services.Wherein, described
Memory configurations data may include data block size, data number of blocks needed for the data service and shared memory file road
Diameter.
Step S120, shared drive needed for calculating the data service according to the data block size and data number of blocks are empty
Between.
Step S130 opens shared memory file according to the shared memory file path, and according to the shared drive
The shared memory file is mapped in shared drive region by preset memory mapping mode, and obtains mapping by space
Mapping return value after the completion.
Illustratively, which can be mmap mode, and mmap can reflect shared memory file
It is mapped in shared drive region in the address space of corresponding process, realizes in file disk address and process virtual address space
The enantiomorphic relationship one by one of one section of virtual address, after realizing such mapping relations, process can be read by the way of pointer
This section of memory of write operation.After completing the shared memory file mapping, a mmap return value can be returned to, as the mapping
Return value.
Step S140 obtains the starting of corresponding mapping area in the shared drive region according to the mapping return value
Pointer address and offset record pointer address, and the number is recorded in the head pointer address and offset record pointer address
In memory according to the corresponding application program of business.
Step S150 is the shared section key after the atom map of shared drive region progress auxiliary data
Domain distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the data industry
Business is handled without lock shared drive.
Based on above-mentioned design, the shared memory file is mapped to shared drive in such a way that memory maps by the application
In region, and according to the mapping return value after the completion of mapping by the starting of corresponding mapping area in obtained shared drive region
Pointer address is recorded in the memory of the corresponding application program of the data service with offset record pointer address, on this basis
With the atomic data structure of atomic update, and index area, Hash array data area, extraction pond are divided by data and is deposited
The method for storing up multiple atom integers of data field, without setting privately owned retrieval data for each process or thread, in data
Committed memory is smaller when high concurrent processing, so that can provide the data of high concurrent in live broadcast service for high speed development business
Read-write service, and the high performance read-write of high concurrent and storage scheme are provided for memory database.
In a kind of possible embodiment, for step S150, auxiliary data is carried out to the shared drive region
The mode of atom map may is that
Generate the remark information area in the shared drive region；
The position configuration flag value of first character section in the remark information area；
The position of second byte in the remark information area configures current data capacity；
The position configuration data block size of third byte in the remark information area；
The position configuration data number of blocks of the 4th byte in the remark information area；
The first character is respectively directed to by the int type pointer of four atoms in the memory of the application program respectively
The data word locking nub of section, second byte, the third byte and the 4th byte data, to pass through described four
The int type pointer type of atom controls the first character section, second byte, the third byte and respectively
Four bytes are read and write without lock.
In detail, the remark information area in the shared drive region of generation be blank area, first in the blank area
A byte, second byte, the configuration mode of third byte and the 4th byte are as follows:
Data number of blocks: s_blockCount=((au64*) m_mem.data ())+2；
Data block size: s_blockSize=((au64*) m_mem.data ())+1；
Mark value: s_flags=(au32*) m_mem.data ()；
Current data capacity: s_cnt=((au32*) m_mem.data ())+1；
Then, the first character section, second described is controlled by the int type pointer type of four atoms respectively
The configuration process without lock read-write of byte, the third byte and the 4th byte is as follows:
au32*s_flags；
au32*s_cnt；
au64*s_blockSize；
au64*s_blockCount；
On this basis, further, Fig. 2 is please referred to, in a kind of possible embodiment, step S150 can
To be realized by following sub-step:
Sub-step S151 is the shared drive region distribution index area, and distribution is used for data in the index area
The multiple VerIdx arrays and check information list of index are stored with the number pair of each data block in the VerIdx array
The cryptographic Hash answered, the check information list include the storage byte obtained according to the quantity of Hash bucket and the storage byte
Extended field after adding 16 bytes.
Sub-step S152 distributes Hash array data area for the shared drive region, and to the Hash array data
Area carries out atom map, and the metadata of each data block is recorded by data link table, wherein the metadata passes through atom shaping
It is stored, the metadata includes data block length, data block version number, data block identifier, data block cryptographic Hash, data block
Delete the combination of one or more of label and data block reference count.Wherein, above-mentioned a variety of combinations can be
Two or more combination.
Sub-step S153 extracts pond for shared drive region distribution, and the extraction pond is referred to by 64 atom pointers
To next region of memory that can be written into, and the position of a data block is recorded, the corresponding atom shaping in the extraction pond
The identification information of next data block of global data block information and each data block is had recorded in pointer alignment.
Sub-step S154 distributes storage data area for the shared drive region, and the size of the storage data area is equal to
The quantity of the Hash bucket and the product of size.
In this way, by above-mentioned design, by the atomic data structure with atomic update, and being divided into index by data
Area, Hash array data area, extract pond and storage data area multiple atom integers method, without for each process
Or thread sets privately owned retrieval data.
On the basis of foregoing description, below with reference to Fig. 3 to based on it is above-mentioned to the data service without lock shared drive at
The application of data structure after reason carries out exemplary description, referring to Fig. 3, the present embodiment mentions after step S150
What is supplied can also include the following steps: without lock shared drive processing method
Step S160, when receiving the concurrent data processing request for data service, at the concurrent data
Reason request obtains the identification number of pending data.
In detail, when receiving the concurrent data processing request for data service, for different data services, all
It is corresponding with an identification number FLAG relevant to pending data.
Step S160 calculates the corresponding cryptographic Hash of the identification number, and according to the cryptographic Hash to the pending data
Execute corresponding operation.
In the present embodiment, the mode for calculating the corresponding cryptographic Hash of the identification number can be executed by following code:
U32 hash=CncrHsh::HashBytes (key, klen)；
In a kind of possible example, if concurrent data processing request is data write request, this step is first
Then data number of blocks needed for calculating the pending data according to the cryptographic Hash is extracted from the extraction pond of above-mentioned configuration
Freed data blocks corresponding with the data number of blocks, and the information of the freed data blocks is written to the Hash array number
According in area.Then, the pending data is written to data corresponding with the freed data blocks in the storage data area
Qu Zhong, so as to complete data write-in, the VerIdx being finally written to the information of the freed data blocks in the index area
In array (data query entrance), in order to which user carries out data query.In this way, completing concurrently writing for the pending data
Enter.
For example, if extracting idle data corresponding with the data number of blocks from the extraction pond of above-mentioned configuration is data
Then the information of data block 0, data block 1 and data block 2 is written to the Hash number by block 0, data block 1 and data block 2
In group data area, and the pending data is written in the storage data area and data block 0, data block 1 and data
In the corresponding data field of block 2, finally the information of data block 0, data block 1 and data block 2 is written in the index area
In VerIdx array.
In alternatively possible example, if concurrent data processing request is data read request, this step is first
Number and the version number of corresponding data block are first obtained from the VerIdx array of the index area according to the cryptographic Hash, then
The corresponding Brief block of information of the data block is obtained from Hash array data area according to the number of the data block, it should
Brief block of information can refer to the block of information for recording the basic summary info of the data block.Next, it is determined that the cryptographic Hash is
Whether the length of the identification number of no pending data identical, described as the cryptographic Hash of the Brief block of information is believed with the Brief
Cease the identification number of block length is identical and the version number of the data block whether version number's phase with the Brief block of information
Together.If the length of the identification number of cryptographic Hash pending data identical, described as the cryptographic Hash of the Brief block of information with
The length of the identification number of the Brief block of information is identical, and the version of the version number of the data block and the Brief block of information
It is number identical, then judge whether the identification number of the data block is identical as the identification number of the Brief block of information.If the data
The identification number of block is identical as the identification number of the Brief block of information, then it is described to from that needs are obtained from the Brief block of information
Reason data are simultaneously read away.
In addition, in above-mentioned deterministic process, if the cryptographic Hash it is different from the cryptographic Hash of the Brief block of information or
The length of the identification number of pending data described in the person number different or described from the length of identification number of the Brief block of information
It is different from the version number of the Brief block of information according to the version number of block, then it is write from the VerIdx array of the index area
The number of one data block and version number, and return execution judge the cryptographic Hash whether the Hash with the Brief block of information
Be worth the identification number of identical, the described pending data length it is whether identical as the length of identification number of the Brief block of information with
And the version number of the data block whether the step identical as the version number of the Brief block of information.
In alternatively possible example, if concurrent data processing request is data removal request, this step can
Quasi- mark data block corresponding in the VerIdx array of the index area is carried out deletion label according to the cryptographic Hash, and
It is detecting there is currently when the data read request to the quasi- mark data block, if detecting the quasi- mark data block
It is marked in the presence of deleting, then returns and deleted prompt information.If detecting that there is currently no to the quasi- mark data block
When data read request, then by the corresponding Brief block of information of the quasi- mark data block in Hash array data area into
Row delete label, then by the version number of the Brief block of information be labeled as 0, and to the data in the Brief block of information into
Row recycling.In this way, being carried out when actually deleting by quasi- mark data block corresponding in the VerIdx array by the index area
Label is deleted, in this way when needing to read the quasi- mark data block there are data read request, if detecting the quasi- reference numerals
Exist according to block and delete label, then the information returned in the quasi- mark data block has been deleted, and reading data is no longer performed the next step
Step.
Fig. 4 shows provided by the embodiments of the present application for realizing the above-mentioned electronic equipment without lock shared drive processing method
100 schematic diagram, the electronic equipment 100 can be for providing main broadcaster's server of service.In the present embodiment, which is set
Standby 100 may include storage medium 110, processor 120 and without lock shared drive processing unit 130.
Wherein, processor 120 can be a general central processing unit (Central Processing Unit,
CPU), microprocessor, application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC),
Or it is one or more for controlling the integrated of the programs execution without lock shared drive processing method of above method embodiment offer
Circuit.
Storage medium 110 can be ROM or can store the other kinds of static storage device of static information and instruction,
RAM or the other kinds of dynamic memory that can store information and instruction, are also possible to the read-only storage of electric erazable programmable
Device (Electrically Erasable Programmabler-Only Memory, EEPROM), CD-ROM
(Compactdisc Read-Only Memory, CD-ROM) or other optical disc storages, optical disc storage (including compression optical disc, swash
Optical disc, optical disc, Digital Versatile Disc, Blu-ray Disc etc.), magnetic disk storage medium or other magnetic storage apparatus or can use
In carry or storage have instruction or data structure form desired program code and can by computer access it is any its
His medium, but not limited to this.Storage medium 110, which can be, to be individually present, and is connected by communication bus with processor 120.It deposits
Storage media 110 can also be integrated with processor.Wherein, storage medium 110, which is used to store, executes answering for application scheme
With program code, such as without lock shared drive processing unit 130 shown in Fig. 4, and execution is controlled by processor 120.Place
Reason device 120 is for executing the application code stored in storage medium 110, such as without lock shared drive processing unit 130, with
Execute above method embodiment without lock shared drive processing method.
The application can be drawn according to above method embodiment to without the lock progress of shared drive processing unit 130 functional module
Point, for example, each functional module of each function division can be corresponded to, two or more functions can also be integrated in one
In a processing module.Above-mentioned integrated module both can take the form of hardware realization, can also be using software function module
Form is realized.It should be noted that be schematical, only a kind of logical function partition to the division of module in the application,
There may be another division manner in actual implementation.For example, the functional module each using corresponding each function division the case where
Under, shown in Fig. 4 is a kind of schematic device without lock shared drive processing unit 130.In detail, no lock shared drive processing
Device 130 may include data configuration module 131, computing module 132, memory mapping block 133, address logging modle 134 and
Distribution module 135 separately below explains the function of each functional module without lock shared drive processing unit 130 in detail
It states.
Data configuration module 131 configures the memory configurations data of the data service, institute for being directed to each data service
Stating memory configurations data includes required data block size, data number of blocks and shared memory file path.It is appreciated that should
Data configuration module 131 can be used for executing above-mentioned steps S110, and the detailed implementation about the data configuration module 131 can
With referring to above-mentioned to the related content of step S110.
Computing module 132, for being total to needed for calculating the data service according to the data block size and data number of blocks
Enjoy memory headroom.It is appreciated that the computing module 132 can be used for executing above-mentioned steps S120, about the computing module 132
Detailed implementation is referred to above-mentioned to the related content of step S120.
Memory mapping block 133, for opening shared memory file according to the shared memory file path, and according to institute
Shared memory space is stated, the shared memory file is mapped in shared drive region by preset memory mapping mode,
And obtain the mapping return value after the completion of mapping.It is appreciated that the memory mapping block 133 can be used for executing above-mentioned steps
S130, the detailed implementation about the memory mapping block 133 are referred to above-mentioned to the related content of step S130.
Address logging modle 134, for obtaining corresponding in the shared drive region reflect according to the mapping return value
The head pointer address and offset record pointer address in region are penetrated, and by the head pointer address and deviates record pointer address
It is recorded in the memory of the corresponding application program of the data service.It is appreciated that the address logging modle 134 can be used for holding
Row above-mentioned steps S140, the detailed implementation about the address logging modle 134 are referred to above-mentioned related to step S140
Content.
Distribution module 135 is described shared after the atom map to shared drive region progress auxiliary data
Region of memory distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete to be somebody's turn to do
Data service is handled without lock shared drive.It is appreciated that the distribution module 135 can be used for executing above-mentioned steps S150, close
It is referred in the detailed implementation of the distribution module 135 above-mentioned to the related content of step S150.
Due to it is provided by the embodiments of the present application without lock shared drive processing unit 130 be it is above-mentioned without lock shared drive processing side
Another way of realization of method, and can be used for executing provided by above-described embodiment without lock without lock shared drive processing unit 130
Shared drive processing method, therefore it can be obtained technical effect can refer to above method embodiment, details are not described herein.
Further, based on the same inventive concept, the embodiment of the present application also provides a kind of computer readable storage medium,
It is stored with computer program on the computer readable storage medium, which executes above-mentioned no lock when being run by processor
The step of shared drive processing method.
Specifically, which can be general storage medium, such as mobile disk, hard disk, on the storage medium
Computer program when being run, be able to carry out above-mentioned without lock shared drive processing method.
The embodiment of the present application be referring to according to the method for the embodiment of the present application, equipment (electronic equipment 100 of such as Fig. 4) and
The flowchart and/or the block diagram of computer program product describes.It should be understood that flow chart can be realized by computer program instructions
And/or the knot of the process and/or box in each flow and/or block and flowchart and/or the block diagram in block diagram
It closes.These computer program instructions be can provide to general purpose computer, special purpose computer, Embedded Processor or other programmable numbers
According to the processor of processing equipment to generate a machine, so that passing through the processing of computer or other programmable data processing devices
The instruction that device executes generates for realizing in one box of one or more flows of the flowchart and/or block diagram or multiple sides
The device for the function of being specified in frame.
Although the application is described in conjunction with each embodiment herein, however, implementing the application claimed
In the process, those skilled in the art are by checking the attached drawing, disclosure and the appended claims, it will be appreciated that and it is real
Other variations of the existing open embodiment.In the claims, one word of " comprising " is not excluded for other components or step,
"a" or "an" is not excluded for multiple situations.Single processor or other units may be implemented to enumerate in claim several
Item function.Mutually different has been recited in mutually different dependent certain measures, it is not intended that these measures cannot group close
To generate good effect.
The above, the only various embodiments of the application, but the protection scope of the application is not limited thereto, it is any
Those familiar with the art within the technical scope of the present application, can easily think of the change or the replacement, and should all contain
Lid is within the scope of protection of this application.Therefore, the protection scope of the application shall be subject to the protection scope of the claim.
Claims (11)
1. a kind of without lock shared drive processing method, which is characterized in that be applied to electronic equipment, which comprises
For each data service, the memory configurations data of the data service are configured, the memory configurations data include required
Data block size, data number of blocks and shared memory file path；
Shared memory space needed for calculating the data service according to the data block size and data number of blocks；
Shared memory file is opened according to the shared memory file path, and according to the shared memory space, by default
Memory mapping mode the shared memory file is mapped in shared drive region, and obtain mapping after the completion of mapping return
Return value；
The head pointer address of corresponding mapping area and partially is obtained in the shared drive region according to the mapping return value
It moves record pointer address, and by the head pointer address and deviates record pointer address that the data service is recorded is corresponding
In the memory of application program；
To the shared drive region carry out auxiliary data atom map after, be the shared drive region distribution index area,
Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the shared without lock of the data service
Memory processing.
2. according to claim 1 without lock shared drive processing method, which is characterized in that described to the shared section key
Domain carries out the step of atom map of auxiliary data, comprising:
Generate the remark information area in the shared drive region；
The position configuration flag value of first character section in the remark information area；
The position of second byte in the remark information area configures current data capacity；
The position configuration data block size of third byte in the remark information area；
The position configuration data number of blocks of the 4th byte in the remark information area；
Respectively by the int type pointer of four atoms in the memory of the application program be respectively directed to the first character section,
The data word locking nub of second byte, the third byte and the 4th byte data, to pass through four originals
The int type pointer type of son controls the first character section, second byte, the third byte and the 4th respectively
A byte is read and write without lock.
3. according to claim 1 without lock shared drive processing method, which is characterized in that described is the shared section key
Hash array data area, extract the step of atomic data structure of pond and storage data area in domain distribution index area, comprising:
For the shared drive region distribution index area, and distribution is used for multiple VerIdx of data directory in the index area
Array and check information list are stored with the corresponding cryptographic Hash of number of each data block in the VerIdx array, described
Check information list includes after the storage byte obtained according to the quantity of Hash bucket and the storage byte add 16 bytes
Extended field；
Hash array data area is distributed for the shared drive region, and atom map is carried out to Hash array data area,
The metadata of each data block is recorded by data link table, wherein the metadata is stored by atom shaping, the member
Data include data block length, data block version number, data block identifier, data block cryptographic Hash, data block deletion label and number
According to the combination of one or more of block reference count；
It is distributed for the shared drive region and extracts pond, next can be written into is directed toward by 64 atom pointers in the extraction pond
Region of memory, and record the position of a data block, had recorded in the corresponding atom shaping pointer alignment in the extraction pond
The identification information of next data block of global data block information and each data block；
Storage data area is distributed for the shared drive region, the size of the storage data area is equal to the quantity of the Hash bucket
With the product of size.
4. according to claim 1 without lock shared drive processing method described in any one of -3, which is characterized in that the method
Further include:
When receiving the concurrent data processing request for data service, request is handled according to the concurrent data and is obtained wait locate
Manage the identification number of data；
The corresponding cryptographic Hash of the identification number is calculated, and corresponding behaviour is executed to the pending data according to the cryptographic Hash
Make.
5. according to claim 4 without lock shared drive processing method, which is characterized in that if concurrent data processing is asked
The step of asking as data write request, then corresponding operation being executed to the pending data according to the cryptographic Hash, comprising:
Data number of blocks needed for calculating the pending data according to the cryptographic Hash；
Extract corresponding with data number of blocks freed data blocks from the extraction pond of configuration, and by the freed data blocks
Information is written in Hash array data area；
The pending data is written in the storage data area in data field corresponding with the freed data blocks；
The information of the freed data blocks is written in the VerIdx array in the index area.
6. according to claim 4 without lock shared drive processing method, which is characterized in that if concurrent data processing is asked
The step of asking as data read request, then corresponding operation being executed to the pending data according to the cryptographic Hash, comprising:
Number and the version number of corresponding data block are obtained from the VerIdx array of the index area according to the cryptographic Hash；
The corresponding Brief information of the data block is obtained from Hash array data area according to the number of the data block
Block；
Judge the identification number of cryptographic Hash pending data whether identical, described as the cryptographic Hash of the Brief block of information
The version number of length data block whether identical and described with the length of identification number of the Brief block of information whether with it is described
The version number of Brief block of information is identical；
If the length of the identification number of cryptographic Hash pending data identical, described as the cryptographic Hash of the Brief block of information
It is identical as the length of identification number of the Brief block of information, and the version of the version number of the data block and the Brief block of information
This number identical, then judges whether the identification number of the data block is identical as the identification number of the Brief block of information；
If the identification number of the data block is identical as the identification number of the Brief block of information, from the Brief block of information
It obtains and needs the pending data and read away.
7. according to claim 6 without lock shared drive processing method, which is characterized in that the method also includes:
If the identification number of the cryptographic Hash pending data different or described from the cryptographic Hash of the Brief block of information
The version number of the length data block different or described from the length of identification number of the Brief block of information and the Brief information
The version number of block is different, then the number and version number of a data block are write from the VerIdx array of the index area, and
Return to the mark for executing and judging cryptographic Hash pending data whether identical, described as the cryptographic Hash of the Brief block of information
Number length data block whether identical and described with the length of identification number of the Brief block of information version number whether with institute
State the identical step of the version number of Brief block of information.
8. according to claim 4 without lock shared drive processing method, which is characterized in that if concurrent data processing is asked
The step of asking as data removal request, then corresponding operation being executed to the pending data according to the cryptographic Hash, comprising:
Quasi- mark data block corresponding in the VerIdx array of the index area is subjected to deletion label according to the cryptographic Hash；
It is detecting there is currently when the data read request to the quasi- mark data block, if detecting the quasi- reference numerals
Exist according to block and delete label, then returns and deleted prompt information；
It is detecting there is currently no when the data read request to the quasi- mark data block, by Hash array data area
In the corresponding Brief block of information of the quasi- mark data block carry out deletion label；
The version number of the Brief block of information is labeled as 0, and the data in the Brief block of information are recycled.
9. a kind of without lock shared drive processing unit, which is characterized in that be applied to electronic equipment, described device includes:
Data configuration module configures the memory configurations data of the data service, the memory is matched for being directed to each data service
Setting data includes required data block size, data number of blocks and shared memory file path；
Computing module, for the sky of shared drive needed for calculating the data service according to the data block size and data number of blocks
Between；
Memory mapping block, for opening shared memory file according to the shared memory file path, and according to described shared
The shared memory file is mapped in shared drive region by preset memory mapping mode, and obtained by memory headroom
Mapping return value after the completion of mapping；
Address logging modle, for obtaining corresponding mapping area in the shared drive region according to the mapping return value
Head pointer address and offset record pointer address, and institute is recorded in the head pointer address and offset record pointer address
In the memory for stating the corresponding application program of data service；
Distribution module is the shared section key after the atom map to shared drive region progress auxiliary data
Domain distribution index area, Hash array data area, the atomic data structure for extracting pond and storage data area, to complete the data industry
Business is handled without lock shared drive.
10. a kind of electronic equipment, which is characterized in that the electronic equipment includes machine readable storage medium and processor, described
Machine readable storage medium is stored with machine-executable instruction, and the processor makes institute when executing the machine-executable instruction
Electronic equipment is stated to realize described in any one of claim 1-8 without lock shared drive processing method.
11. a kind of readable storage medium storing program for executing, which is characterized in that be stored with machine-executable instruction, institute in the readable storage medium storing program for executing
Machine-executable instruction is stated to be performed described in any one of realization claim 1-8 without lock shared drive processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591481.0A CN110287044B (en) | 2019-07-02 | 2019-07-02 | Lock-free shared memory processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591481.0A CN110287044B (en) | 2019-07-02 | 2019-07-02 | Lock-free shared memory processing method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110287044A true CN110287044A (en) | 2019-09-27 |
| CN110287044B CN110287044B (en) | 2021-08-03 |
Family
ID=68020275
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910591481.0A Active CN110287044B (en) | 2019-07-02 | 2019-07-02 | Lock-free shared memory processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110287044B (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110674052A (en) * | 2019-09-30 | 2020-01-10 | 广州虎牙科技有限公司 | Memory management method, server and readable storage medium |
| CN110933448A (en) * | 2019-11-29 | 2020-03-27 | 广州市百果园信息技术有限公司 | Live list service system and method |
| CN111092865A (en) * | 2019-12-04 | 2020-05-01 | 全球能源互联网研究院有限公司 | Security event analysis method and system |
| CN112306695A (en) * | 2020-11-19 | 2021-02-02 | 中国民航信息网络股份有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN112328435A (en) * | 2020-12-07 | 2021-02-05 | 武汉绿色网络信息服务有限责任公司 | Method, device, equipment and storage medium for backing up and recovering target data |
| CN112463333A (en) * | 2020-12-03 | 2021-03-09 | 北京浪潮数据技术有限公司 | Data access method, device and medium based on multithreading concurrency |
| CN112463306A (en) * | 2020-12-03 | 2021-03-09 | 南京机敏软件科技有限公司 | Method for sharing disk data consistency in virtual machine |
| CN112732194A (en) * | 2021-01-13 | 2021-04-30 | 同盾科技有限公司 | Irregular data storage method, device and storage medium |
| CN112947856A (en) * | 2021-02-05 | 2021-06-11 | 彩讯科技股份有限公司 | Memory data management method and device, computer equipment and storage medium |
| CN113127415A (en) * | 2019-12-31 | 2021-07-16 | 浙江宇视科技有限公司 | Method, device, medium and electronic equipment for processing real-time stream file |
| CN113157199A (en) * | 2020-01-22 | 2021-07-23 | 阿里巴巴集团控股有限公司 | Snapshot occupation space calculation method and device, electronic equipment and storage medium |
| CN113194266A (en) * | 2021-04-28 | 2021-07-30 | 深圳迪乐普数码科技有限公司 | Image sequence frame real-time rendering method and device, computer equipment and storage medium |
| CN113535437A (en) * | 2021-08-03 | 2021-10-22 | 上海节卡机器人科技有限公司 | Module data interaction method of robot, electronic equipment and storage medium |
| CN113688068A (en) * | 2021-10-25 | 2021-11-23 | 支付宝(杭州)信息技术有限公司 | Graph data loading method and device |
| CN113778674A (en) * | 2021-08-31 | 2021-12-10 | 上海弘积信息科技有限公司 | Lock-free implementation method of load balancing equipment configuration management under multi-core |
| CN114356589A (en) * | 2021-12-09 | 2022-04-15 | 北京华云安信息技术有限公司 | Data storage and reading method, device and equipment for multiple writers and multiple readers |
| CN118034610A (en) * | 2024-04-07 | 2024-05-14 | 深圳市纽创信安科技开发有限公司 | Key data processing method applied to memory, device and equipment |
| WO2024109068A1 (en) * | 2022-11-25 | 2024-05-30 | 惠州市德赛西威智能交通技术研究院有限公司 | Program monitoring method and apparatus, and electronic device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101582092A (en) * | 2009-06-12 | 2009-11-18 | 中兴通讯股份有限公司 | Method and device for realizing the store of date in memory |
| US20130262426A1 (en) * | 2010-02-12 | 2013-10-03 | Huawei Technologies Co., Ltd. | Method and apparatus for accessing database and database application system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN103593485A (en) * | 2013-12-04 | 2014-02-19 | 网易传媒科技（北京）有限公司 | Method and device for achieving real-time operation on data base |
| CN105975407A (en) * | 2016-03-22 | 2016-09-28 | 华为技术有限公司 | Mapping method of memory address, and equipment |
| US20170262172A1 (en) * | 2014-11-28 | 2017-09-14 | Huawei Technologies Co., Ltd. | File Access Method and Apparatus, and Storage Device |
| CN109298935A (en) * | 2018-09-06 | 2019-02-01 | 华泰证券股份有限公司 | A kind of method and application of the multi-process single-write and multiple-read without lock shared drive |
-
2019
- 2019-07-02 CN CN201910591481.0A patent/CN110287044B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101582092A (en) * | 2009-06-12 | 2009-11-18 | 中兴通讯股份有限公司 | Method and device for realizing the store of date in memory |
| US20130262426A1 (en) * | 2010-02-12 | 2013-10-03 | Huawei Technologies Co., Ltd. | Method and apparatus for accessing database and database application system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN103593485A (en) * | 2013-12-04 | 2014-02-19 | 网易传媒科技（北京）有限公司 | Method and device for achieving real-time operation on data base |
| US20170262172A1 (en) * | 2014-11-28 | 2017-09-14 | Huawei Technologies Co., Ltd. | File Access Method and Apparatus, and Storage Device |
| CN105975407A (en) * | 2016-03-22 | 2016-09-28 | 华为技术有限公司 | Mapping method of memory address, and equipment |
| CN109298935A (en) * | 2018-09-06 | 2019-02-01 | 华泰证券股份有限公司 | A kind of method and application of the multi-process single-write and multiple-read without lock shared drive |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110674052A (en) * | 2019-09-30 | 2020-01-10 | 广州虎牙科技有限公司 | Memory management method, server and readable storage medium |
| CN110933448A (en) * | 2019-11-29 | 2020-03-27 | 广州市百果园信息技术有限公司 | Live list service system and method |
| CN111092865A (en) * | 2019-12-04 | 2020-05-01 | 全球能源互联网研究院有限公司 | Security event analysis method and system |
| CN113127415A (en) * | 2019-12-31 | 2021-07-16 | 浙江宇视科技有限公司 | Method, device, medium and electronic equipment for processing real-time stream file |
| CN113127415B (en) * | 2019-12-31 | 2024-02-27 | 浙江宇视科技有限公司 | Real-time stream file processing method, device, medium and electronic equipment |
| CN113157199A (en) * | 2020-01-22 | 2021-07-23 | 阿里巴巴集团控股有限公司 | Snapshot occupation space calculation method and device, electronic equipment and storage medium |
| CN112306695A (en) * | 2020-11-19 | 2021-02-02 | 中国民航信息网络股份有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN112463333A (en) * | 2020-12-03 | 2021-03-09 | 北京浪潮数据技术有限公司 | Data access method, device and medium based on multithreading concurrency |
| CN112463306A (en) * | 2020-12-03 | 2021-03-09 | 南京机敏软件科技有限公司 | Method for sharing disk data consistency in virtual machine |
| CN112328435A (en) * | 2020-12-07 | 2021-02-05 | 武汉绿色网络信息服务有限责任公司 | Method, device, equipment and storage medium for backing up and recovering target data |
| CN112328435B (en) * | 2020-12-07 | 2023-09-12 | 武汉绿色网络信息服务有限责任公司 | Method, device, equipment and storage medium for backing up and recovering target data |
| CN112732194A (en) * | 2021-01-13 | 2021-04-30 | 同盾科技有限公司 | Irregular data storage method, device and storage medium |
| CN112732194B (en) * | 2021-01-13 | 2022-08-19 | 同盾科技有限公司 | Irregular data storage method, device and storage medium |
| CN112947856A (en) * | 2021-02-05 | 2021-06-11 | 彩讯科技股份有限公司 | Memory data management method and device, computer equipment and storage medium |
| CN112947856B (en) * | 2021-02-05 | 2024-05-03 | 彩讯科技股份有限公司 | Memory data management method and device, computer equipment and storage medium |
| CN113194266A (en) * | 2021-04-28 | 2021-07-30 | 深圳迪乐普数码科技有限公司 | Image sequence frame real-time rendering method and device, computer equipment and storage medium |
| CN113535437A (en) * | 2021-08-03 | 2021-10-22 | 上海节卡机器人科技有限公司 | Module data interaction method of robot, electronic equipment and storage medium |
| CN113535437B (en) * | 2021-08-03 | 2023-04-07 | 节卡机器人股份有限公司 | Module data interaction method of robot, electronic equipment and storage medium |
| CN113778674A (en) * | 2021-08-31 | 2021-12-10 | 上海弘积信息科技有限公司 | Lock-free implementation method of load balancing equipment configuration management under multi-core |
| CN113688068B (en) * | 2021-10-25 | 2022-02-15 | 支付宝(杭州)信息技术有限公司 | Graph data loading method and device |
| CN113688068A (en) * | 2021-10-25 | 2021-11-23 | 支付宝(杭州)信息技术有限公司 | Graph data loading method and device |
| CN114356589A (en) * | 2021-12-09 | 2022-04-15 | 北京华云安信息技术有限公司 | Data storage and reading method, device and equipment for multiple writers and multiple readers |
| CN114356589B (en) * | 2021-12-09 | 2024-04-12 | 北京华云安信息技术有限公司 | Multi-writer and multi-reader data storage and reading method, device and equipment |
| WO2024109068A1 (en) * | 2022-11-25 | 2024-05-30 | 惠州市德赛西威智能交通技术研究院有限公司 | Program monitoring method and apparatus, and electronic device and storage medium |
| CN118034610A (en) * | 2024-04-07 | 2024-05-14 | 深圳市纽创信安科技开发有限公司 | Key data processing method applied to memory, device and equipment |
| CN118034610B (en) * | 2024-04-07 | 2024-07-02 | 深圳市纽创信安科技开发有限公司 | Key data processing method applied to memory, device and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110287044B (en) | 2021-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110287044A (en) | Without lock shared drive processing method, device, electronic equipment and readable storage medium storing program for executing | |
| US6789094B2 (en) | Method and apparatus for providing extended file attributes in an extended attribute namespace | |
| US20170075718A1 (en) | Method and apparatus of accessing data of virtual machine | |
| US11182083B2 (en) | Bloom filters in a flash memory | |
| US20160267132A1 (en) | Abstraction layer between a database query engine and a distributed file system | |
| US11093471B2 (en) | Large range lookups for Bϵ-tree | |
| US10983909B2 (en) | Trading off cache space and write amplification for Bε-trees | |
| US20110302365A1 (en) | Storage system using a rapid storage device as a cache | |
| CN109445685B (en) | User mode file system processing method | |
| JPH10501086A (en) | Storage plane organization and storage system based thereon | |
| CN109947363A (en) | Data caching method of distributed storage system | |
| CN107817946B (en) | Method and device for reading and writing data of hybrid storage device | |
| CN107239569A (en) | A kind of distributed file system subtree storage method and device | |
| CN107402821A (en) | Access control method, device and the equipment of shared resource | |
| CN106407355A (en) | Data storage method and device | |
| CN107408132B (en) | Method and system for moving hierarchical data objects across multiple types of storage | |
| CN106980665A (en) | Data dictionary implementation method, device and data dictionary management system | |
| CN100458792C (en) | Method and data processing system for managing a mass storage system | |
| CN111177143A (en) | Key value data storage method and device, storage medium and electronic equipment | |
| CN109460406A (en) | Data processing method and device | |
| CN105260139B (en) | A kind of disk management method and system | |
| CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
| CN113204520B (en) | Remote sensing data rapid concurrent read-write method based on distributed file system | |
| CN110851407A (en) | Data distributed storage system and method | |
| US11487731B2 (en) | Read iterator for pre-fetching nodes of a B-tree into memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |