CN103593485B - The method and apparatus for realizing database real-time operation - Google Patents
The method and apparatus for realizing database real-time operation Download PDFInfo
- Publication number
- CN103593485B CN103593485B CN201310648091.5A CN201310648091A CN103593485B CN 103593485 B CN103593485 B CN 103593485B CN 201310648091 A CN201310648091 A CN 201310648091A CN 103593485 B CN103593485 B CN 103593485B
- Authority
- CN
- China
- Prior art keywords
- data
- memory block
- database
- operating system
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An aspect of of the present present invention provides a kind of method for realizing database real-time operation.The method includes：Database is created in operating system shared drive on the server；In response to the request of the operation to the database, the targeted memory block of the operation is determined from the operating system shared drive；Pair determine memory block operate.By creating database, " real-time operation " that the method for the present invention can be with effective guarantee to database in operating system shared drive on the server.Further, decrease to the system resource shared by database real-time operation process.Additionally, another aspect provides a kind of equipment for realizing database real-time operation, for example, the software installed on server.
Description
Technical field
Embodiments of the present invention are related to database field, more specifically, embodiments of the present invention are related to realize data
The method and apparatus of storehouse real-time operation.
Background technology
This part is it is intended that the embodiments of the present invention stated in claims provide background or context.Herein
Description may include the concept that can be probed into, but be not necessarily the concept for having contemplated that or having probed into before.Therefore, unless
This points out that the content otherwise described in this part is not existing skill for the description and claims of this application
Art, and not because just recognizing it is prior art in being included in this part.
In a typical real-time price quotations application system, such as in stock market system, wish to look into positioned at the user of client
See newest price data.In such a system, it is necessary to use a database purchase price data.It is also desirable to real
Now to " the real-time operation " of price data.
So-called " real-time operation " is generally referred to, rapidly will be old to replace in newest price data write into Databasce
Price data, and rapidly price data read out from database be supplied to client so that the data of reading
Most matched between the data of write-in.
In the prior art, there is some type of database.Such as, relevant database and memory database.Its
In, relevant database is to save the data in the database operated on disk, and memory database is to preserve data
The database operated in internal memory.
The content of the invention
But, the present inventor has found in research process, in the prior art, either relevant database, or it is interior
Deposit data storehouse, for write-in process and reading process, is required for the memory headroom of oneself.When write-in process write-in
That data are transferred in internal memory or disk from the memory headroom of oneself during data, when read process read data when, be by
Data are transferred in the memory headroom of oneself from internal memory or disk.
Therefore according to prior art, when being operated to above-mentioned database, due to needing the transfer process of experience data,
That is, data are transferred to the process of internal memory or disk from memory headroom, and data are transferred to memory headroom from internal memory or disk
Process, and transfer process needs to take certain hour in itself, therefore, it is impossible to effective guarantee " real-time operation " requirement.
In addition, the memory headroom used in transfer process also can occupying system resources.
Therefore, a kind of improved method and apparatus for realizing database real-time operation is highly desirable to, to solve prior art
In cannot effective guarantee " real-time operation " this technical problem to the operation of database.Further, it is also desirable to reduce right
System resource shared by database real-time operation process.
In the present context, embodiments of the present invention expectation offer is a kind of realizes the method for database real-time operation and sets
It is standby.
In the first aspect of embodiment of the present invention, there is provided a kind of method for realizing database real-time operation, including：
Database is created in operating system shared drive on the server；In response to the request of the operation to the database, from institute
State the memory block for determining that the operation is targeted in operating system shared drive；Pair determine memory block operate.
Preferably, the operation includes writing data manipulation and read data operation.
Preferably, methods described also includes：Data in the operating system shared drive are mapped to the server
On disk in.
Preferably, methods described also includes：Data in the operating system shared drive are mapped to the server
On operating system memory file in.
Preferably, for create the database the operating system shared drive memory address during running not
Discharged and reclaimed by operating system.
Preferably, methods described also includes：When occupied memory block is idle in the operating system shared drive, will
The idle corresponding memory address of memory block is put into the presets list.
Preferably, the corresponding memory block of memory address in described the presets list can be reused for preserving the data
The data in storehouse.
Preferably, when the number of the memory address in described the presets list is more than or equal to 2, in described the presets list
The corresponding memory block of memory address be just reused for preserving the data of the database.
Preferably, the corresponding memory address of memory block by the free time is put into the presets list includes：According to memory block
Size, the corresponding memory address of memory block is put into corresponding the presets list so that size identical memory block correspondence
Memory address put into same the presets list, the corresponding memory address of memory block of different sizes is put into different pre-
If in list.
Preferably, the memory block of described pair of determination carries out operation includes：According to the internal memory that HashMap data structures pair determine
Block is operated.
Preferably, when the operation is for write-in data manipulation, the memory block of determination is write data into using single thread
In.
Preferably, when the operation is for read data operation, it is possible to use multithreading reads from the memory block for determining
Data.
In the second aspect of embodiment of the present invention, there is provided a kind of equipment for realizing database real-time operation, including：
Database creation unit, for creating database in operating system shared drive on the server；Memory block determining unit, uses
In the request in response to the operation to the database, operation requests institute pin is determined from the operating system shared drive
To memory block；Operating unit, operates for pair memory block for determining.
Preferably, the operation includes writing data manipulation and read data operation.
Preferably, the equipment also includes：First map unit, for by the data in the operating system shared drive
It is mapped in the disk on the server.
Preferably, the equipment also includes：Second map unit, for by the data in the operating system shared drive
It is mapped in the operating system memory file on the server.
Preferably, for create the database the operating system shared drive memory address during running not
Discharged and reclaimed by operating system.
Preferably, the equipment also includes：Memory block management unit, for when occupied in the operating system shared drive
When memory block is idle, the idle corresponding memory address of memory block is put into the presets list.
Preferably, the corresponding memory block of memory address in described the presets list can be reused for preserving the data
The data in storehouse.
Preferably, when the number of the memory address in described the presets list is more than or equal to 2, in described the presets list
The corresponding memory block of memory address be just reused for preserving the data of the database.
Preferably, the memory block management unit is configured to：According to the size of memory block, by the corresponding internal memory ground of memory block
Location is put into corresponding the presets list so that the corresponding memory address of size identical memory block puts into same default row
In table, the corresponding memory address of memory block of different sizes is put into different the presets lists.
Preferably, the operating unit is configured to：The memory block determined according to HashMap data structures pair is operated.
Preferably, when the operation is for write-in data manipulation, the operating unit is configured to：Using single thread by data
It is written in the memory block of determination.
Preferably, when the operation is for read data operation, the operating unit concrete configuration is：Can use multi-thread
Journey reads data from the memory block for determining.
In embodiments of the present invention, because database is created in operating system shared drive on the server,
Therefore, write-in process can directly be operated as being operated to local memory with reading process to database, and nothing
Data transfer, the time consumed such that it is able to save factor data transfer process and system resource need to be carried out, and then effectively
Ensure and " the real-time operation " of database is required.
In addition, idle memory block is reclaimed and reused by the presets list, the product of memory fragmentation is not only avoid
It is raw, but also reached the controllable purpose of memory block.
Also in addition, avoiding read/write conflict by the way of without lock, system architecture is simplified, substantially increase systematic function.
Brief description of the drawings
Detailed description below, above-mentioned and other mesh of exemplary embodiment of the invention are read by reference to accompanying drawing
, feature and advantage will become prone to understand.In the accompanying drawings, if showing by way of example, and not by way of limitation of the invention
Dry implementation method, wherein：
Fig. 1 shows the block diagram of the exemplary computer system for being adapted for carrying out embodiment of the present invention；
Fig. 2 schematically shows an exemplary application scene of embodiments of the present invention；
Fig. 3 schematically shows the flow of the method for realizing database real-time operation according to the embodiment of the present invention
Figure；
Fig. 4 shows the schematic diagram of the memory block management according to one embodiment of the present invention；
Fig. 5 shows the schematic diagram of the presets list according to one embodiment of the present invention；
Fig. 6 shows the schematic diagram of the HashMap data structures according to one embodiment of the present invention；
Fig. 7 schematically shows the structural framing figure of the equipment according to one embodiment of the present invention.
In the accompanying drawings, identical or corresponding label represents identical or corresponding part.
Specific embodiment
Principle of the invention and spirit are described below with reference to some illustrative embodiments.It should be appreciated that providing this
A little implementation methods are used for the purpose of better understood when those skilled in the art and then realizing the present invention, and not with any
Mode limits the scope of the present invention.Conversely, these embodiments are provided so that the disclosure is more thorough and complete, and energy
It is enough that the scope of the present disclosure is intactly conveyed into those skilled in the art.
Fig. 1 shows the block diagram of the exemplary computer system 100 for being adapted for carrying out embodiment of the present invention.As shown in figure 1, meter
Calculation system 100 can include：CPU（CPU）101st, random access memory（RAM）102nd, read-only storage（ROM）
103rd, system bus 104, hard disk controller 105, KBC 106, serial interface controller 107, parallel interface controller
108th, display controller 109, hard disk 110, keyboard 111, serial peripheral equipment 112, concurrent peripheral equipment 113 and display 114.
In these equipment, what is coupled with system bus 104 has CPU101, RAM102, ROM103, hard disk controller 105, KBC
106th, serial interface controller 107, parallel interface controller 108 and display controller 109.Hard disk 110 and hard disk controller 105
Coupling, keyboard 111 is coupled with KBC 106, and serial peripheral equipment 112 is coupled with serial interface controller 107, parallel outer
Portion's equipment 113 is coupled with parallel interface controller 108, and display 114 is coupled with display controller 109.It should be appreciated that figure
Structured flowchart described in 1 is used for the purpose of the purpose of example, without limiting the scope of the present invention.In some cases, may be used
To increase or decrease some equipment as the case may be.
Art technology technical staff knows that embodiments of the present invention can be implemented as a kind of system, method or calculating
Machine program product.Therefore, the disclosure can be implemented as following form, i.e.,：Complete hardware, complete software（Including solid
Part, resident software, microcode etc.）, or the form that hardware and software is combined, referred to generally herein as " circuit ", " module " or " it is
System ".Additionally, in certain embodiments, the present invention is also implemented as the calculating in one or more computer-readable mediums
The form of machine program product, includes computer-readable program code in the computer-readable medium.
Can be using any combination of one or more computer-readable media.Computer-readable medium can be calculated
Machine readable signal medium or computer-readable recording medium.Computer-readable recording medium for example may be, but not limited to,
The system of electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, device or device, or it is any more than combination.Computer-readable
The more specifically example of storage medium（Non-exhaustive examples）Can for example include：Electrical connection with one or more wires, just
Take formula computer disk, hard disk, random access memory（RAM）, read-only storage (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only storage (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In this document, computer-readable recording medium can be it is any comprising or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device and use or in connection.
Computer-readable signal media can include the data-signal propagated in a base band or as a carrier wave part,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but not limit
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Read any computer-readable medium beyond storage medium, the computer-readable medium can send, propagates or transmit to be used for
Used by instruction execution system, device or device or program in connection.
The program code included on computer-readable medium can use any appropriate medium to transmit, including but not limited to without
Line, electric wire, optical cable, RF etc., or above-mentioned any appropriate combination.
Computer for performing present invention operation can be write with one or more programming language or its combination
Program code, described program design language includes object oriented program language-such as Java, Smalltalk, C++,
Also including conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
Fully perform on the user computer, partly perform on the user computer, performed as an independent software kit, portion
Part on the user computer is divided to perform on the remote computer or performed on remote computer or server completely.
It is related in the situation of remote computer, remote computer can be by the network of any kind（Including LAN (LAN) or wide area
Net (WAN)）Subscriber computer is connected to, or, it may be connected to outer computer（For example using ISP come
By Internet connection）.
Below with reference to the flow chart and equipment of the method for the embodiment of the present invention（Or system）Block diagram describe of the invention
Implementation method.It should be appreciated that in each square frame and flow chart and/or block diagram of flow chart and/or block diagram each square frame combination
Can be realized by computer program instructions.These computer program instructions can be supplied to all-purpose computer, special-purpose computer
Or the processor of other programmable data processing units, so as to produce a kind of machine, these computer program instructions are by meter
Calculation machine or other programmable data processing units are performed, and generate work(specified in the square frame realized in flow chart and/or block diagram
The device of energy/operation.
The storage of these computer program instructions can also caused computer or other programmable data processing units
In the computer-readable medium for working in a specific way, so, the instruction stored in computer-readable medium just produces one
It is individual including realizing flow chart and/or block diagram in square frame specified in function/operation command device product.
Computer program instructions can also be loaded into computer, other programmable data processing units or miscellaneous equipment
On so that series of operation steps is performed on computer, other programmable data processing units or miscellaneous equipment, in terms of producing
The process that calculation machine is realized, so that the instruction performed on computer or other programmable devices can be provided and realize flow chart
And/or the process of function/operation specified in the square frame in block diagram.
According to the embodiment of the present invention, it is proposed that a kind of method and apparatus for realizing database real-time operation.
Herein, it is to be understood that any number of elements in accompanying drawing is used to example and unrestricted and any
Name is only used for distinguishing, without any limitation.
Below with reference to some representative embodiments of the invention, principle of the invention and spirit are explained in detail.
Summary of the invention
The inventors discovered that, it is right due to being operated to relevant database of the prior art and memory database
In write-in process and reading process for, be required for the memory headroom of oneself, when write process write-in data when, be by
Data are transferred in internal memory or disk from the memory headroom of oneself, are by data from internal memory when the process of reading reads data
Or be transferred in disk in the memory headroom of oneself, therefore, it is necessary to experience multiple data transfer process could database
Operation.And data transfer process needs to take certain hour in itself, like this, will certainly influence to the real-time of database
Operation, and then cannot " real-time operation " requirement of effective guarantee to database.
In addition, the memory headroom used in data transfer process also can occupying system resources.If entered to database
Can reduce or even avoid data transfer process during row operation, it is possible to the time saved factor data transfer process and consume
And system resource, and then effectively ensure " real-time operation " requirement to database.
After general principle of the invention is described, lower mask body introduces various non-limiting embodiment party of the invention
Formula.
Application scenarios overview
With reference first to Fig. 2, Fig. 2 schematically shows an exemplary application scene of embodiments of the present invention.Its
In, on the operating system shared drive of server 20 create have database 201, the reading process 21 on server 20 in response to
The request of user, reads existing data, there is provided to user from database 201.And the write-in process 22 on server 20 to
Database 201 writes newest data.Server 20 can be Web server, or other kinds of server, for example
APP servers.It will be understood by those skilled in the art that the schematic diagram shown in Fig. 2 be only embodiments of the present invention can be at it
In an example being achieved.The scope of application of embodiment of the present invention is not limited by any aspect of the framework.
Illustrative methods
With reference to the application scenarios of Fig. 2, be described with reference to Figure 3 according to exemplary embodiment of the invention, for reality
The method of existing database real-time operation.It should be noted that above-mentioned application scenarios are for only for ease of understanding spirit of the invention
Shown with principle, embodiments of the present invention are unrestricted in this regard.Conversely, embodiments of the present invention can be answered
For any scene being applicable.
Refer to Fig. 3, that schematically shows according to the embodiment of the present invention realize database real-time operation
The flow chart of method, method specifically can for example include：
Step 301：Database is created in operating system shared drive on the server.
Wherein, shared drive is mainstream operation system（Such as, Linux, Unix and Windows operating system）One for all supporting
System underlying mechanisms are planted, if creating database in shared drive, is read for the data write-in process and data on server
For taking process, when the database for being pointed to shared drive is operated, without extra memory headroom.That is, working as
During write-in process write-in data, in directly writing data into the database of shared drive, when the process of reading reads data,
It is that data are directly read from the database of shared drive.
According to the embodiment of the present invention, first, by write-in process initialization database.When carrying out data library initialization
When, write-in process is first passed through carrys out applying operation system shared drive to operating system transmission mmap system call requests, should
Operating system shared drive can accommodate all of data.
In addition, data are with key-value pair（key-value）Mode be stored in database, key and value all be character string
Type, e.g., content is during stock code 600100 is stored in database：{"symbol":"600100","price":8.0,"
high":8.03,"low":7.79,"time":"2013/08/1215:03:05"...}.Wherein symbol, price, high,
Low, time are key, and 600100,8.0,8.03,7.79,2013/08/1215:03:05 be key symbol, price, high,
The value of low, time.
Step 302：In response to the request of the operation to the database, institute is determined from the operating system shared drive
State the targeted memory block of operation.
Step 303：Pair determine memory block operate.
An implementation method of the invention, the operation includes write-in data manipulation and read data operation.
For example, with reference to shown in Fig. 2, the request of the read data operation to database 201 is sent with user orientation server 20, take
Reading process 21 on business device 20 determines the reading in response to the request of the read data operation from operating system shared drive
The targeted memory block of data manipulation, and further pair memory block for determining is read out data manipulation.Or, on server 20
Write-in process 22 in response to write data manipulation request, the write-in data manipulation institute is determined from operating system shared drive
For memory block, and the memory block for further pair determining carries out data write operation.
Database is created in operating system shared drive, data are stored temporarily in shared drive, once restart
Operating system, the data in database will lose.Therefore, according to the implementation method, server in the prior art is being solved
To the operation of database cannot effective guarantee " real-time operation ", and, solve database real-time operation process and expend system
While the problem of resource, the unstable problem of data is there is also.
In order to further solve the problems, such as that data are unstable, according to the first preferred embodiment, method also includes：By the behaviour
In making the disk that the data in system shared drive are mapped on the server.
So, data are permanently stored in disk, even if the data in causing database because of reboot operation system
Lose, it is also possible to find and recover the data lost from disk again.
In the specific implementation, it is that the data in operating system shared drive are mapped on server by operating system control
Disk in.For example, when the data in shared drive reach some data volume, operating system automatically will reflect data
It is mapped in the disk on server.But, according to the program, the frequency that operating system carries out data mapping is often higher, so that
Excessive I/O operation is produced, substantial amounts of operating system overhead is consumed.
For the I/O operation for not making generation excessive and substantial amounts of operating system overhead, according to replacing, above-mentioned the first is preferred
Another preferred embodiment of mode, method also includes：Data in the operating system shared drive are mapped to the service
In operating system memory file on device.
For example, the data in operating system shared drive are mapped in operating system memory file/dev/shm.If
Data in operating system shared drive are mapped in operating system memory file, the mapping process will not produce any I/O
Operation, thus would not also consume operating system overhead.
In addition, the data being mapped in operating system memory file equally exist the unstable problem of data, in order to solve
This problem, in the operating system memory file being mapped to the data in the operating system shared drive on the server
In after, method can further include：Data in the operating system memory file are backuped on the server
Disk in.
It is emphasized that the data in operating system memory file to be backuped to process in the disk on server not
It is to be controlled by operating system, and can be to be controlled by the process of user, that is to say, that the process of user can be by control
The frequency of data backup, i.e. reduce the frequency of data backup, and then reduce the I/O operation number of times produced by data backup procedure,
Thus decrease operating system overhead.
In order to avoid producing memory fragmentation, the controllable purpose of internal memory is reached, a kind of implementation method of the invention is used
A kind of special shared-memory management mechanism, i.e. in the operating system shared drive for creating the database
Address is deposited not discharged and reclaimed by operating system during running.Based on this, method also includes：In the operating system is shared
When occupied memory block is idle in depositing, the idle corresponding memory address of memory block is put into the presets list.Wherein, it is described
The corresponding memory block of memory address in the presets list can be reused for preserving the data of the database.
In order to avoid the read/write conflict caused due to concurrent reading and writing, a kind of implementation method of the invention, when described
When the number of memory address is more than or equal to 2 in the presets list, the corresponding memory block of memory address in described the presets list
Just it is reused for preserving the data of the database.
In addition, for more convenient and quicker from the corresponding all memory blocks of the memory address in the presets list for being distributed
Find the memory block suitable for being used for data storage every time, a kind of implementation method of the invention, by idle memory block
Corresponding memory address is put into the presets list to be included：According to the size of memory block, the corresponding memory address of memory block is put into
To in corresponding the presets list so that the corresponding memory address of size identical memory block is put into same the presets list,
The corresponding memory address of memory block of different sizes is put into different the presets lists.
For example, being Memory Allocation to be carried out in units of the memory block of fixed size and is returned in operating system shared drive
Receive.According to concrete scene, the size of memory block, between 1024 bytes, is incremented by, such as in 64 bytes with the amplitude of every 64 byte
Shown in Fig. 4, it illustrates the schematic diagram of memory block distribution.The use of size is 64 bytes if the data of storage are 12 byte
Memory block store the data, if storage data be 68 bytes, store the number using the memory block that size is 128 bytes
According to.
For the memory block of each size, the corresponding memory address of size identical memory block is put into same pre-
If in list, the corresponding memory address of memory block of different sizes is put into different the presets lists.Fig. 5 shows basis
The schematic diagram of the presets list of one embodiment of the present invention.As shown in figure 5, the memory block that all sizes are 64 bytes is corresponded to
Memory address put into first the presets list（Call number is 0）, memory block that all sizes are 128 bytes is corresponding
Memory address is put into second the presets list（Call number is 1）, by memory block that all sizes are 192 bytes it is corresponding in
Address is deposited to put into the 3rd the presets list（Call number is 2）, by the corresponding internal memory of memory block that all sizes are 256 bytes
Address is put into the 4th the presets list（Call number is 3）, the like.
In order to realize the fast operating to database, i.e. quick to write data to database, and, quickly from database
Middle reading data, an implementation method of the invention, pair memory block for determining carries out operation to be included：According to HashMap numbers
The memory block determined according to structure pair is operated.
Fig. 6 shows the schematic diagram of the HashMap data structures according to one embodiment of the present invention.As shown in fig. 6,
The starting position of operating system shared drive retains a small block space, for preserving HashMap headers（Header）And bucket
（Bucket）Array.Each Bucket preserves first node in the node listing pointed to corresponding to the Bucket
（First Entry）Pointer, last node（Last Entry）Pointer and the node listing size（size）.Often
One node（Entry）Previous node is included respectively（PreEntry）Pointer and latter node（Next Entry）Finger
Pin, key（Key）Hash（Hash）Value, Yi Jijian（Key）And value（Value）Pointer.
Based on above-mentioned HashMap data structures, the mode that write-in process performs write-in data manipulation is：First to write
Key carry out Hash（Hash）Computing, obtains the position of Bucket；Then the corresponding Entry lists of traversal Bucket, checking will
The key of write-in whether there is, if it does not exist, then the key that will be write is added to the end of Entry lists, if existing,
Then replace the value of key.
For each write-in data manipulation, if substituted for old value, then the memory block for storing the old value is corresponding interior
Deposit address to be accomplished by being reentered into the presets list, so that when new write-in data manipulation needs memory block, meeting is according to data
Size, reuse the presets list in the corresponding memory block of memory address, so as to reach internal memory reuse purpose.
Based on above-mentioned HashMap data structures, the mode that reading process performs read data operation is：First to read
Key carry out Hash（Hash）Computing, obtains the position of Bucket；Then the corresponding Entry lists of traversal Bucket, from
The key to be read is searched in Entry lists, and returns to the value of the key to be read.
In order to avoid the write-in conflict produced in the multiple write-in data manipulations of executed in parallel, a reality of the invention
Mode is applied, when the operation is for write-in data manipulation, in writing data into the memory block of determination using single thread.
But, an implementation method of the invention, when the operation is for read data operation, it is possible to use multi-thread
Journey reads data from the memory block for determining.
For example, by taking Nginx as an example, one Nginx module of standard of exploitation is used as the process of reading.The Nginx modules can be with
Simultaneously（Multithreading can be supported）The request of read data operation is parsed, is obtained included in the request of the read data operation
Parameter（Parameter is one or more key）, call the reading interface that database is provided to obtain the value of key according to key, will
The value of acquisition returns to user after being packaged by the response of read data operation.
Wherein, the reading interface that database is provided is to support that many key assignments read simultaneously, for example, a read data operation
Request be：
http://www.example.com/querykey=001,002,003,004.Reading process in server exists
When calling reading interface, the reading interface is passed to by key=001,002,003,004, read interface and read from memory block respectively
The value of individual key, and the value of each key is returned to reading process.
Additionally, one of the invention embodiment there is provided simple DLL, for realizing that database is initial
Change, write data, read data and staqtistical data base state：
1st, database initialization interface（Need to specify shared drive size）
bool map_init(int capacity,int mem_size,const char*data_file_path)；
2nd, data-interface is write（If key is existing, the value of key is covered）
void map_put(const char*key,const char*value)；
3rd, data-interface is read（Obtain the value of key）
char*map_get(const char*key)；
4th, the interface of the current statistic state of database is obtained
void map_stat(int*key_stat,int*value_stat)。
It should be noted that the various DLLs of database can be realized using C language, database write-in process and reading
The program development language realization that the process of taking can be bound using any support C language, including but not limited to：C、C++、Java、
Python and NodeJs etc..
According to the present invention, because database is created in operating system shared drive on the server, therefore, write-in
Process can directly be operated as being operated to local memory with reading process to database, and need not carry out data
Transfer.The time consumed such that it is able to save factor data transfer process and system resource, and then effectively ensure to data
" the real-time operation " in storehouse is required.
In addition, idle memory block is reclaimed and reused by the presets list, the product of memory fragmentation is not only avoid
It is raw, but also reached the controllable purpose of memory block.
Also in addition, avoiding read/write conflict by the way of without lock, simplified system architecture substantially increases systematic function.
Example devices
After method is described, next, with reference to Fig. 7, that schematically shows according to an embodiment party of the invention
The structural framing figure of the equipment of formula describes the equipment for realizing database real-time operation.Specifically, the equipment for example can be with
Including：
Database creation unit 701, for creating database in operating system shared drive on the server；
Memory block determining unit 702, it is common from the operating system for the request in response to the operation to the database
Enjoy and determine the targeted memory block of the operation requests in internal memory；
Operating unit 703, operates for pair memory block for determining.
An implementation method of the invention, the operation includes write-in data manipulation and read data operation.
An implementation method of the invention, when the operation is for write-in data manipulation, the operating unit configuration
For：In the memory block of determination being write data into using single thread；When the operation is for read data operation, the operation list
Unit is configured to：Using multithreading data can be read from the memory block for determining.
An implementation method of the invention, the operating unit is configured to：
The memory block determined according to HashMap data structures pair is operated.
According to some embodiments of the present invention, equipment can also include：
First map unit, for the magnetic being mapped to the data in the operating system shared drive on the server
In disk.
According to some embodiments of the present invention, equipment can also include：
Second map unit, for the behaviour being mapped to the data in the operating system shared drive on the server
In making Installed System Memory file.
According to some embodiments of the present invention, in the operating system shared drive for creating the database
Address is deposited not discharged and reclaimed by operating system during running.
According to some embodiments of the present invention, equipment can also include：
Memory block management unit, for when occupied memory block is idle in the operating system shared drive, by sky
The not busy corresponding memory address of memory block is put into the presets list.
An implementation method of the invention, the memory block management unit is configured to：
According to the size of memory block, the corresponding memory address of memory block is put into corresponding the presets list so that big
The corresponding memory address of small identical memory block is put into same the presets list, the corresponding internal memory of memory block of different sizes
Address is put into different the presets lists.
An implementation method of the invention, the corresponding memory block of memory address in described the presets list can be weighed
The new data for being used to preserve the database.
An implementation method of the invention, when the number of the memory address in described the presets list is more than or equal to
When 2, the corresponding memory block of memory address in described the presets list is just reused for preserving the data of the database.
According to the present invention, because database is created in operating system shared drive on the server, therefore, write-in
Process can directly be operated as being operated to local memory with reading process to database, and need not carry out data
Transfer.The time consumed such that it is able to save factor data transfer process and system resource, and then effectively ensure to data
" the real-time operation " in storehouse is required.
In addition, idle memory block is reclaimed and reused by the presets list, the product of memory fragmentation is not only avoid
It is raw, but also reached the controllable purpose of memory block.
Also in addition, avoiding read/write conflict by the way of without lock, simplified system architecture substantially increases systematic function.
If although it should be noted that being referred to the equipment for drying of the equipment for realizing database real-time operation in above-detailed
Or sub-device, but this division is only show row not enforceable.In fact, according to the embodiment of the present invention,
The feature and function of above-described two or more devices can be realized in one apparatus.Or it is on the contrary, described above
The feature and function of a device can be further divided into being realized by multiple devices.
Additionally, although the operation of the inventive method is described with particular order in the accompanying drawings, this do not require that or
Hint must be performed according to the particular order these operation, or the operation having to carry out shown in whole could realize it is desired
As a result.Conversely, the step of describing in flow chart can change execution sequence.Additionally or alternatively, it is convenient to omit some steps,
Multiple steps are merged into a step to perform, and/or a step is decomposed into execution of multiple steps.
Referred in application documents verb " including ", "comprising" and its paradigmatic using being not excluded for except application documents
Described in those elements or element or step outside step presence.Article "a" or "an" before element is not excluded for many
The presence of individual this element.
Although describing spirit and principles of the present invention by reference to some specific embodiments, it should be appreciated that, this
Invention is not limited to disclosed specific embodiment, and the division to each side does not mean that the feature in these aspects can not yet
Combination is this to divide merely to the convenience of statement to be benefited.It is contemplated that cover appended claims spirit and
In the range of included various modifications and equivalent arrangements.Scope of the following claims meets broadest explanation, so that comprising
All such modifications and equivalent structure and function.
Claims (12)
1. a kind of method, including：
Database is created in operating system shared drive on the server；
In response to the request of the operation to the database, determine that the operation is targeted from the operating system shared drive
Memory block；
Pair determine memory block operate；The operation includes writing data manipulation and read data operation；When the operation
During for write-in data manipulation, in writing data into the memory block of determination according to HashMap data structures using write-in thread；When
When the operation is for read data operation, read from the memory block for determining according to HashMap data structures using thread is read
Data；The mode of said write data manipulation is：Hash operation is carried out to the key Key to be write, the HashMap data are obtained
The position of bucket Bucket in structure；The corresponding node listings of the Bucket are traveled through, checks that the Key to be write whether there is, if
Do not exist, then the Key that will be write is added to the end of the node listing, if it is present replacing the value Value of the Key；Institute
The mode for stating read data operation is：Hash operation is carried out to the Key to be read, in obtaining the HashMap data structures
The position of Bucket；The corresponding node listings of the Bucket are traveled through, the Key to be read is searched from the node listing, returning will
The Value of the Key of reading；
Data in the operating system shared drive are mapped in the disk on the server；Or, by the operation
Data in system shared drive are mapped in the operating system memory file on the server, by the operating system memory
Data in file are backuped in the disk on the server.
2. method according to claim 1, wherein, the operating system shared drive for creating the database
Memory address is not discharged and is reclaimed during running by operating system.
3. method according to claim 2, also includes：
When occupied memory block is idle in the operating system shared drive, by the idle corresponding memory address of memory block
It is put into the presets list.
4. method according to claim 3, wherein, the corresponding memory block of memory address in described the presets list can be by
It is reused for preserving the data of the database.
5. method according to claim 4, wherein, when the number of the memory address in described the presets list is more than or waits
When 2, the corresponding memory block of memory address in described the presets list is just reused for preserving the data of the database.
6. method according to claim 3, wherein, it is described that the idle corresponding memory address of memory block is put into default row
Table includes：
According to the size of memory block, the corresponding memory address of memory block is put into corresponding the presets list so that size phase
The same corresponding memory address of memory block is put into same the presets list, the corresponding memory address of memory block of different sizes
Put into different the presets lists.
7. a kind of equipment, including：
Database creation unit, for creating database in operating system shared drive on the server；
Memory block determining unit, for the request in response to the operation to the database, from the operating system shared drive
The middle memory block for determining that the operation requests are targeted；
Operating unit, operates for pair memory block for determining, the operation includes write-in data manipulation and reads data behaviour
Make；When the operation is for write-in data manipulation, the operating unit is configured to：Using write-in thread according to HashMap data
Structure is write data into the memory block of determination；When the operation is for read data operation, the operating unit is configured to：
Using thread is read data are read according to HashMap data structures from the memory block for determining；The side of said write data manipulation
Formula is：Hash operation is carried out to the key Key to be write, the position of bucket Bucket in the HashMap data structures is obtained；Traversal
The corresponding node listings of the Bucket, check that the Key to be write whether there is, if it does not exist, then the Key additions that will be write
To the end of the node listing, if it is present replacing the value Value of the Key；The mode of the read data operation is：It is right
The Key to be read carries out Hash operation, obtains the position of Bucket in the HashMap data structures；Travel through this Bucket pairs
The node listing answered, searches the Key to be read from the node listing, returns to the Value of the Key to be read；
First map unit, for the disk being mapped to the data in the operating system shared drive on the server
In；
Second map unit, for the operation system being mapped to the data in the operating system shared drive on the server
In system memory file, during the data in the operating system memory file are backuped into the disk on the server.
8. equipment according to claim 7, wherein, the operating system shared drive for creating the database
Memory address is not discharged and is reclaimed during running by operating system.
9. equipment according to claim 8, also includes：
Memory block management unit, for when occupied memory block is idle in the operating system shared drive, by the free time
The corresponding memory address of memory block is put into the presets list.
10. equipment according to claim 9, wherein,
The corresponding memory block of memory address in described the presets list can be reused for preserving the data of the database.
11. equipment according to claim 10, wherein,
When the number of the memory address in described the presets list is more than or equal to 2, the memory address in described the presets list
Corresponding memory block is just reused for preserving the data of the database.
12. equipment according to claim 9, wherein, the memory block management unit is configured to：
According to the size of memory block, the corresponding memory address of memory block is put into corresponding the presets list so that size phase
The same corresponding memory address of memory block is put into same the presets list, the corresponding memory address of memory block of different sizes
Put into different the presets lists.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310648091.5A CN103593485B (en) | 2013-12-04 | 2013-12-04 | The method and apparatus for realizing database real-time operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310648091.5A CN103593485B (en) | 2013-12-04 | 2013-12-04 | The method and apparatus for realizing database real-time operation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103593485A CN103593485A (en) | 2014-02-19 |
| CN103593485B true CN103593485B (en) | 2017-06-16 |
Family
ID=50083626
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310648091.5A Active CN103593485B (en) | 2013-12-04 | 2013-12-04 | The method and apparatus for realizing database real-time operation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103593485B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10127260B2 (en) * | 2014-11-25 | 2018-11-13 | Sap Se | In-memory database system providing lockless read and write operations for OLAP and OLTP transactions |
| CN105989049A (en) * | 2015-02-05 | 2016-10-05 | 天脉聚源（北京）科技有限公司 | Data middle layer realizing method and system |
| CN107506375A (en) * | 2017-07-17 | 2017-12-22 | 上海斐讯数据通信技术有限公司 | Date storage method and device |
| US10699070B2 (en) | 2018-03-05 | 2020-06-30 | Sap Se | Dynamic retrieval and rendering of user interface content |
| CN110287044B (en) * | 2019-07-02 | 2021-08-03 | 广州虎牙科技有限公司 | Lock-free shared memory processing method and device, electronic equipment and readable storage medium |
| CN114416426A (en) * | 2021-12-17 | 2022-04-29 | 阿里巴巴（中国）有限公司 | Process copying method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1427342A (en) * | 2001-12-21 | 2003-07-02 | 上海贝尔有限公司 | Internal storage management system and its distribution method |
| CN1851676A (en) * | 2006-05-25 | 2006-10-25 | 浙江大学 | Embedded system buffer internal memory distribution method |
| CN102420814A (en) * | 2011-11-21 | 2012-04-18 | 航天科工深圳（集团）有限公司 | Data access method and device, and server |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315628B (en) * | 2007-06-01 | 2011-01-05 | 华为技术有限公司 | Internal memory database system and method and device for implementing internal memory data base |
| US8261025B2 (en) * | 2007-11-12 | 2012-09-04 | International Business Machines Corporation | Software pipelining on a network on chip |
| CN101582092B (en) * | 2009-06-12 | 2011-04-20 | 中兴通讯股份有限公司 | Method and device for realizing the store of date in memory |
-
2013
- 2013-12-04 CN CN201310648091.5A patent/CN103593485B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1427342A (en) * | 2001-12-21 | 2003-07-02 | 上海贝尔有限公司 | Internal storage management system and its distribution method |
| CN1851676A (en) * | 2006-05-25 | 2006-10-25 | 浙江大学 | Embedded system buffer internal memory distribution method |
| CN102420814A (en) * | 2011-11-21 | 2012-04-18 | 航天科工深圳（集团）有限公司 | Data access method and device, and server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103593485A (en) | 2014-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103593485B (en) | The method and apparatus for realizing database real-time operation | |
| US9342376B2 (en) | Method, system, and device for dynamic energy efficient job scheduling in a cloud computing environment | |
| CN104750559B (en) | The pond of memory resource across multinode | |
| US7373475B2 (en) | Methods for optimizing memory unit usage to maximize packet throughput for multi-processor multi-threaded architectures | |
| US10331669B2 (en) | Fast query processing in columnar databases with GPUs | |
| US20150095384A1 (en) | File transfer to a distributed file system | |
| US11636072B2 (en) | Parallel processing of a keyed index file system | |
| US20140059094A1 (en) | Making use of a file path to determine file locality for applications | |
| US11940915B2 (en) | Cache allocation method and device, storage medium, and electronic device | |
| US9218198B2 (en) | Method and system for specifying the layout of computer system resources | |
| JP2023036774A (en) | Access control method of shared memory, access control device of shared memory, electronic apparatus, and autonomous vehicle | |
| US20220374742A1 (en) | Method, device and storage medium for running inference service platform | |
| CN105302536A (en) | Configuration method and apparatus for related parameters of MapReduce application | |
| US20140337391A1 (en) | Method and system for data plane abstraction to enable a network storage platform ecosystem | |
| CN111651286A (en) | Data communication method, device, computing equipment and storage medium | |
| US11030714B2 (en) | Wide key hash table for a graphics processing unit | |
| US20160371239A1 (en) | Domain specific representation of document text for accelerated natural language processing | |
| US11138291B2 (en) | Assymetric allocation of SRAM and data layout for efficient matrix multiplication | |
| CN109918381A (en) | Method and apparatus for storing data | |
| JP2021131897A (en) | Scheduling method, device, equipment, storage equipment, and program | |
| González-Domínguez et al. | parSRA: A framework for the parallel execution of short read aligners on compute clusters | |
| Yu et al. | Hypergef: A framework enabling efficient fusion for hypergraph neural network on gpus | |
| US20140074785A1 (en) | Open file rebalance | |
| US11010349B2 (en) | File system with distributed entity state | |
| CN113127430A (en) | Mirror image information processing method and device, computer readable medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |