CN109753245A - A kind of multiple disks load balancing asynchronous read and write dispatching method and device - Google Patents
A kind of multiple disks load balancing asynchronous read and write dispatching method and device Download PDFInfo
- Publication number
- CN109753245A CN109753245A CN201811653968.9A CN201811653968A CN109753245A CN 109753245 A CN109753245 A CN 109753245A CN 201811653968 A CN201811653968 A CN 201811653968A CN 109753245 A CN109753245 A CN 109753245A
- Authority
- CN
- China
- Prior art keywords
- disk
- read
- write
- reading
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Multiple disks load balancing asynchronous read and write dispatching method and device, configure HDFS, return the back end of HDFS system to namenode disc information, and start metadata query service processes in namenode；Local all disc informations are obtained, and disk number is carried out according to the disk type in disc information；Independent reading writing working thread is respectively created to disk, makes disk that upper layer be waited to send read-write task；Metadata query process receives the upper layer user HDFS request of data to be read, and the disk number for the HDFS data to be read is obtained by the metadata query service processes of namenode；Disk number is sent together when sending read-write task to reading writing working thread to notify to give upper layer user after the completion according to the respective read-write task of disc information asynchronous process when reading writing working thread is connected to read-write task.Multiple threads are fought under reduction high concurrent and multiple disks fight for bring performance cost, improve data read-write efficiency.
Description
Technical field
The present embodiments relate to database technical fields, more particularly to a kind of multiple disks load balancing asynchronous read and write tune
Spend method and device.
Background technique
HDFS refers to distributed file system, and HDFS is usually designed to be the distribution text being suitble to operate on common hardware
Part system.HDFS is the system of an Error Tolerance, is suitble to be deployed on cheap machine, and the data that can provide high-throughput are visited
It asks, the application being very suitable on large-scale dataset.HDFS is a host-guest architecture, and a HDFS cluster is by a name byte
Point, namenode are a management file name space and the primary server for adjusting client access file, and there are also some data
Node, one machine of a usually node, back end manage the storage of corresponding node.HDFS opening file designation
Space simultaneously allows user data to store with document form.
Currently, real data is placed into multiple disks by back end under HDFS system, include common mechanical disk
With SSD solid state hard disk, database is also required to for ephemeral data to be stored in these disks of back end, big portion in the process of implementation
Divided data library directly reads and writes HDFS by the interface of HDFS respectively and is stored in local data, and database also passes through local file system
The local temporary file of the interface read-write of system, causes under concurrent multiple requests to be synchronous to the read-write of multiple and different type disks
, and be that the read-write of different type disk mutually mixes, it is fought for so as to cause threads multiple under high concurrent and more
A disk fights for bring performance cost.There is no carry out the asynchronous of load balancing to multiple disks under HDFS system at this stage
Read and write technical solution.
Summary of the invention
For this purpose, the embodiment of the present invention provides a kind of multiple disks load balancing asynchronous read and write dispatching method and device, to disk
Disk number is carried out, the disk type and disk number of back end can be got by HDFS, makes reading data and local
True disk type and disk number correctly correspond to, and distinguish the different disk of asynchronous process by different reading writing working threads
Read-write requests.
To achieve the goals above, embodiments of the present invention provide the following technical solutions: a kind of multiple disks load balancing
Asynchronous read and write dispatching method, comprising:
HDFS is configured, returns the back end of HDFS system to namenode disc information, and in name
Node starts metadata query service processes, and the disc information includes disk unit number and disk type；
Local all disc informations are obtained by the interface of native operating sys-tern when database starting, and according to described
Disk type in disc information carries out disk number；
Independent reading writing working thread is respectively created to multiple disks, makes disk that upper layer be waited to send read-write task；
When database executes SQL, the metadata query process receives the upper layer user HDFS data to be read and asks
It asks, the disk number for the HDFS data to be read is obtained by the metadata query service processes of the namenode, when to institute
It states when reading writing working thread sends read-write task and sends disk number, root when the reading writing working thread is connected to read-write task together
According to the respective read-write task of the disc information asynchronous process, notify to give upper layer user after the completion.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching method, the disk type includes mechanical disk
And solid state hard disk, the metadata query service processes safeguard the mapping of disk number according to the difference of disk type.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching method, fail when obtaining the disc information
When, 1 is set by the quantity of disk, disk type is set as solid state hard disk；
When the HDFS request of data of reading is sent by network remote equipment, set the disk type to
Network.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching method, not for different disk type distribution
With the reading writing working thread of quantity.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching method, to the disc-pack read into
Row maintenance, after the metadata query process is connected to the HDFS request of data to be read, if looked into the disc-pack
It looks for disk unit number and disk type to fail, number is re-started to the disk of requests data reading；If in the disk collection
Disk unit number and disk type success are searched in conjunction, and existing disk number is used to the disk of requests data reading.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching method, HDFS number of the upper layer user in reading
According to when the disk number where data and data passed into read-write subsystem, reading task is sent to by the read-write subsystem
The reading writing working thread process of the corresponding disk of disk number simultaneously waits completion to be read, after the completion of reading HDFS data, read-write
The data of reading are returned to upper layer user by worker thread；
When upper layer user is when reading and writing temporary file to local disk, corresponding disk unit is found by local directory
Number, and corresponding disk number is searched by disk unit number, then the disk number where temporary file data is passed to
Corresponding reading writing working thread process simultaneously waits read-write to complete, after reading writing working thread completes the read-write of temporary file data,
Return to upper layer user.
The embodiment of the present invention also provides a kind of multiple disks load balancing asynchronous read and write dispatching device, comprising:
Node configuration module, for being configured to HDFS, make the back end of HDFS system by disc information return to
Namenode, and start metadata query service processes in namenode；
Disc information obtains module, local all for being obtained when database starts by the interface of native operating sys-tern
Disc information；
Number module, for carrying out disk number according to the disk type in disc information；
Task creation module makes disk that upper layer be waited to send for creating independent reading writing working thread to each disk
Read-write task；
Receiving module is read and write, is wanted for receiving upper layer user by metadata query process when database executes SQL
The HDFS request of data of reading,
Task processing module, it is respective according to disc information asynchronous process when being connected to read-write task for reading writing working thread
Read-write task；
Feedback module is read and write, for sending read-write task and disk number to reading writing working thread and notifying to use to upper layer
Person.
It further include maintenance module as the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching device, for first number
The mapping of disk number is safeguarded according to the difference of disk type it is investigated that asking service processes, the disk type includes machinery
Disk and solid state hard disk；
When obtaining disc information failure, 1 is set by the quantity of disk, disk type is set as solid state hard disk；
When the HDFS request of data of reading is sent by network remote equipment, set the disk type to
Network；
For the reading writing working line of different disk type distribution different numbers.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching device, to the disc-pack read into
Row maintenance, after metadata query process is connected to the HDFS request of data to be read, if searching magnetic in the disc-pack
Disc apparatus number and disk type fail, and re-start number to the disk of requests data reading；If in the disc-pack
Disk unit number and disk type success are searched, existing disk number is used to the disk of requests data reading.
As the preferred embodiment of multiple disks load balancing asynchronous read and write dispatching device, HDFS number of the upper layer user in reading
According to when the disk number where data and data passed into read-write subsystem, reading task is sent to by the read-write subsystem
The reading writing working thread process of the corresponding disk of disk number simultaneously waits completion to be read, after the completion of reading HDFS data, read-write
The data of reading are returned to upper layer user by worker thread；
When upper layer user is when reading and writing temporary file to local disk, corresponding disk unit is found by local directory
Number, and corresponding disk number is searched by disk unit number, then the disk number where temporary file data is passed to
Corresponding reading writing working thread process simultaneously waits read-write to complete, after reading writing working thread completes the read-write of temporary file data,
Return to upper layer user.
Embodiments of the present invention, which have the advantages that, carries out disk number to disk, can get number by HDFS
According to the disk type and disk number of node, and reading data and local true disk type and disk number correctly correspond to,
The read-write requests that the different disk of asynchronous process is distinguished by different reading writing working threads, make same disk under multiple disks
The read-write task of HDFS read-write and local file is by same several read-write thread asynchronous process, it is possible to reduce multiple under high concurrent
Thread is fought for and multiple disks fight for bring performance cost, improves data read-write efficiency.
Detailed description of the invention
It, below will be to embodiment party in order to illustrate more clearly of embodiments of the present invention or technical solution in the prior art
Formula or attached drawing needed to be used in the description of the prior art are briefly described.It should be evident that the accompanying drawings in the following description is only
It is merely exemplary, it for those of ordinary skill in the art, without creative efforts, can also basis
The attached drawing of offer, which is extended, obtains other implementation attached drawings.
Fig. 1 is a kind of multiple disks load balancing asynchronous read and write dispatching method flow chart provided in the embodiment of the present invention；
Fig. 2 is a kind of multiple disks load balancing asynchronous read and write dispatching device schematic diagram provided in the embodiment of the present invention；
In figure: 1, node configuration module；2, disc information obtains module；3, number module；4, task creation module；5, it reads
Write receiving module；6, task processing module；7, feedback module is read and write；8, maintenance module.
Specific embodiment
Embodiments of the present invention are illustrated by particular specific embodiment below, those skilled in the art can be by this explanation
Content disclosed by book is understood other advantages and efficacy of the present invention easily, it is clear that described embodiment is the present invention one
Section Example, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art are not doing
Every other embodiment obtained under the premise of creative work out, shall fall within the protection scope of the present invention.
Referring to Fig. 1, the present embodiment provides a kind of multiple disks load balancing asynchronous read and write dispatching methods, comprising the following steps:
S1: configuring HDFS, returns the back end of HDFS system to namenode disc information, and in name
Byte point starts metadata query service processes, and the disc information includes disk unit number and disk type；
S2: when database starting by the local all disc informations of interface acquisition of native operating sys-tern, and according to
Disk type in the disc information carries out disk number；
S3: respectively creating independent reading writing working thread to multiple disks, makes disk that upper layer be waited to send read-write and appoints
Business；
S4: when database executes SQL, the metadata query process receives the upper layer user HDFS data to be read
Request obtains the disk number for the HDFS data to be read by the metadata query service processes of the namenode；
S5: disk number, the reading writing working line are sent together when sending read-write task to the reading writing working thread
According to the respective read-write task of the disc information asynchronous process when journey is connected to read-write task, notify to use to upper layer after the completion
Person.
In one embodiment of multiple disks load balancing asynchronous read and write dispatching method, the disk type includes mechanical disk
And solid state hard disk, the metadata query service processes safeguard the mapping of disk number according to the difference of disk type.
Specifically, in S2 when database start when, from Linux file system /proc/partitions in obtain all disks and phase
The disk unit name device-name answered, and this is obtained from/sys/block/device-name/queue/rotational
Whether equipment is ordinary magnetic disc.For standard machinery disk, disk number increases since 0, and each mechanical disk number increases
1；It for solid state hard disk, numbers since -10, each solid state hard disk number subtracts 1；For network type, disk number is arranged
It is -1, due to only one this type example, such design method will not be overlapped different types of disk number.Due to not
The disk number disunity of same type establishes a Map to be managed collectively, for above-mentioned disk number to exist with disk
Local serial number is mapped.
Specifically, by the configuration item dfs.datanode.hdfs-blocks-metadata.enabled of HDFS in S1
It is set as true, such back end can open the disks information report such as disk type to namenode, and in namenode
Dynamic metadata query service processes.
Specifically, the hdfs-site.xml configuration file of the modification each back end of HDFS, increases following configuration item
And restart all processes of HDFS service.
Specifically, setting 1 for the quantity of disk, disk type is set as solid when obtaining disc information failure
State hard disk；When the HDFS request of data of reading is sent by network remote equipment, set the disk type to
Network.
Specifically, for the reading writing working thread of different disk type distribution different numbers.For standard machinery disk,
Distribute 1 independent reading writing working thread；For solid state hard disk, 8 independent reading writing working threads are distributed；For network, distribution 16
A independent reading writing working thread.The above quantity can be modified in configuration file, use convenient for tuning.Per thread is one corresponding
Disk task queue, it is in the queue when idle etc., do not consume resource.If there is task is put into queue, can wake up at once
Reading writing working thread is written and read.
Specifically, metadata query service starting after, create two Map, respectively correspond standard machinery disk HDDMap and
The disk number of solid-state hard disk SSD Map maps.The Key of Map is node name, and Value is the equipment of all disks under this node
Number and corresponding number, Value itself be also a Map structure (device number of Key disk, Value are the numbers of disk).Its
Middle device number is obtained from HDFS, and each disk is unique constant number.
In one embodiment of multiple disks load balancing asynchronous read and write dispatching method, to the disc-pack read into
Row maintenance, after the metadata query process is connected to the HDFS request of data to be read, if looked into the disc-pack
It looks for disk unit number and disk type to fail, number is re-started to the disk of requests data reading；If in the disk collection
Disk unit number and disk type success are searched in conjunction, and existing disk number is used to the disk of requests data reading.
In one embodiment of multiple disks load balancing asynchronous read and write dispatching method, HDFS number of the upper layer user in reading
According to when the disk number where data and data passed into read-write subsystem, reading task is sent to by the read-write subsystem
The reading writing working thread process of the corresponding disk of disk number simultaneously waits completion to be read, after the completion of reading HDFS data, read-write
The data of reading are returned to upper layer user by worker thread；
When upper layer user is when reading and writing temporary file to local disk, corresponding disk unit is found by local directory
Number, and corresponding disk number is searched by disk unit number, then the disk number where temporary file data is passed to
Corresponding reading writing working thread process simultaneously waits read-write to complete, after reading writing working thread completes the read-write of temporary file data,
Return to upper layer user.
Specifically, database execute SQL (structured query language, Structured Query Language) when
Wait, first pass through namenode metadata query service processes obtain the data to be read disk number, then reality to
Read-write line work journey takes disk number when sending reading task, while arriving if needing to read and write temporary file in implementation procedure
Local disk also takes disk number when sending read-write task.The reading writing working thread of different disk is connected to read-write and appoints
According to the respective read-write task of the asynchronous processing of these information when business, notify to give upper layer user after the completion.
Specifically, first inquiring this number to namenode after metadata query process is connected to the HDFS request of data to be read
According to the device number of the disk of the back end and place at place and the type of disk.If this disk is standard machinery magnetic
Disk, and to read for the first time, i.e., it can not find from ordinary magnetic disc set, then disk number is set as 0 plus standard machinery disk
The element number of set, and this disk is added to standard machinery disc-pack；If this disk is solid state hard disk, and is first
Secondary reading can not find from solid state hard disk set, then number is set as -10 element numbers for subtracting solid state hard disk set, and
This disk is added to solid state hard disk set；If this data is not in local disk, but in remote network devices, then number is set
It is set to -1.Data read request if it is second and later, then can find, and return from corresponding disc-pack
The number being ready for for the first time.
Specifically, then searching whether exist with back end in HDDMap if it is standard machinery disk, if do not deposited
A Map is then being constructed, and back end and this Map are inserted into HDDMap.If it is present taking out corresponding
Value then proceedes to be searched in Value with the device number of disk, if it is found, then taking out number of the Value as disk
Caller is returned to, if it is not found, then scurries into disk unit number and this number using the size of this Value as number
Into Value, and send back to caller.If it is solid state hard disk, SSDMap is similarly operated, is not uniquely both,
When number, -10 are used to subtract the size of Value as number.
SQL will find corresponding disk queue in practical implementation by disk number, and read-write task is put into
In disk queue, the corresponding read-write thread of each disk queue can get these read-write tasks at once, be written and read behaviour respectively
Make, the caller on upper layer is notified that after completing.
Referring to fig. 2, the embodiment of the present invention also provides a kind of multiple disks load balancing asynchronous read and write dispatching device, comprising:
Node configuration module 1, for being configured to HDFS, make the back end of HDFS system by disc information return to
Namenode, and start metadata query service processes in namenode；
Disc information obtains module 2, for obtaining local institute by the interface of native operating sys-tern when database starts
Some disc informations；
Number module 3, for carrying out disk number according to the disk type in disc information；
Task creation module 4 makes disk that upper layer be waited to send for creating independent reading writing working thread to each disk
Read-write task；
Receiving module 5 is read and write, is wanted for receiving upper layer user by metadata query process when database executes SQL
The HDFS request of data of reading,
Task processing module 6, when being connected to read-write task for reading writing working thread respectively according to disc information asynchronous process
Read-write task；
Feedback module 7 is read and write, for sending read-write task and disk number to reading writing working thread and notifying to make to upper layer
User.
It further include maintenance module 8 in one embodiment of multiple disks load balancing asynchronous read and write dispatching device, for first number
The mapping of disk number is safeguarded according to the difference of disk type it is investigated that asking service processes, the disk type includes machinery
Disk and solid state hard disk；When obtaining disc information failure, 1 is set by the quantity of disk, disk type is set as solid
State hard disk；When the HDFS request of data of reading is sent by network remote equipment, set the disk type to
Network；For the reading writing working line of different disk type distribution different numbers.
In one embodiment of multiple disks load balancing asynchronous read and write dispatching device, to the disc-pack read into
Row maintenance, after metadata query process is connected to the HDFS request of data to be read, if searching magnetic in the disc-pack
Disc apparatus number and disk type fail, and re-start number to the disk of requests data reading；If in the disc-pack
Disk unit number and disk type success are searched, existing disk number is used to the disk of requests data reading.
In one embodiment of multiple disks load balancing asynchronous read and write dispatching device, HDFS number of the upper layer user in reading
According to when the disk number where data and data passed into read-write subsystem, reading task is sent to by the read-write subsystem
The reading writing working thread process of the corresponding disk of disk number simultaneously waits completion to be read, after the completion of reading HDFS data, read-write
The data of reading are returned to upper layer user by worker thread；When upper layer user is when reading and writing temporary file to local disk,
Corresponding disk unit number is found by local directory, and corresponding disk number is searched by disk unit number, then will be faced
When file data where disk number pass to corresponding reading writing working thread process and wait read-write complete, reading writing working line
After journey completes the read-write of temporary file data, upper layer user is returned to.
The technical solution of the embodiment of the present invention, by the configuration item dfs.datanode.hdfs-blocks- of HDFS
Metadata.enabled is set as true, such back end can by information reports such as disk types to namenode, and
Start metadata query service processes in namenode；The interface of local (SuSE) Linux OS obtains local when database starts
All disks and disk type, and give different numbers respectively according to type；Database is each to multiple disks when starting
From the independent reading writing working thread of creation, and upper layer is waited to send read-write task；Database first passes through when executing SQL
The metadata query service processes of namenode obtain the disk number for the data to be read, then in reality to reading writing working line
Journey takes disk number when sending reading task, while if needing to read and write temporary file in implementation procedure to local disk,
Disk number is also taken when sending read-write task.According to this when the reading writing working thread of different disk is connected to read-write task
The respective read-write task of a little asynchronous processing of information, notifies to give upper layer user after the completion.Make same disk under multiple disks
The read-write task of HDFS read-write and local file is by same several read-write thread asynchronous process, it is possible to reduce multiple under high concurrent
Thread is fought for and multiple disks fight for bring performance cost, improves data read-write efficiency.
Although above having used general explanation and specific embodiment, the present invention is described in detail, at this
On the basis of invention, it can be made some modifications or improvements, this will be apparent to those skilled in the art.Therefore,
These modifications or improvements without departing from theon the basis of the spirit of the present invention are fallen within the scope of the claimed invention.
Claims (10)
1. a kind of multiple disks load balancing asynchronous read and write dispatching method characterized by comprising
HDFS is configured, returns the back end of HDFS system to namenode disc information, and in namenode
Start metadata query service processes, the disc information includes disk unit number and disk type；
Local all disc informations are obtained by the interface of native operating sys-tern when database starting, and according to the disk
Disk type in information carries out disk number；
Independent reading writing working thread is respectively created to multiple disks, makes disk that upper layer be waited to send read-write task；
When database executes SQL, the metadata query process receives the upper layer user HDFS request of data to be read, and leads to
The metadata query service processes for crossing the namenode obtain the disk number for the HDFS data to be read；
Disk number is sent together when sending read-write task to the reading writing working thread, and the reading writing working thread is connected to reading
According to the respective read-write task of the disc information asynchronous process when writing task, notify to give upper layer user after the completion.
2. a kind of multiple disks load balancing asynchronous read and write dispatching method according to claim 1, which is characterized in that the magnetic
Disc-type includes mechanical disk and solid state hard disk, and the metadata query service processes compile disk according to the different of disk type
Number mapping safeguarded.
3. a kind of multiple disks load balancing asynchronous read and write dispatching method according to claim 1, which is characterized in that work as acquisition
When the disc information fails, 1 is set by the quantity of disk, disk type is set as solid state hard disk；
When the HDFS request of data of reading is sent by network remote equipment, net is set by the disk type
Network.
4. a kind of multiple disks load balancing asynchronous read and write dispatching method according to claim 1, which is characterized in that for not
The reading writing working thread of same disk type distribution different number.
5. a kind of multiple disks load balancing asynchronous read and write dispatching method according to claim 1, which is characterized in that
The disc-pack read is safeguarded, after the metadata query process is connected to the HDFS request of data to be read, if
Disk unit number is searched in the disc-pack and disk type fails, and volume is re-started to the disk of requests data reading
Number；If searching disk unit number and disk type success in the disc-pack, the disk of requests data reading is used
Existing disk number.
6. a kind of multiple disks load balancing asynchronous read and write dispatching method according to claim 1, which is characterized in that upper layer makes
Disk number where data and data is passed to read-write subsystem, read-write in the HDFS data of reading by user
Reading task is sent to the reading writing working thread process of the corresponding disk of disk number and waits completion to be read by system, works as reading
After the completion of HDFS data, the data of reading are returned to upper layer user by reading writing working thread；
When upper layer user is when reading and writing temporary file to local disk, corresponding disk unit number is found by local directory,
And corresponding disk number is searched by disk unit number, the disk number where temporary file data is then passed into correspondence
Reading writing working thread process and wait read-write complete, reading writing working thread complete temporary file data read-write after, return
Give upper layer user.
7. a kind of multiple disks load balancing asynchronous read and write dispatching device characterized by comprising
Node configuration module returns the back end of HDFS system to name disc information for configuring to HDFS
Node, and start metadata query service processes in namenode；
Disc information obtains module, for obtaining local all magnetic by the interface of native operating sys-tern when database starts
Disk information；
Number module, for carrying out disk number according to the disk type in disc information；
Task creation module makes disk that upper layer be waited to send read-write for creating independent reading writing working thread to each disk
Task；
Receiving module is read and write, to be read for receiving upper layer user by metadata query process when database executes SQL
HDFS request of data,
Task processing module, according to the respective read-write of disc information asynchronous process when being connected to read-write task for reading writing working thread
Task；
Feedback module is read and write, for sending read-write task and disk number to reading writing working thread and notifying to give upper layer user.
8. a kind of multiple disks load balancing asynchronous read and write dispatching device according to claim 7, which is characterized in that further include
Maintenance module is safeguarded the mapping of disk number according to the difference of disk type for metadata query service processes, institute
Stating disk type includes mechanical disk and solid state hard disk；
When obtaining disc information failure, 1 is set by the quantity of disk, disk type is set as solid state hard disk；
When the HDFS request of data of reading is sent by network remote equipment, net is set by the disk type
Network；
For the reading writing working line of different disk type distribution different numbers.
9. a kind of multiple disks load balancing asynchronous read and write dispatching device according to claim 7, which is characterized in that
The disc-pack read is safeguarded, after metadata query process is connected to the HDFS request of data to be read, if in institute
It states and searches disk unit number and disk type failure in disc-pack, number is re-started to the disk of requests data reading；Such as
Fruit searches disk unit number and disk type success in the disc-pack, uses to the disk of requests data reading existing
Disk number.
10. a kind of multiple disks load balancing asynchronous read and write dispatching device according to claim 7, which is characterized in that upper layer
Disk number where data and data is passed to read-write subsystem, the read-write in the HDFS data of reading by user
Reading task is sent to the reading writing working thread process of the corresponding disk of disk number and waits completion to be read by subsystem, works as reading
After the completion of HDFS data, the data of reading are returned to upper layer user by reading writing working thread；
When upper layer user is when reading and writing temporary file to local disk, corresponding disk unit number is found by local directory,
And corresponding disk number is searched by disk unit number, the disk number where temporary file data is then passed into correspondence
Reading writing working thread process and wait read-write complete, reading writing working thread complete temporary file data read-write after, return
Give upper layer user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811653968.9A CN109753245B (en) | 2018-12-28 | 2018-12-28 | Multi-disk load balancing asynchronous read-write scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811653968.9A CN109753245B (en) | 2018-12-28 | 2018-12-28 | Multi-disk load balancing asynchronous read-write scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109753245A true CN109753245A (en) | 2019-05-14 |
| CN109753245B CN109753245B (en) | 2022-02-18 |
Family
ID=66404989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811653968.9A Active CN109753245B (en) | 2018-12-28 | 2018-12-28 | Multi-disk load balancing asynchronous read-write scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109753245B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110673795A (en) * | 2019-09-19 | 2020-01-10 | 深圳市网心科技有限公司 | Data writing method and device, computer device and storage medium |
| CN114253477A (en) * | 2021-12-15 | 2022-03-29 | 威创集团股份有限公司 | Ultrahigh concurrent image data writing method and device and readable storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120084503A1 (en) * | 2010-10-01 | 2012-04-05 | Canon Kabushiki Kaisha | Disk control apparatus, disk control method, and storage medium storing disk control program |
| CN102541466A (en) * | 2011-10-27 | 2012-07-04 | 忆正存储技术（武汉）有限公司 | Hybrid storage control system and method |
-
2018
- 2018-12-28 CN CN201811653968.9A patent/CN109753245B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120084503A1 (en) * | 2010-10-01 | 2012-04-05 | Canon Kabushiki Kaisha | Disk control apparatus, disk control method, and storage medium storing disk control program |
| CN102541466A (en) * | 2011-10-27 | 2012-07-04 | 忆正存储技术（武汉）有限公司 | Hybrid storage control system and method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110673795A (en) * | 2019-09-19 | 2020-01-10 | 深圳市网心科技有限公司 | Data writing method and device, computer device and storage medium |
| CN114253477A (en) * | 2021-12-15 | 2022-03-29 | 威创集团股份有限公司 | Ultrahigh concurrent image data writing method and device and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109753245B (en) | 2022-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10795905B2 (en) | Data stream ingestion and persistence techniques | |
| US10691716B2 (en) | Dynamic partitioning techniques for data streams | |
| US9276959B2 (en) | Client-configurable security options for data streams | |
| US10635644B2 (en) | Partition-based data stream processing framework | |
| US9794135B2 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| US8713046B2 (en) | Snapshot isolation support for distributed query processing in a shared disk database cluster | |
| CA2930026C (en) | Data stream ingestion and persistence techniques | |
| US8924357B2 (en) | Storage performance optimization | |
| US20130024484A1 (en) | System management in datacenter using a non-relational database | |
| US20160364407A1 (en) | Method and Device for Responding to Request, and Distributed File System | |
| CN109542861B (en) | File management method, device and system | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| US11151081B1 (en) | Data tiering service with cold tier indexing | |
| CN116069778A (en) | Metadata management method, related device, equipment and storage medium | |
| CN109753245A (en) | A kind of multiple disks load balancing asynchronous read and write dispatching method and device | |
| CN113127526A (en) | Distributed data storage and retrieval system based on Kubernetes | |
| CN103365740A (en) | Data cold standby method and device | |
| US20170270149A1 (en) | Database systems with re-ordered replicas and methods of accessing and backing up databases | |
| CN111414356A (en) | Data storage method and device, non-relational database system and storage medium | |
| US11727022B2 (en) | Generating a global delta in distributed databases | |
| US11232095B2 (en) | Composite metadata objects for database systems | |
| Liu et al. | User-oriented Mobile Filesystem Middleware for Mobile Cloud Systems. | |
| CN115757421A (en) | Data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |