CN102375694B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN102375694B CN102375694B CN201010259653.3A CN201010259653A CN102375694B CN 102375694 B CN102375694 B CN 102375694B CN 201010259653 A CN201010259653 A CN 201010259653A CN 102375694 B CN102375694 B CN 102375694B
- Authority
- CN
- China
- Prior art keywords
- storage unit
- data
- passage
- secondary index
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention provides a data processing method and a data processing device. The method comprises the following steps of: A, dividing a space except a reserved space in a disk array into more than one storage unit group of the same size, and dividing a space except a preset second-stage index space in each storage unit group into more than one storage unit of the same size; and B, sequentially writing data received within the same unit time into the same storage unit at the interval of unit time by using a cache of the disk array, and adding a first-stage index table and a second-stage index table of a video coder/decoder (EC) channel corresponding to each path of data into the reserved space and the second-stage index space respectively after the data is written into the same storage unit so as to inquire the path of data. By the method and the device, the data can be written into the disk array in a completely sequential manner.
Description
Technical field
The present invention relates to Video Supervision Technique, particularly data processing method and device.
Background technology
Redundant Array of Independent Disks (RAID) (RAID:Redundant Array of Independent Disks), be called for short disk array, its principle is to utilize multiple hard disks to be linked to be an array, realize and read and write data in magnetic disk in quick, accurate and safe mode, thereby reach the read or write speed and the security that improve data.
Utilize at present in the method for disk array processing video data, the wiring method of video data mainly comprised the following steps:
Step 1, for each passage of Video Codec (EC) is formulated corresponding storage plan.
Code stream size, the EC passage that the storage plan of EC passage comprises EC passage requires its time (being designated as the data holding time) that data of brush will be preserved down etc.
Step 2, for each EC passage, is disk array corresponding to this EC channel selecting according to the storage plan of this EC passage.
Step 2 be according to the data holding time in storage plan be EC channel selecting disk array.
Step 3, creates the continuous LUN resource in address in step 2 on the disk array of selecting, this LUN resource is associated with above-mentioned EC passage.
The size of LUN resource is to calculate according to the data holding time in storage plan, EC channel bit stream size.
If for 4 passages of EC are such as passage 1 to passage 4 has been selected identical redundant array of inexpensive disks 1, while carrying out step 3, can create the LUN resource that is associated with respectively these 4 passages on this redundant array of inexpensive disks 1 in step 2.Such as, on the redundant array of inexpensive disks 1 of selecting in step 2, create 4 LUN resources, be respectively LUN1 to LUN4, it is associated with respectively passage 1 to passage 4, specifically as shown in Figure 1.
Step 4, when select to step 2 disk array data writing time, by the lower brush data in order of each passage on EC, so that order is to the LUN resource data writing of this each EC passage association.
The highest owing to sequentially ensureing that to disk array data writing data write treatment effeciency, therefore, this step 4 orders, to LUN resource data writing, can improve data write efficiency, also improve the I/O processing power of disk array simultaneously.
But, in actual application, there will be multichannel EC passage with brushing at present the situation of data to disk array, such as the passage 1 in Fig. 1, passage 3 and passage 4 are with brushing data to redundant array of inexpensive disks 1 at present, due to the associated different LUN resource of different passages, so, just need the ceaselessly tracking of magnetic head of disk, presented the phenomenon that writes data at random disk array, this has reduced data write efficiency, has also reduced the I/O processing power of disk array simultaneously.
Summary of the invention
The invention provides data processing method and device, realize at multichannel EC passage simultaneously when brushing data under disk array, write these data to disk array with all-cis sequential mode, improve data write efficiency, also improve the I/O processing power of this disk array simultaneously.
Technical scheme provided by the invention comprises:
A kind of data processing method, comprising:
A is the identical storage unit group of more than 1 size by the spatial division except retaining space in disk array; Be the identical storage unit of more than 1 size by the spatial division except default secondary index space in each storage unit group;
B, the buffer memory of described disk array writes to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval, and after completing and writing, in described retaining space and described secondary index space, add respectively one-level concordance list and the secondary index table of EC passage corresponding to every circuit-switched data, for this circuit-switched data of inquiry.
A kind of data processing equipment, comprising: writing module, interpolation module and enquiry module, wherein,
Writing module, for writing to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval;
Add module, for after completing at writing module and writing, in retaining space and secondary index space, add respectively one-level concordance list and the secondary index table of EC passage corresponding to every circuit-switched data, inquire about this circuit-switched data for described enquiry module;
Described storage unit is by the spatial division except described secondary index space in storage unit group is obtained, and described storage unit group is by the spatial division except retaining space in disk array is obtained.
As can be seen from the above technical solutions, in the present invention, when multichannel EC passage is simultaneously when brushing data under same disk array, taking the unit interval as interval, the multichannel data order receiving in the same unit interval is write to same storage unit, realized the object of order to disk array data writing, improve data write efficiency, also improved the I/O processing power of disk array, reduced the risk that slowly causes loss of data in video traffic due to the response of storage end simultaneously.
Brief description of the drawings
Fig. 1 is the schematic diagram that available data writes disk array;
The disk array that Fig. 2 provides for the embodiment of the present invention is divided the system construction drawing of formation;
The processing flow chart that Fig. 3 provides for the embodiment of the present invention;
The processing flow chart of the step 308 that Fig. 4 provides for the embodiment of the present invention;
Fig. 5 reads process flow diagram for the data that the embodiment of the present invention provides;
The structure drawing of device that Fig. 6 provides for the embodiment of the present invention.
Embodiment
Method provided by the invention mainly comprises: A is the identical storage unit group of more than 1 size by the spatial division except retaining space in disk array; Be the identical storage unit of more than 1 size by the spatial division except secondary index space in each storage unit group; B, the buffer memory of this disk array writes to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval, and after completing and writing, in described retaining space and described secondary index space, add respectively one-level concordance list and the secondary index table of EC passage corresponding to every circuit-switched data, for this circuit-switched data of inquiry.Pass through the method, realize the object to data writing in disk array with all-cis sequential mode, improve data write efficiency, also improved the I/O processing power of disk array simultaneously, and, by one-level concordance list and the secondary index table of EC passage are set, can realize fast playback and the inquiry of data.
In order to make the object, technical solutions and advantages of the present invention clearer, the disc array structure first embodiment of the present invention being provided is below described.
If such as the storage plan of passage 1 has been selected redundant array of inexpensive disks 1, wherein, the data that comprise passage 1 time brush in storage plan need to be saved the code stream size of time (being designated as data holding time T1) and passage 1 etc. according to EC passage.If this redundant array of inexpensive disks 1 is to be selected first, this disk array is carried out to following operation, to obtain the structure shown in Fig. 2:
First, the space that reservation size is 9MB in redundant array of inexpensive disks 1, it is generally the space of a 9MB in redundant array of inexpensive disks 1.This space can comprise: the superblock space that the one-level index space that size is 8MB and size are 1MB.Wherein, one-level index space is for depositing the one-level concordance list (specifically below describing) of EC passage; And superblock space is for recording the superblock information of EC passage, it is little with the implementation relation of the embodiment of the present invention, no longer detailed description.
Secondly, spatial division except retaining space in redundant array of inexpensive disks 1 is become to M the storage unit group (group) that size is identical, and the spatial division of removing in each storage unit group outside default secondary index space (for depositing the secondary index table of EC passage, specifically in below description) is become to N the storage unit that size is identical.Conventionally, in the time dividing storage unit group, can determine 1 storage unit group according to practical application comprises how many storage unit (in the present embodiment, storage unit is by the unit of unit interval store data, therefore, it in fact namely determines that 1 storage unit group holds at most the data that receive in how many unit interval), determine M according to the data holding time T1 in this definite number and storage plan afterwards.
Such as, unit interval is 1 second, the number N that 1 group comprises storage unit is 1024, represent that 1 group holds at most the data that receive in 1024 seconds, so, if data holding time T1 is 20 days (be designated as 20*24*3600 second) in storage plan, the group number M dividing is:
M=20*24*3600 (second)/1024 (second)=1688.
If the space in redundant array of inexpensive disks 1 except retaining space is 16TB, the size of each storage unit group is: 16*1024*1024/1688=9939MB.
In addition, if secondary index space size is 16MB in each storage unit group, be generally the space of a 16MB in each storage unit group, the size of storage unit group is 9939B, and in storage unit group, the size of each storage unit is (9939-16)/1024=9.69MB.
So, by top-operation, redundant array of inexpensive disks 1 can form the structure shown in Fig. 2.
Based on the disc array structure shown in Fig. 2, the flow process below embodiment of the present invention being provided is described in detail.
Referring to Fig. 3, the detail flowchart that Fig. 3 provides for the embodiment of the present invention.As shown in Figure 3, this flow process can comprise the following steps:
Step 301, for each passage of EC, for this passage (taking passage 1 as example) is formulated storage plan.
Step 302, is that passage 1 is selected disk array according to the data holding time T1 in storage plan, and the disk array of this selection, for the data of 1 time brush of memory channel, is designated as redundant array of inexpensive disks 1.
This step can be that passage 1 is selected disk array according to existing mode, if the disk array quantity of selecting is many, can be using one of them as redundant array of inexpensive disks 1, and carry out following step.
Step 303, judges that whether redundant array of inexpensive disks 1 is by other EC channel selecting, if not, and execution step 304, if so, execution step 305.
Because redundant array of inexpensive disks 1 is to select according to the data holding time in storage plan, and in the storage plan of different EC passages, can comprise the identical data holding time, so exist redundant array of inexpensive disks 1 by the possibility of other EC channel selecting.So, while carrying out step 303, just need to judge that whether redundant array of inexpensive disks 1 is by other EC channel selecting, and carry out different operations according to different judged results, see that respectively step 304 is to step 305.
Step 304, becomes M the storage unit group that size is identical by the spatial division except retaining space in redundant array of inexpensive disks 1, and the spatial division except secondary index space in each storage unit group is become to N the storage unit that size is identical.The concrete operations of this step are described above.Perform step afterwards 307.
Step 305, judges whether redundant array of inexpensive disks 1 can also hold the data volume of 1 time brush of passage, if not, and execution step 306, if so, execution step 307.
If before execution step 302, existing passage 2 and 3 has been selected redundant array of inexpensive disks 1, and also selected redundant array of inexpensive disks 1 for passage 1 in step 302, so, while carrying out this step 305, judge this redundant array of inexpensive disks 1 except holding the data volume of 2 and 3 times brushes of passage, whether can also further hold the data volume of 1 time brush of passage, wherein, the data volume of brushing under passage is to calculate according to the code stream size of the data holding time in the storage plan of this passage and this passage, if can not, execution step 306, if can, execution step 307.
Step 306, if can also select other disk arrays except redundant array of inexpensive disks 1 to carry out the data that storage channels is brushed for 1 time according to the data holding time T1 in storage plan, carries out the operation of redundant array of inexpensive disks 1 according to step 303 for the disk array of this selection; Otherwise, finish current flow process.
Selection operation in this step 306 can be carried out according to existing mode.
Step 307, allows passage 1 to brush data under redundant array of inexpensive disks 1.Perform step afterwards 308.
Can determine to the passage of brushing data under redundant array of inexpensive disks 1 by above step 305 to step 307.All be allowed to brush data under redundant array of inexpensive disks 1 if determine multichannel EC passage by above-mentioned steps 305 to step 307, when this multichannel EC passage is simultaneously when brushing data under redundant array of inexpensive disks 1, execution step 308.
Step 308, the buffer memory of redundant array of inexpensive disks 1 writes to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval, and after completing and writing, in the retaining space (being specially the one-level index space in this retaining space) of redundant array of inexpensive disks 1 and the secondary index space of this storage unit place storage unit group, add respectively one-level concordance list and the secondary index table of EC passage corresponding to every circuit-switched data, for this circuit-switched data of inquiry.
How to realize as for step 308, be described below by the flow process shown in Fig. 4:
Step 401, in the time writing to redundant array of inexpensive disks 1 the data that receive in the current unit interval, judge current whether first to redundant array of inexpensive disks 1 data writing, if so, execution step 402; Otherwise, execution step 403;
Step 402, writes to the data that receive in this current unit interval first storage unit of first storage unit group in redundant array of inexpensive disks 1, performs step afterwards 406;
Step 403, if last storage unit that a upper storage unit is last storage unit group in redundant array of inexpensive disks 1 performs step 404; Otherwise, execution step 405.
The storage unit that the data that a described upper storage unit received for a upper unit interval write.
Step 404, the data of upgrading first storage unit in first storage unit group are the data that receive in the current unit interval, perform step afterwards 406.
That is to say, when data write on redundant array of inexpensive disks 1 after all storage unit, need to come back to first storage unit of first storage unit group, the data of preserving before covering.
Step 405, if the current data that are not also written into of the next storage unit of a upper storage unit directly write to this next one storage unit by the data that receive in the current unit interval, performs step 406 afterwards; Otherwise the data of upgrading in this next one storage unit are the data that receive in the current unit interval, perform step afterwards 406.
Can find out to step 405 by above step 401, the present embodiment is, taking the unit interval as interval, the data that receive in the same unit interval are write to same storage unit.Certainly, because the storage unit in redundant array of inexpensive disks 1 was divided according to the unit interval, if storage unit can not be held the data that receive in the same unit interval completely, can first fill up current storage unit to be written, the data that have more are write to next storage unit, and in the time writing the data of next unit interval reception, the data that this next one unit interval is received then write the vacant space of this storage unit, the like.
Realize with all-cis sequential mode to disk array data writing by above step 401 to step 405.
Step 406, for the every circuit-switched data that newly writes to storage unit, determines the EC passage that this route data is corresponding, judge whether the data of brushing under this EC passage last time write to the storage unit group at this storage unit place, to perform step 407, otherwise, execution step 408.
The every circuit-switched data that newly writes to storage unit in this step 406 specifically can be every circuit-switched data that step 402 or step 405 write, and also can be every circuit-switched data that renewal occurs in step 404 or step 405.
Step 407 is added the one-level concordance list of this EC passage in one-level index space, performs step afterwards 408.
That is to say, if data of this lower brush of EC passage and the deposit data of brushing under last time are in different storage unit groups, can write to after storage unit in the data of this lower brush, in one-level index space, record the one-level concordance list of this EC passage.
Preferably, the present embodiment adds operating speed for accelerating, and can first in buffer memory, carry out this interpolation operation, afterwards the one-level concordance list of interpolation is copied to one-level index space.
Step 408 is added the secondary index table of this EC passage in the secondary index space of this storage unit place storage unit group.
In the present embodiment, one-level concordance list and secondary index table are to be mainly convenient to follow-up data query manipulation, below one-level concordance list and secondary index table are described.
In the present embodiment, one-level concordance list size can be 128KB, and it is for recording the one-level index information of EC passage.The data of brushing under position in storage unit group of the title that wherein, the one-level index information of EC passage comprises this EC passage, the secondary index table of this EC passage (its can by start offset quantitative character), this EC passage write to the time of storage unit group and the numbering of this storage unit group.Taking the definite EC tunnel name of step 406 as EC_206-204 is as example, the one-level concordance list of this EC passage in one-level index space is as shown in table 1:
Table 1
| Video channel title | EC2004_206-204 |
| Secondary index table start offset amount | 24 |
| Storage unit group # | Write time |
| 1 | 2010_6_4_8:45:15 |
| 2 | 2010_6_4_8:50:15 |
| 3 | 2010_6_4_9:15:23 |
| 4 | 2010_6_4_9:30:23 |
| …… | …… |
In table 1, the secondary index table of EC passage is identical (certainly, also can be different, principle is similar) in the position of each storage unit group, and the start offset amount of secondary index table is identical.
As for secondary index table, its size can be 256KB, for recording the secondary index information of EC passage.Wherein, the secondary index packets of information of EC passage containing the size (by block quantitaes) of the data of brushing under this EC passage, position (numbering and start offset scale by storage unit show) in storage unit and accordingly acquisition time (be specially the time being gathered by EC passage, be designated as data time), specifically can join and be shown in Table 2.
Table 2
In table 2, occur that storage unit numbering is identical, but the situation that storage unit start offset amount is different, such as, in table 2, be numbered 2 storage unit, why there will be this situation, mainly owing to usually existing labile factor such as the different data of multiple data time that unstable networks etc. causes are brushed the buffer memory to redundant array of inexpensive disks 1 under quilt within a unit interval in video traffic, the data that are respectively 2010_6_49:00:08 and 2010_6_49:00:09 such as data time in table 2 arrive buffer memory in the same unit interval, buffer memory will enter these two data sequential write to the i.e. storage unit 2 in table 2 of same storage unit, there is above-mentioned situation.
So far, complete the description of one-level concordance list and secondary index table.Realize the one-level concordance list of EC passage and the interpolation of secondary index table operation by step 407 to step 408 above.It should be noted that, if in above-mentioned data writing process, need to delete the storage plan of a certain road EC passage, search and delete the one-level concordance list of this EC passage from the one-level index space of redundant array of inexpensive disks 1, and the location lookup of this EC passage secondary index table comprising according to this deleted one-level concordance list delete the secondary index table of this EC passage.
It should be noted that, in above-mentioned ablation process, if suspending, certain road EC passage brushes data under redundant array of inexpensive disks 1, here, EC passage suspends to brushing data under redundant array of inexpensive disks 1 and normally represents currently the data of the lower brush of this EC passage to be verified, in order to facilitate this verification work, need to ensure that the data of brushing under this EC passage are not updated, specifically can realize by following operation: the EC passage that first makes to suspend issues and suspends the buffer memory of notifying to redundant array of inexpensive disks 1; When receiving, buffer memory suspends after notice, in one-level index space, search the one-level concordance list of the EC passage of this time-out, find the secondary index table of this EC passage according to EC passage secondary index table position in this one-level concordance list, and in the secondary index table finding, obtain the position of the data of brushing under this EC passage in storage unit, lock afterwards this position, so that the data of this position are not updated.Such as, passage 1 needs to suspend brushes data under redundant array of inexpensive disks 1, this passage 1 will issue to suspend and notify the buffer memory of informing to redundant array of inexpensive disks 1, when this buffer memory receives after notice, in one-level index space, find the one-level concordance list of passage 1, the location lookup of this passage 1 secondary index table comprising according to this one-level concordance list finding is to the secondary index table of passage 1, the position of the data of obtaining this passage 1 time brush in the secondary index table that this finds in each storage unit, lock afterwards this position, so that the data of this position are not updated.
So, the renewal operation that above-mentioned steps 404 or step 405 are carried out specifically can comprise: if this storage unit comprises latched position, upgrading the data except latched position in this storage unit is the data that receive in the current unit interval; Otherwise the data of upgrading in this storage unit are the data that receive in the current unit interval.
In addition, this embodiment is for saving one-level index space and secondary index space resources, can be before step 404 or step 405 execution renewal operation, carry out following steps: the whether latched position of storage unit that first judges current data to be written, if so, determine EC passage corresponding to the data except latched position in this storage unit; If not, determine EC passage corresponding to data of depositing in this storage unit; Afterwards, in one-level index space, delete the one-level concordance list of this definite EC passage, and in the secondary index space of this storage unit place storage unit group, delete the secondary index table of this definite EC passage.
Step 409, the data that receive within for the per unit time complete data and write flow process after all carrying out according to above-mentioned steps 401 to step 408.
So far the data that, realized the embodiment of the present invention and provide write flow process.
Afterwards, if a certain passage such as passage 1 is to the cache request reading out data of redundant array of inexpensive disks 1, the time period of reading out data has been carried in this request, the flow process shown in execution graph 5.
Step 501 finds the one-level concordance list of passage 1 in one-level index space.
The title that comprises passage due to one-level concordance list, therefore, the name that this step 501 can passage 1 is called the one-level concordance list that key word comprises this key word the search of one-level index space.
Step 502, from this one-level concordance list, determine the write time early than and approach the numbering of the storage unit group of this time period initial time most, and determine the numbering of the storage unit group that is later than and approaches most this end time time period.
Such as, if the time period of carrying in request is 2010_6_4_8:55:10 to 2010_6_4_9:28:10, the name of passage 1 is called EC_206-204, according to table 1 can obtain the write time early than and approach most 2010_6_4_8:55:10 storage unit group be numbered 2, and be later than and approach most 2010_6_4_9:28:10 storage unit group be numbered 4.
Step 503, the secondary index table that finds this EC passage in one-level concordance list respectively each between above-mentioned two numberings and above-mentioned two numberings is numbered the position in corresponding storage unit group, the data time, Data Position and the described time period data query that comprise according to this locational secondary index table.
Can find out according to table 1, data are each storage unit groups that write to successively disk array according to time order and function order, therefore, when two storage unit groups corresponding to numbering determining in step 502 meet the above-mentioned time period, each between these two numberings numbered corresponding storage unit group and certainly also met the above-mentioned time period.Such as, if step 502 determine the write time early than, and approach most 2010_6_4_8:55:10 storage unit group be numbered 2, be later than, and approach most 2010_6_4_9:28:10 storage unit group be numbered 4, can know according to table 1, the corresponding storage unit group of numbering 3 between this numbering 2 and numbering 4 also meets this time period, therefore, this step need to determine that the secondary index table of this EC passage is respectively in numbering 2, position in 3 and 4 three storage unit groups, the data time comprising according to this locational secondary index table afterwards, Data Position, and described time period data query.
So far, realize data and read flow process.
The method above embodiment of the present invention being provided is described, and the device below embodiment of the present invention being provided is described.
Referring to Fig. 6, the structure drawing of device that Fig. 6 provides for the embodiment of the present invention.As shown in Figure 6, this device comprises: writing module 601, interpolation module 602 and enquiry module 603,
Wherein, writing module 601, at multichannel EC passage simultaneously when brushing data under disk array, the each circuit-switched data order receiving in the same unit interval is write to storage unit;
Add module 602, for after completing at writing module 601 and writing, in retaining space and secondary index space, add respectively one-level concordance list and the secondary index table of EC passage corresponding to every circuit-switched data, for enquiry module 603 these circuit-switched data of inquiry;
Wherein, described storage unit is by the spatial division except described secondary index space in storage unit group is obtained, and described storage unit group is by the spatial division except retaining space in disk array is obtained.
As shown in Figure 6, while adding module 602 specific implementation, can comprise:
First judges submodule 6021, for the corresponding EC passage of every circuit-switched data writing for writing module 601, judge whether the data of brushing under this EC passage last time write to the storage unit group at described storage unit place, if so, send second and add notice to the second interpolation submodule 6023; Otherwise, send first and add notice to the first interpolation submodule 6022;
First adds submodule 6022, while notice, adds the one-level concordance list of this EC passage in retaining space for receiving described the first interpolation, sends afterwards the 3rd and adds notice to the second interpolation submodule 6023; Numbering, the data that this one-level concordance list comprises described storage unit group write to time, the title of this EC passage and the position of secondary index table thereof of this storage unit group;
Second adds submodule 6023, while adding notice, adds the secondary index table of this EC passage in the secondary index space of this storage unit group for receiving described the second interpolation notice or the 3rd; This secondary index table comprises this circuit-switched data at the position of storage unit and collected data time.
Preferably, as shown in Figure 6, writing module 601 can comprise:
Second judges submodule 6011, in the time writing to disk array the data that receive in the current unit interval, judge current whether first to disk array data writing, if so, send first write notice to writing submodule 6012; Otherwise, judge whether a upper storage unit is last storage unit of last storage unit group in disk array, a described upper storage unit is the storage unit writing a upper unit interval, if so, sends second and writes notice to writing submodule 6012; Otherwise, send the 3rd and write notice to writing submodule 6012;
Write submodule 6012, receive first and write after notice, the data that receive are write to first storage unit of first storage unit group in described disk array in the current unit interval; Receive second and write after notice, the data of upgrading first storage unit in first storage unit group are the data that receive in the current unit interval; Receiving the 3rd writes after notice, if also data writing not of the next storage unit of a upper storage unit, the data that receive in the current unit interval are write to this next one storage unit, otherwise the data of upgrading in this next one storage unit are the data that receive in the current unit interval.
Preferably, as shown in Figure 6, when enquiry module 603 specific implementation, can comprise:
Search submodule 6031, when receiving the data of EC channel request reading disk array, described request has been carried the time period of reading out data, finds the one-level concordance list of this EC passage in retaining space;
Determine submodule 6032, for from search one-level concordance list that submodule 6031 finds determine the write time early than and approach the numbering 1 of the storage unit group of this time period initial time most, and determine the numbering 2 of the storage unit group that is later than and approaches most this end time time period;
Inquiry submodule 6033, for the secondary index table that finds this EC passage at the one-level concordance list position in each storage unit group corresponding to each numbering between numbering 1, numbering 2 and numbering 1 and numbering 2 respectively, the data time comprising according to this locational secondary index table, Data Position and described time period data query.
As can be seen from the above technical solutions, in the present invention, when multichannel EC passage is simultaneously when brushing data under same disk array, taking the unit interval as interval, the multichannel data order receiving in the same unit interval is write to same storage unit, realized the object of order to disk array data writing, improve data write efficiency, also improved the I/O processing power of disk array, reduced the risk that slowly causes loss of data in video traffic due to the response of storage end simultaneously.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any amendment of making, be equal to replacement, improvement etc., within all should being included in the scope of protection of the invention.
Claims (10)
1. a data processing method, is characterized in that, the method comprises:
A is the identical storage unit group of more than 1 size by the spatial division except retaining space in disk array; Be the identical storage unit of more than 1 size by the spatial division except default secondary index space in each storage unit group;
B, the buffer memory of described disk array writes to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval, and after completing and writing, in described retaining space, add the one-level concordance list of EC passage corresponding to every circuit-switched data, in described secondary index space, add the secondary index table of EC passage corresponding to every circuit-switched data, for this circuit-switched data of inquiry.
2. method according to claim 1, is characterized in that, the interpolation in described step B comprises:
B11, for EC passage corresponding to every circuit-switched data, if the data that this EC passage was brushed under last time write to the storage unit group at this storage unit place, execution step B13, otherwise, execution step B12;
B12 adds the one-level concordance list of this EC passage in retaining space, performs step afterwards B13; Numbering, the data that this one-level concordance list comprises this storage unit group write to time, the title of this EC passage and the position of secondary index table thereof of this storage unit group;
B13 adds the secondary index table of this EC passage in the secondary index space of this storage unit group;
This secondary index table comprises this circuit-switched data in the position of storage unit and the time of collected data.
3. method according to claim 1, is characterized in that, in described step B, writes and comprises:
B21, in the time writing to disk array the data that receive in the current unit interval, be first to disk array data writing if current, the data that receive in the current unit interval write to first storage unit of first storage unit group in described disk array; Otherwise, execution step B22;
B22, in the time that a upper storage unit is last storage unit of last storage unit group in disk array, a described upper storage unit is the storage unit writing a upper unit interval, execution step B23; Otherwise, execution step B24;
B23, the data of upgrading first storage unit in first storage unit group are the data that receive in the current unit interval;
B24, in the next storage unit of a upper storage unit also not when data writing, the data that receive in the current unit interval are write to this next one storage unit, otherwise the data of upgrading in this next one storage unit are the data that receive in the current unit interval.
4. method according to claim 3, is characterized in that, in the time receiving the time-out notice that EC passage issues to described buffer memory, the method further comprises:
In retaining space, find the one-level concordance list of this EC passage;
Secondary index table according to the location lookup of this EC passage secondary index table in one-level concordance list to this EC passage;
From this secondary index table, obtain the position of the data of brushing under this EC passage in each storage unit, lock afterwards this position;
In described step B23 or step B24, upgrade and comprise:
If there is latched position in storage unit, upgrading the data except latched position in this storage unit is the data that receive in the current unit interval, otherwise the data of upgrading in this storage unit are the data that receive in the current unit interval.
5. method according to claim 4, is characterized in that, described step B23 or step B24 further comprised before the data of upgrading in storage unit:
Judge whether this storage unit comprises latched position, if so, determine EC passage corresponding to the data except latched position in this storage unit; If not, determine EC passage corresponding to data of depositing in this storage unit;
In described retaining space, delete the one-level concordance list of this definite EC passage, and in the secondary index space of this storage unit place storage unit group, delete the secondary index table of this definite EC passage.
6. method according to claim 1, is characterized in that, the inquiry in described step B comprises:
When receive in EC channel request reading disk array data time, described request has been carried the time period of reading out data, finds the one-level concordance list of this EC passage in retaining space;
From this one-level concordance list, determine the write time early than and approach the numbering 1 of the storage unit group of this time period initial time most, and determine the numbering 2 of the storage unit group that is later than and approaches most this end time time period;
The secondary index table that finds this EC passage in one-level concordance list is the position in each storage unit group corresponding to each numbering between numbering 1, numbering 2 and numbering 1 and numbering 2 respectively, the data time comprising according to this locational secondary index table, Data Position and described time period data query.
7. a data processing equipment, is characterized in that, this device comprises: writing module, interpolation module and enquiry module, wherein,
Writing module, for writing to same storage unit taking the unit interval as interval by the data order receiving in the same unit interval;
Add module, for after writing module completes and writes, in retaining space, add the one-level concordance list of EC passage corresponding to every circuit-switched data, in secondary index space, add the secondary index table of EC passage corresponding to every circuit-switched data, inquire about this circuit-switched data for described enquiry module;
Described storage unit is by the spatial division except described secondary index space in storage unit group is obtained, and described storage unit group is by the spatial division except retaining space in disk array is obtained.
8. device according to claim 7, is characterized in that, described interpolation module comprises:
First judges submodule, for the corresponding EC passage of every circuit-switched data writing for said write module, judge whether the data of brushing under this EC passage last time write to the storage unit group at described storage unit place, if so, send second and add notice to the second interpolation submodule; Otherwise, send first and add notice to the first interpolation submodule;
Described first adds submodule, while notice, adds the one-level concordance list of this EC passage in retaining space for receiving described the first interpolation, sends afterwards the 3rd interpolation notice and adds submodule to described second; Numbering, the data that this one-level concordance list comprises described storage unit group write to time, the title of this EC passage and the position of secondary index table thereof of this storage unit group;
Described second adds submodule, while adding notice, adds the secondary index table of this EC passage in the secondary index space of this storage unit group for receiving described the second interpolation notice or the 3rd; This secondary index table comprises this circuit-switched data in the position of storage unit and the time of collected data.
9. device according to claim 8, is characterized in that, said write module comprises:
Second judges submodule, in the time writing to disk array the data that receive in the current unit interval, judge current whether first to disk array data writing, if so, send first write notice to writing submodule; Otherwise, judge whether a upper storage unit is last storage unit of last storage unit group in disk array, a described upper storage unit is the storage unit writing a upper unit interval, if so, sends second and writes notice to writing submodule; Otherwise, send the 3rd and write notice to writing submodule;
Write submodule, receive first and write after notice, the data that receive are write to first storage unit of first storage unit group in described disk array in the current unit interval; Receive second and write after notice, the data of upgrading first storage unit in first storage unit group are the data that receive in the current unit interval; Receiving the 3rd writes after notice, if also data writing not of the next storage unit of a upper storage unit, the data that receive in the current unit interval are write to this next one storage unit, otherwise the data of upgrading in this next one storage unit are the data that receive in the current unit interval.
10. device according to claim 8, is characterized in that, described enquiry module comprises:
Search submodule, when receiving the data of EC channel request reading disk array, described request has been carried the time period of reading out data, finds the one-level concordance list of this EC passage in retaining space;
Determine submodule, for from search one-level concordance list that submodule finds determine the write time early than and approach the numbering 1 of the storage unit group of this time period initial time most, and determine the numbering 2 of the storage unit group that is later than and approaches most this end time time period;
Inquiry submodule, for the secondary index table that finds this EC passage at the one-level concordance list position in each storage unit group corresponding to each numbering between numbering 1, numbering 2 and numbering 1 and numbering 2 respectively, the data time comprising according to this locational secondary index table, Data Position and described time period data query.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010259653.3A CN102375694B (en) | 2010-08-20 | 2010-08-20 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010259653.3A CN102375694B (en) | 2010-08-20 | 2010-08-20 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102375694A CN102375694A (en) | 2012-03-14 |
| CN102375694B true CN102375694B (en) | 2014-09-10 |
Family
ID=45794351
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010259653.3A Active CN102375694B (en) | 2010-08-20 | 2010-08-20 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102375694B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722454B (en) * | 2012-05-22 | 2015-05-06 | 浙江宇视科技有限公司 | Disk protecting method and device |
| CN103677655B (en) * | 2012-09-26 | 2016-12-21 | 北京信威通信技术股份有限公司 | A kind of two-dimemsional number group data stream reading/writing method on a memory and device |
| CN103678025B (en) * | 2013-12-02 | 2017-01-04 | 创新科软件技术（深圳）有限公司 | A kind of disk failure processing method in disk array |
| CN105354151B (en) * | 2014-08-19 | 2020-09-11 | 阿里巴巴集团控股有限公司 | Cache management method and equipment |
| CN104182354B (en) * | 2014-08-26 | 2017-12-15 | 浙江宇视科技有限公司 | A kind of video management method and device |
| CN104869338B (en) * | 2015-05-18 | 2019-03-12 | 浙江宇视科技有限公司 | A kind of audio-video document unloading and search method |
| CN111506259B (en) * | 2020-03-20 | 2023-06-06 | 普联技术有限公司 | Data storage method, data reading method, data storage device, data reading apparatus, data storage device, and readable storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8666230B2 (en) * | 2005-10-14 | 2014-03-04 | Microsoft Corporation | Multiple physical optical disc formats in backwards compatible order |
| CN101539884B (en) * | 2009-04-30 | 2012-07-11 | 杭州华三通信技术有限公司 | Method and device for data storage |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN101729421B (en) * | 2009-12-11 | 2013-03-20 | 杭州华三通信技术有限公司 | Storage method and device based on time division multiplex |
-
2010
- 2010-08-20 CN CN201010259653.3A patent/CN102375694B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN102375694A (en) | 2012-03-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102375694B (en) | Data processing method and device | |
| US9684462B2 (en) | Method and apparatus utilizing non-uniform hash functions for placing records in non-uniform access memory | |
| CN102156738B (en) | Method for processing data blocks, and data block storage equipment and system | |
| CN103019953B (en) | Construction system and construction method for metadata | |
| CN102968422B (en) | System and method for controlling streaming data storage | |
| CN106201355B (en) | Data processing method and device and storage system | |
| US20140223089A1 (en) | Method and device for storing data in a flash memory using address mapping for supporting various block sizes | |
| CN100585740C (en) | Flash memory apparatus and access method to flash memory | |
| CN106970765B (en) | Data storage method and device | |
| CN102779180A (en) | Operation processing method of data storage system and data storage system | |
| KR20030011100A (en) | Methods and devices for recording or reading files on/from a sequential medium and sequential medium | |
| CN103049390A (en) | Applied metadata processing method and storing system | |
| CN104461390A (en) | Method and device for writing data into imbricate magnetic recording SMR hard disk | |
| CN104156380A (en) | Distributed memory Hash indexing method and system | |
| CN102314485A (en) | Method and device for adding, searching and deleting hash table | |
| CN108399050B (en) | Data processing method and device | |
| JP2005196769A (en) | Data management device and method for flash memory | |
| CN102760136A (en) | File system control method based on rapid video data storage | |
| CN101963977A (en) | A search method and mobile terminal without urban search | |
| US11409720B2 (en) | Metadata reduction in a distributed storage system | |
| CN102129619A (en) | Service data processing method and system based on cloud storage | |
| CN102426601B (en) | Data deleting method and device | |
| CN104050057A (en) | Historical sensed data duplicate removal fragment eliminating method and system | |
| CN103761194B (en) | A kind of EMS memory management process and device | |
| CN102622315A (en) | Method, device and system for controlling disc visiting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| ASS | Succession or assignment of patent right |
Owner name: ZHEJIANG UNIVIEW TECHNOLOGY CO., LTD. Free format text: FORMER OWNER: HUASAN COMMUNICATION TECHNOLOGY CO., LTD. Effective date: 20120222 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20120222 Address after: Hangzhou City, Zhejiang province 310053 Binjiang District Dongxin Road No. 66 building two or three layer A C Applicant after: Zhejiang Uniview Technology Co., Ltd. Address before: 310053 Hangzhou hi tech Industrial Development Zone, Zhejiang province science and Technology Industrial Park, No. 310 and No. six road, HUAWEI, Hangzhou production base Applicant before: Huasan Communication Technology Co., Ltd. |
|
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |