CN104572655A - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN104572655A CN104572655A CN201310476079.0A CN201310476079A CN104572655A CN 104572655 A CN104572655 A CN 104572655A CN 201310476079 A CN201310476079 A CN 201310476079A CN 104572655 A CN104572655 A CN 104572655A
- Authority
- CN
- China
- Prior art keywords
- bit
- data
- project
- destination
- item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a data processing device and a data processing system, relates to the technical field of computers, and solves the problem that a memory resource is wasted because subsequently added data is stored by adding storage fields in the prior art. The method comprises the following steps: acquiring bits occupied by each existing project corresponding to the corresponding field in a data structure and the maximum length value of the data under each existing project; determining applied bits of each existing project according to the maximum length value; determining a target project in the existing projects according to the occupied bits and the applied bits; determining unapplied bits according to the occupied bits and the applied bits of the target project; distributing the unapplied bits to a to-be-added project; acquiring the data of the to-be-added project, and writing the data of the to-be-added project into the unapplied bits. The data processing method, the data processing device and the data processing system are suitable for a shared memory technology.
Description
Technical field
The present invention relates to field of computer technology, particularly relate to a kind of method of data processing, Apparatus and system.
Background technology
Current, along with the development of computer technology, share memory technology is more and more general.Shared drive refers in the computer system having multiple processor, all addressable Large Copacity internal memory of different processors.When multiple processor needs to conduct interviews to the data in storer, need to carry out buffer memory to the data in storer.The real-time of share memory technology is comparatively strong, and after a processor carries out Data Update to buffer memory wherein, the data in shared drive also need to upgrade immediately, to avoid other processor access to different data.At present, for the system that access real-time is stronger, general employing shared drive stores data structure.
In data structure, can store according to data type, such as char type data may take 8 bit storage positions, represent with unit8_t, and such as short type data may take 16 bit storage positions again, represent with unit16_t.In a data structure in shared drive, several data can be stored, such as when a certain bar microblogging in microblogging field is forwarded, in a data structure can this microblogging of storage forwarding User Identity, relay comment the identify label of this microblogging, the relay counting etc. of this microblogging.In addition, in this data structure, be also previously provided with reserved storage field, when needs are added to the data needing to store, application is reserved storage field and is stored.Store field if reserved to be employed, and also need to be added to new data, then need to expand shared drive, continue increase and store field.Storing the follow-up data added by increasing storage field, causing the waste of memory source.
Summary of the invention
Embodiments of the invention provide a kind of method, Apparatus and system of data processing, can solve in prior art and store the follow-up data added owing to storing field by increase, cause the problem of the waste of memory source.
For achieving the above object, the present invention adopts following technical scheme:
A method for data processing, comprising:
Obtain each length maximal value that there are the bit shared by project and the data under respectively having there is project that each field in data structure is corresponding;
According to described length maximal value, determine the bit of application that respectively there is project;
According to described shared bit and describedly apply bit, determine the destination item deposited in the project;
Bit shared by described destination item and applied bit and determine not apply bit;
Described bit of not applying is distributed to project to be increased;
Obtain described item destination data to be increased, and do not apply in bit described in described item destination data to be increased is written to.
A method for data processing, comprising:
Obtain the project information of each field in data structure; Described project information comprises the position that project corresponding to described each field and described item destination data store in described field;
Judge whether described field comprises multiple destination datas according to described project information;
If determine described field to comprise multiple destination datas, according to the position that described item destination data stores in described field, every destination data is read.
A kind of data storage device, comprising:
Each length maximal value that there are the bit shared by project and the data under respectively having there is project that in acquiring unit, acquisition data structure, each field is corresponding;
Determining unit, for the described length maximal value obtained according to described acquiring unit, determines the bit of application that respectively there is project;
Described determining unit, also for according to described shared bit and describedly apply bit, determines the destination item deposited in the project;
Described determining unit, also for the bit shared by described destination item and applied bit and determine not apply bit;
Allocation units, distribute to project to be increased for not applying bit described in described determining unit being determined;
Writing unit, for obtaining the item destination data described to be increased that described allocation units distribute, and does not apply in bit described in described item destination data to be increased being written to.
A kind of data fetch device, comprising:
Acquiring unit, for obtaining the project information of each field in data structure; Described project information comprises the position that project corresponding to described each field and described item destination data store in described field;
Judging unit, the described project information for obtaining according to described acquiring unit judges whether described field comprises multiple destination datas;
Reading unit, comprising multiple destination datas for determining described field at described judging unit, reading according to the position that described item destination data stores in described field to every destination data.
A system for data processing, comprises above-mentioned data storage device and above-mentioned data fetch device.
The method of the data processing that the embodiment of the present invention provides, Apparatus and system, determine in data structure the destination item deposited in the project, and determine described destination item application bit, this is not applied bit and does not store described target item destination data, thus described bit of not applying is distributed to project to be increased, and do not apply in bit described in described item destination data to be increased is written to, thus store field without the need to increasing, can store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
The process flow diagram of the method for the data processing that Fig. 1 provides for the embodiment of the present invention;
The process flow diagram of the method for the data processing that Fig. 2 provides for further embodiment of this invention;
The process flow diagram one of the method for the data processing that Fig. 3 provides for another embodiment of the present invention;
Fig. 4 is the schematic diagram of a kind of data structure in the embodiment of the present invention;
Fig. 5 is the schematic diagram of the another kind of data structure in the embodiment of the present invention;
The flowchart 2 of the method for the data processing that Fig. 6 provides for another embodiment of the present invention;
The flow chart 3 of the method for the data processing that Fig. 7 provides for another embodiment of the present invention;
The structural representation one of the data storage device that Fig. 8 provides for the embodiment of the present invention;
The structural representation two of the data storage device that Fig. 9 provides for the embodiment of the present invention;
The structural representation one of the data fetch device that Figure 10 provides for the embodiment of the present invention;
The structural representation two of the data fetch device that Figure 11 provides for the embodiment of the present invention;
The schematic diagram of the system of the data processing that Figure 12 provides for the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
For making the advantage of technical solution of the present invention clearly, below in conjunction with drawings and Examples, the present invention is elaborated.
As shown in Figure 1, the method for the data processing that the embodiment of the present invention provides, comprising:
101, each length maximal value that there are the bit shared by project and the data under respectively having there is project that each field in data structure is corresponding is obtained.
In data structure, as multiple field can be had in the data structure of shared drive, each field can store the project data of different types of data, generally different project datas, data type is also different, such as, for the situation that a certain bar microblogging in microblogging field is forwarded, in data structure, generally can record following project: User Identity (Identity, be called for short ID), turn comment ID, relay counting, comment counting etc.Generally can determine that it needs the bit taken according to the length maximal value of the data under above-mentioned project, if the figure place of such as user ID is 9 figure places to the maximum, namely maximal value is for then to need 32 bits to carry out record, namely adopts int type, take 32 bits, represent with unit32_t.But the bit also existed shared by project is far longer than the length maximal value of this project data, such as, under relaying counting, the length value of data is generally 5-7 position, namely relays counting and may reach several ten thousand to millions of, but the bit of relaying shared by counting also can be take 32 bits.
102, according to described length maximal value, the bit of application that respectively there is project is determined.
Described length maximal value represents the bit applicable cases that there is project, thus determine the bit of application that there is project, under relay counting as escribed above, the length maximal value of data is 7, namely relay counting maximum possible and reach millions of, then the bit of its actual needs is 24 bits, therefore, determines to determine that having applied bit is 24 bits.
103, according to described shared bit and describedly apply bit, the destination item deposited in the project is determined.
Generally, if shared bit is comparatively close with the value having applied bit, then illustrate that not apply bit in shared bit less.Therefore, can adopt according to the bit existed shared by project and describedly apply bit, determine described shared bit and the described difference having applied bit, judge whether described difference is greater than a predetermined threshold value, if described difference is greater than described predetermined threshold value, then having there is project described in determining is described destination item.Such as described predetermined threshold value can be 8 compare top grade.
104, the bit shared by described destination item and applied bit and determine not apply bit.
In the bit shared by described destination item, can using except the bit applied except bit be as not applying bit.In addition, can also be shifted to applying bit, determining after shifting not apply bit.Such as shared bit is uint32_t, apply bit and occupy low 24 bits in uint32_t, and high 8 bits are vacant, then can using high 8 bits as not applying bit, also 8 bits can be moved applying bit to a high position, thus vacant go out low 8 bits, then can using low 8 bits as not applying bit.
105, described bit of not applying is distributed to project to be increased.
Bit will do not applied and distribute to project to be increased, need the mapping relations recording the project to be increased bit corresponding with it, in follow-up data write with when reading, can directly get the position that will write and read according to project to be increased.
106, obtain described item destination data to be increased, and do not apply in bit described in described item destination data to be increased is written to.
When obtaining item destination data to be increased, also increased a destination data in acquisition simultaneously, generally that projects data are first stored in the temporary storage of the correspondence of projects, and when being written in the bit increased shared by project, need first to be shifted to value corresponding to the data of project to be increased and the value that increased a destination data corresponding, carry out position or process again, thus in a field, store at least two item destination datas.
What deserves to be explained is, the executive agent of the method for the data processing that the embodiment that the embodiment of the present invention provides provides can be a kind of data storage device, can write data in shared drive.
The method of the data processing that the embodiment of the present invention provides, determine in data structure the destination item deposited in the project, and determine described destination item application bit, this is not applied bit and does not store described target item destination data, thus described bit of not applying is distributed to project to be increased, and do not apply in bit described in described item destination data to be increased is written to, thus store field without the need to increasing, can store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
As shown in Figure 2, the method for the data processing that further embodiment of this invention provides, comprising:
201, the project information of each field in data structure is obtained.
Wherein, described project information comprises the position that project corresponding to described each field and described item destination data store in described field.The data that such as data structure comprises field 1, field 2, field 1 only store project 1, field 2 store the data of project 2 and project 3, wherein, the data of project 2 take high 24 bits of field 2, and the data of project 3 take low 8 bits of field 2, but are not only confined to this.
202, judge whether described field comprises multiple destination datas according to described project information.
By project corresponding in each field, directly can determine a field and whether store multiple destination datas.
If 203 determine described field comprise multiple destination datas, according to the position that described item destination data stores in described field, every destination data is read.
What deserves to be explained is, the executive agent of the method for the data processing that the embodiment that the embodiment of the present invention provides provides can be a kind of data fetch device, may be used for the reading to the data in shared drive.
The method of the data processing that further embodiment of this invention provides, multiple the destination datas due to the field store in data structure, thus store field without the need to increasing, can to store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
Enumerate a more specifically embodiment below, as shown in Figure 3, the method for the data processing that another embodiment of the present invention provides, comprising:
301, each length maximal value that there are the bit shared by project and the data under respectively having there is project that each field in data structure is corresponding is obtained.
Its specific implementation see shown in the step 101 in Fig. 1, can repeat no more herein.
302, according to described length maximal value, the bit of application that respectively there is project is determined.
Its specific implementation see shown in the step 102 in Fig. 1, can repeat no more herein.
303, according to the bit existed shared by project and describedly apply bit, described shared bit and the described difference having applied bit is determined.
Generally, if shared bit is less with the difference having applied bit, then illustrate that not apply bit in shared bit less, otherwise, if shared bit is comparatively large with the difference having applied bit, then do not apply bit larger.
304, judge whether described difference is greater than a predetermined threshold value.
If described difference is less than or equal to described predetermined threshold value, perform step 305.If described difference is greater than described predetermined threshold value, perform step 306.
305, determine described in there is project for described destination item.
Be not the project of destination item, in the bit shared by it not apply bit less.
306, determine described in have there is project be described destination item.Continue afterwards to perform step 307.
307, in described shared bit, bit except bit has been applied as not applying bit using except described.
Concrete, apply bit except bit as before not applying bit described using except described, described application in the bit of bit shared by described can also have been carried out shifting processing, described in redefining, apply bit.
In addition, what one can also have been existed project does not apply bit the has existed project bit of application as another, thus vacant go out another there is the former of project and applied bit.Specifically described destination item comprises first object project and the second destination item, the bit of not applying of described first object project is distributed to described second destination item, described first object project do not apply bit for partly or entirely applying bit as described second destination item.Thus in the bit shared by described second destination item, redefine the bit of application of the second destination item, apply bit except bit as not applying bit using except described.
Such as, data structure schematic diagram as shown in Figure 4, field 1 is included in a data structure, field 2, field 3 and field 4, corresponding each bit existed shared by project is 32 bits, 64 bits, 32 bits and 32 bits, in the diagram, field 1 and field 2 correspondence be non-destination item, and field 3 and field 4 correspondence is first object project and the second destination item respectively, first object project and the second destination item have been applied bit and have been 24 bits, then can using the application bit of high 24 bits of field 3 as first object project, using low 8 bits of field 3 as not applying bit, using the application bit of high 24 bits of field 4 as the second destination item, using low 8 bits of field 4 as not applying bit, in addition, as shown in Figure 5, can also using high 8 bits of field 3 and field 4 as not applying bit, low 24 bits of field 3 are as the bit of application of first object project, and low 24 bits of field 4 are as the bit of application of the second destination item.Above-mentioned Fig. 4, Fig. 5 are only two kinds of bit allocation scheme; except above-mentioned two kinds of bit allocation scheme; can also have plurality of distribution mode, the allocation scheme that those skilled in the art obtain when not paying creative work is all in protection scope of the present invention.
308, the bit of not applying of one or more destination item is distributed to described project to be increased.After step 308, perform step 309, or perform step 311.
Bit will do not applied and distribute to project to be increased, need the mapping relations recording the project to be increased bit corresponding with it, in follow-up data write with when reading, can directly get the position that will write and read according to project to be increased.
If the bit of not applying of multiple destination item is distributed to described project to be increased by 309, obtaining described item destination data to be increased, is multiple partial data by the Data Division of described project to be increased.
310, described multiple partial data is written to not applying in bit of each destination item respectively.
Such as, situation about being forwarded for a certain bar microblogging in microblogging field, its data structure are
Struct wbTweetInfo{
Uint32_t dwUid; / * user ID */
Uint64_t ddwTweetId; / * turn comment ID*/
Uint32_t dwCount; / * relay counting */
Uint32_t dwCommentCount; / * comment counting */
};
Wherein, wbTweetInfo represents described data structure, 4 fields are had in this wbTweetInfo data structure, dwUid represents the user ID accounting for 32 bits, ddwTweetId represents that accounting for turning of 64 bits comments ID, dwCount represents the relay counting accounting for 32 bits, and dwCommentCount represents the comment counting accounting for 32 bits.The bit of application of relaying counting and comment counting is 24 bits, and not applying bit is 8 bits, such as all using high 24 bits as applying bit, and using low 8 bits as not applying bit, store the data of sundry item.
When needs increase project in above-mentioned wbTweetInfo data structure, such as increase this microblogging in micro-group by the number of times of relaying, namely micro-group counting, represents with wWQCount, then the bit of not applying of above-mentioned two low 8 bits can be distributed to described micro-group counting project.
When getting every destination data, can first be stored in temporary storage by projects, the interim data structure stored be:
Struct TweetInfoNew{
Uint32_t dwNewUid; / * user ID */
Uint64_t ddwNewTweetId; / * turn comment ID*/
Uint32_t dwNewCount; / * relay counting */
Uint32_t dwNewCommentCount; / * comment counting */
Uint16_t wWQCount; / * micro-group counting */
};
Wherein, TweetInfoNew represents this data structure temporarily stored, dwNewUid represents the user ID in interim storage, ddwNewTweetId represents that turning in interim storage comments ID, dwNewCount represents the relay counting in interim storage, and dwNewCommentCount represents the comment counting in interim storage.Specifically need the every destination data in the data structure in above-mentioned temporary storage to be written in the data structure that wbTweetInfo represents, concrete mode can be as follows:
WbTweetInfo.dwUid=TweetInfoNew.dwNewUid; This formula represents that the direct value by each bit of user ID in storage is temporarily written in each bit of user ID corresponding field in wbTweetInfo data structure.
WbTweetInfo.ddwTweetId=TweetInfoNew.ddwNewTweetId; This formula represents that directly commenting the value of each bit of ID to be written to the transfer of wbTweetInfo data structure turning in storage temporarily comments in each bit of ID corresponding field.
wbTweetInfo.dwCount=(TweetInfoNew.dwNewCount<<8)|
(TweetInfoNew.wWQCount>>8); This formula represents each bit of relaying counting in temporarily storing is moved 8 bits to a high position, obtain the first value, each bit of micro-group counting in storage is temporarily moved 8 bits to low level, obtain the second value, described first value and the second value are carried out position or process, thus the value of each bit is written in each bit of relaying counting corresponding field in wbTweetInfo data structure.
Such as: each bit of relaying counting in interim storage is:
00000000111111111111111011101111
Then described first value is: 11111111111111101110111100000000, and after visible first value produces, vacant is low 8 bits.
In interim storage, each bit of micro-group counting is:
1111111111110111
Then described second value is: 0000000011111111
After then the first value and the second value carry out position or process, the value of each bit obtained is:
11111111111111101110111111111111, visible first value and the second value carry out position or after value be the value of high 8 bits of the value of each bit and the micro-group counting of relaying counting.
WbTweetInfo.dwCommentCount=(TweetInfoNew.dwNewCommentCount<LEssT.LTssT. LT8) | (TweetInfoNew.wWQCount<<8GreatT .GreaT.GT>8); This formula represents each bit commenting on counting in temporarily storing is moved 8 bits to a high position, obtain the 3rd value, by each bit of micro-group counting in storage temporarily to a high position first mobile 8 bits, 8 bits are moved again to low level, obtain the 4th value, described 3rd value and the 4th value are carried out position or process, thus the value of each bit is written in each bit of comment counting corresponding field in wbTweetInfo data structure.
Such as: each bit commenting on counting in interim storage is:
00000000111111111111111111111111
Then described 3rd value is: 11111111111111111111111100000000, and after visible 3rd value produces, vacant is low 8 bits.
In interim storage, each bit of micro-group counting is:
1111111111110111
Then described 4th value is: 0000000011110111
After then the 3rd value and the 4th value carry out position or process, the value of each bit obtained is:
11111111111111111111111111110111, visible 3rd value and the 4th value carry out position or after value be the value of the value of each bit of comment counting and low 8 bits of micro-group counting.
If the bit of not applying of a destination item is only distributed to described project to be increased by 311, obtain described item destination data to be increased, do not apply in bit described in directly described item destination data to be increased being written to.The example of the situation that its specific implementation can be forwarded see a certain bar microblogging in above-mentioned microblogging field, repeats no more herein.
Further, as shown in Figure 6, after step 310 and step 311, also comprise:
312, the project information of each field in data structure is obtained.
Wherein, described project information comprises the position that project corresponding to described each field and described item destination data store in described field.
313, judge whether described field comprises multiple destination datas according to described project information.
If determine described field to comprise multiple destination datas, perform step 314.If determine described field only comprise an item destination data, perform step 315.
314, the position stored in described field according to described item destination data is read every destination data.
The data that such as data structure comprises field 1, field 2, field 1 only store project 1, field 2 store the data of project 2 and project 3, wherein, the data of project 2 take high 24 bits of field 2, the data of project 3 take low 8 bits of field 2, then when reading, obtain the data of project 2 from high 24 bits of field 2, obtain the data of project 3 from low 8 bits.
315, described item destination data is read.
In addition, as shown in Figure 7, after step 312, can also comprise:
316, judge whether an item destination data is stored in different field according to described project information.
If determining an item destination data is stored in different field, perform step 317; If determine an item destination data to store in the same field, perform step 319.
317, from described different field, read each several part data of described project.
318, described each several part data are merged, obtain described item destination data.
319, from the field at this destination data place, directly this destination data is read.
Such as, situation about being forwarded for a certain bar microblogging in microblogging field, its data structure is
Struct wbTweetInfo{
Uint32_t dwUid; / * user ID */
Uint64_t ddwTweetId; / * turn comment ID*/
Uint32_t dwCount; / * relay counting */
Uint32_t dwCommentCount; / * comment counting */
};
Wherein, what high 24 bits of relaying the field of counting correspondence stored is the data relaying counting, and what low 8 bits stored is the most-significant byte data that micro-group counts.What high 24 bits commenting on the field of counting correspondence stored is the data that comment counts, and what low 8 bits stored is the least-significant byte data that micro-group counts.
Can in the following way when reading:
TweetInfoNew.dwNewUid=wbTweetInfo.dwUid; This formula represents and directly the value of each bit of user ID corresponding field in wbTweetInfo data structure is read, the value of each bit of user ID in temporarily being stored.
TweetInfoNew.ddwNewTweetId=wbTweetInfo.ddwTweetId; This formula represents directly comments the value of each bit of ID corresponding field to read the transfer of wbTweetInfo data structure, the value of each bit of user ID in temporarily being stored.
TweetInfoNew.dwNewCount=wbTweetInfo.dwCount>>8; This formula represents that each bit relay in wbTweetInfo data structure being counted corresponding field is shifted to low 8 bits, and the value obtained is relay the value of counting.
Such as, relay counting corresponding field in data structure is:
11111111111111101110111111111111
The value of the relay counting obtained in the manner described above is:
00000000111111111111111011101111
TweetInfoNew.dwNewCommentCount=wbTweetInfo.dwCommentCount >>8; This formula represents that each bit comment in wbTweetInfo data structure being counted corresponding field is shifted to low 8 bits, and the value obtained is the value of comment counting.
Such as, comment counting corresponding field in data structure is:
11111111111111111111111111110111
The value of the comment counting obtained in the manner described above is:
00000000111111111111111111111111
TweetInfoNew.wWQCount=(wbTweetInfo.dwCount<<24>>16)|
(wbTweetInfo.dwCommentCount<<24>>24)
Data due to micro-group counting are stored in relays field corresponding to the counting field corresponding with comment counting, so need two parts data to read respectively, and merge, concrete mode is that each bit of the field relaying counting is moved 24 bits to a high position, move 16 bits to low level again, obtain the 5th value.
Such as, relay counting corresponding field in data structure is:
11111111111111101110111111111111
Then described 5th value is:
00000000000000001111111100000000
Each bit of the field of comment counting is moved 24 bits to a high position, then moves 24 bits to low level, obtain the 6th value.
Such as, comment counting corresponding field in data structure is:
11111111111111111111111111110111
Then described 6th value is:
00000000000000000000000011110111
5th value and the 6th value are carried out position or process, and can obtain micro-group counting item destination data is:
00000000000000001111111111110111
In addition, if the mode storing projects data in data structure is different, then carry out being shifted and position or mode also different, repeat no more other modes herein.
The method of the data processing that another embodiment of the present invention provides, determine in data structure the destination item deposited in the project, and determine described destination item application bit, this is not applied bit and does not store described target item destination data, thus described bit of not applying is distributed to project to be increased, and do not apply in bit described in described item destination data to be increased is written to, thus store field without the need to increasing, can store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
Corresponding to the embodiment of the method for above-mentioned data processing, as shown in Figure 8, the data storage device that the embodiment of the present invention provides, comprising:
Acquiring unit 41, obtains each length maximal value that there are the bit shared by project and the data under respectively having there is project that each field in data structure is corresponding.
Determining unit 42, for the described length maximal value obtained according to described acquiring unit 41, determines the bit of application that respectively there is project.
Described determining unit 42, also for according to described shared bit and describedly apply bit, determines the destination item deposited in the project.
Described determining unit 42, also for the bit shared by described destination item and applied bit and determine not apply bit.
Allocation units 43, distribute to project to be increased for not applying bit described in described determining unit 42 being determined.
Writing unit 44, for obtaining the item destination data described to be increased that described allocation units 43 distribute, and does not apply in bit described in described item destination data to be increased being written to.
Further, as shown in Figure 9, described determining unit 42, comprising:
Determination module 421, for according to the bit existed shared by project and describedly apply bit, determines described shared bit and the described difference having applied bit.
Judge module 422, for judging whether described difference is greater than a predetermined threshold value.
Described determination module 421, also for when described difference is greater than described predetermined threshold value, having there is project described in determining is described destination item.
Concrete, described determining unit 42, for:
In described shared bit, apply bit except bit as not applying bit using except described.
As shown in Figure 9, described data storage device, also comprises:
Shift unit 45, carrying out shifting processing for having applied bit described in described determining unit 42 being determined, applying bit described in redefining in described shared bit.
Concrete, described destination item comprises first object project and the second destination item.As shown in Figure 9:
Described allocation units 43, also for the bit of not applying of described first object project is distributed to described second destination item, described first object project do not apply bit for partly or entirely applying bit as described second destination item.
Described determining unit 42, also for:
In the bit shared by described second destination item, redefine the bit of application of the second destination item.
Bit except bit has been applied as not applying bit using except described.
Concrete, as shown in Figure 9, described allocation units 43, for:
The bit of not applying of the one or more destination items described determining unit determined distributes to described project to be increased.
Concrete, as shown in Figure 9, said write unit 44, comprising:
Split module 441, for by multiple destination item do not apply bit distribute to described project to be increased time, be multiple partial data by the Data Division of described project to be increased.
Writing module 442, is written to not applying in bit of each destination item respectively by described multiple partial data.
What deserves to be explained is, the specific implementation of the data storage device that the embodiment of the present invention provides see the specific embodiment of Fig. 1 and Fig. 3, can repeat no more herein.
The data storage device that the embodiment of the present invention provides, determine in data structure the destination item deposited in the project, and determine described destination item application bit, this is not applied bit and does not store described target item destination data, thus described bit of not applying is distributed to project to be increased, and do not apply in bit described in described item destination data to be increased is written to, thus store field without the need to increasing, can store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
Corresponding to the embodiment of the method for above-mentioned data processing, as shown in Figure 10, the data fetch device that the embodiment of the present invention provides, comprising:
Acquiring unit 51, for obtaining the project information of each field in data structure.
Wherein, described project information comprises the position that project corresponding to described each field and described item destination data store in described field.
For the described project information obtained according to described acquiring unit 51, judging unit 52, judges whether described field comprises multiple destination datas.
Reading unit 53, comprising multiple destination datas for determining described field at described judging unit 52, reading according to the position that described item destination data stores in described field to every destination data.
Further, described judging unit 52, also for:
Judge whether an item destination data is stored in different field according to the described project information that described acquiring unit obtains.
Described reading unit 53, being also stored in different field for determining an item destination data at described judging unit 52, reading each several part data of described project from described different field.
Further, as shown in figure 11, described data fetch device, also comprises:
Merge cells 54, for described each several part data being merged, obtains described item destination data.
What deserves to be explained is, the specific implementation of the data fetch device that the embodiment of the present invention provides see the specific embodiment of Fig. 1 and Fig. 3, can repeat no more herein.
The data fetch device that the embodiment of the present invention provides, multiple the destination datas due to the field store in data structure, thus store field without the need to increasing, can to store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
As Figure 12, the system of the data processing that the embodiment of the present invention provides, comprises above-mentioned data storage device 61 and above-mentioned data fetch device 62.
Its specific implementation see the specific implementation of above-mentioned data storage device and above-mentioned data fetch device, can repeat no more herein.
The system of the data processing that the embodiment of the present invention provides, determine in data structure the destination item deposited in the project, and determine described destination item application bit, this is not applied bit and does not store described target item destination data, thus described bit of not applying is distributed to project to be increased, and do not apply in bit described in described item destination data to be increased is written to, thus store field without the need to increasing, can store the data of project to be increased.And in the prior art, need to store the follow-up data added by increasing storage field, cause the waste of memory source.Therefore, the present invention can save memory source, improves the utilization factor of memory source.
Through the above description of the embodiments, those skilled in the art can be well understood to the mode that the present invention can add required common hardware by software and realize, and can certainly pass through hardware, but in a lot of situation, the former is better embodiment.Based on such understanding, technical scheme of the present invention can embody with the form of software product the part that prior art contributes in essence in other words, this computer software product is stored in the storage medium that can read, as the floppy disk of computing machine, hard disk or CD etc., comprise some instructions and perform method described in each embodiment of the present invention in order to make a computer equipment (can be personal computer, server, or the network equipment etc.).
The above; be only the specific embodiment of the present invention, but protection scope of the present invention is not limited thereto, is anyly familiar with those skilled in the art in the technical scope that the present invention discloses; change can be expected easily or replace, all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should described be as the criterion with the protection domain of claim.
Claims (21)
1. a method for data processing, is characterized in that, comprising:
Obtain each length maximal value that there are the bit shared by project and the data under respectively having there is project that each field in data structure is corresponding;
According to described length maximal value, determine the bit of application that respectively there is project;
According to described shared bit and describedly apply bit, determine the destination item deposited in the project;
Bit shared by described destination item and applied bit and determine not apply bit;
Described bit of not applying is distributed to project to be increased;
Obtain described item destination data to be increased, and do not apply in bit described in described item destination data to be increased is written to.
2. the method for data processing according to claim 1, is characterized in that, described according to described shared bit and describedly apply bit, determines the destination item deposited in the project, comprising:
According to the bit existed shared by project and describedly apply bit, determine described shared bit and the described difference having applied bit;
Judge whether described difference is greater than a predetermined threshold value;
If described difference is greater than described predetermined threshold value, then having there is project described in determining is described destination item.
3. the method for data processing according to claim 1, is characterized in that, described bit shared by described destination item and applied bit and determine not apply bit, comprising:
In described shared bit, apply bit except bit as not applying bit using except described.
4. the method for data processing according to claim 3, is characterized in that, has applied bit except bit as before not applying bit, also comprise described using except described:
Described bit of having applied is carried out shifting processing in described shared bit, described in redefining, applies bit.
5. the method for data processing according to claim 3, is characterized in that, described destination item comprises first object project and the second destination item;
Apply bit except bit as before not applying bit described using except described, also comprised:
The bit of not applying of described first object project is distributed to described second destination item, described first object project do not apply bit for partly or entirely applying bit as described second destination item.
6. the method for data processing according to claim 5, is characterized in that, described in described shared bit, having applied bit except bit as not applying bit, comprising except described:
In the bit shared by described second destination item, redefine the bit of application of the second destination item;
Bit except bit has been applied as not applying bit using except described.
7. the method for data processing according to claim 1, is characterized in that, described described bit of not applying is distributed to project to be increased, comprising:
The bit of not applying of one or more destination item is distributed to described project to be increased.
8. the method for data processing according to claim 7, is characterized in that, the described item destination data to be increased of described acquisition, and does not apply in bit described in described item destination data to be increased being written to, and comprising:
If the bit of not applying of multiple destination item is distributed to described project to be increased, be multiple partial data by the Data Division of described project to be increased;
Described multiple partial data is written to not applying in bit of each destination item respectively.
9. a method for data processing, is characterized in that, comprising:
Obtain the project information of each field in data structure; Described project information comprises the position that project corresponding to described each field and described item destination data store in described field;
Judge whether described field comprises multiple destination datas according to described project information;
If determine described field to comprise multiple destination datas, according to the position that described item destination data stores in described field, every destination data is read.
10. the method for data processing according to claim 9, is characterized in that, in described acquisition data structure each field project information after, also comprise:
Judge whether an item destination data is stored in different field according to described project information;
If determining an item destination data is stored in different field, from described different field, read each several part data of described project;
Described each several part data are merged, obtains described item destination data.
11. 1 kinds of data storage devices, is characterized in that, comprising:
Each length maximal value that there are the bit shared by project and the data under respectively having there is project that in acquiring unit, acquisition data structure, each field is corresponding;
Determining unit, for the described length maximal value obtained according to described acquiring unit, determines the bit of application that respectively there is project;
Described determining unit, also for according to described shared bit and describedly apply bit, determines the destination item deposited in the project;
Described determining unit, also for the bit shared by described destination item and applied bit and determine not apply bit;
Allocation units, distribute to project to be increased for not applying bit described in described determining unit being determined;
Writing unit, for obtaining the item destination data described to be increased that described allocation units distribute, and does not apply in bit described in described item destination data to be increased being written to.
12. data storage devices according to claim 11, is characterized in that, described determining unit, comprising:
Determination module, for according to the bit existed shared by project and describedly apply bit, determines described shared bit and the described difference having applied bit;
Judge module, for judging whether described difference is greater than a predetermined threshold value;
Described determination module, also for when described difference is greater than described predetermined threshold value, having there is project described in determining is described destination item.
13. data storage devices according to claim 11, is characterized in that, described determining unit, for:
In described shared bit, apply bit except bit as not applying bit using except described.
14. data storage devices according to claim 13, is characterized in that, also comprise:
Shift unit, carrying out shifting processing for having applied bit described in described determining unit being determined, applying bit described in redefining in described shared bit.
15. data storage devices according to claim 13, is characterized in that, described destination item comprises first object project and the second destination item;
Described allocation units, also for the bit of not applying of described first object project is distributed to described second destination item, described first object project do not apply bit for partly or entirely applying bit as described second destination item.
16. data storage devices according to claim 15, is characterized in that, described determining unit, also for:
In the bit shared by described second destination item, redefine the bit of application of the second destination item;
Bit except bit has been applied as not applying bit using except described.
17. data storage devices according to claim 11, is characterized in that, described allocation units, for:
The bit of not applying of the one or more destination items described determining unit determined distributes to described project to be increased.
18. data storage devices according to claim 17, is characterized in that, said write unit, comprising:
Split module, for by multiple destination item do not apply bit distribute to described project to be increased time, be multiple partial data by the Data Division of described project to be increased;
Writing module, is written to not applying in bit of each destination item respectively by described multiple partial data.
19. 1 kinds of data fetch devices, is characterized in that, comprising:
Acquiring unit, for obtaining the project information of each field in data structure; Described project information comprises the position that project corresponding to described each field and described item destination data store in described field;
Judging unit, the described project information for obtaining according to described acquiring unit judges whether described field comprises multiple destination datas;
Reading unit, comprising multiple destination datas for determining described field at described judging unit, reading according to the position that described item destination data stores in described field to every destination data.
20. data fetch devices according to claim 9, is characterized in that, described judging unit, also for:
Judge whether an item destination data is stored in different field according to the described project information that described acquiring unit obtains;
Described reading unit, is also stored in different field for determining an item destination data at described judging unit, reads each several part data of described project from described different field;
Described data fetch device, also comprises:
Merge cells, for described each several part data being merged, obtains described item destination data.
The system of 21. 1 kinds of data processings, is characterized in that, comprises the data storage device described in claim 11-18 and the data fetch device described in claim 19-20.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310476079.0A CN104572655B (en) | 2013-10-12 | 2013-10-12 | The method, apparatus and system of data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310476079.0A CN104572655B (en) | 2013-10-12 | 2013-10-12 | The method, apparatus and system of data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104572655A true CN104572655A (en) | 2015-04-29 |
| CN104572655B CN104572655B (en) | 2019-04-12 |
Family
ID=53088753
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310476079.0A Active CN104572655B (en) | 2013-10-12 | 2013-10-12 | The method, apparatus and system of data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104572655B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105373663A (en) * | 2015-11-17 | 2016-03-02 | 无锡江南计算技术研究所 | Method for realizing asynchronous buffer with entry occupation indication function |
| CN105450488A (en) * | 2015-11-11 | 2016-03-30 | 深圳市冠明能源科技有限公司 | Data processing method and related device |
| CN110096693A (en) * | 2018-01-29 | 2019-08-06 | 北京搜狗科技发展有限公司 | A kind of data processing method, device and the device for data processing |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0925687B1 (en) * | 1996-09-13 | 2003-04-02 | Silicon Graphics, Inc. | Compression and decompression scheme performed on shared workstation memory by media coprocessor |
| CN102457283A (en) * | 2010-10-28 | 2012-05-16 | 阿里巴巴集团控股有限公司 | Data compression and decompression method and equipment |
-
2013
- 2013-10-12 CN CN201310476079.0A patent/CN104572655B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0925687B1 (en) * | 1996-09-13 | 2003-04-02 | Silicon Graphics, Inc. | Compression and decompression scheme performed on shared workstation memory by media coprocessor |
| CN102457283A (en) * | 2010-10-28 | 2012-05-16 | 阿里巴巴集团控股有限公司 | Data compression and decompression method and equipment |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105450488A (en) * | 2015-11-11 | 2016-03-30 | 深圳市冠明能源科技有限公司 | Data processing method and related device |
| CN105450488B (en) * | 2015-11-11 | 2019-10-22 | 深圳市冠明能源科技有限公司 | A kind of data processing method and relevant apparatus |
| CN105373663A (en) * | 2015-11-17 | 2016-03-02 | 无锡江南计算技术研究所 | Method for realizing asynchronous buffer with entry occupation indication function |
| CN105373663B (en) * | 2015-11-17 | 2018-06-26 | 无锡江南计算技术研究所 | A kind of asynchronous buffer implementation method that instruction is occupied with entry |
| CN110096693A (en) * | 2018-01-29 | 2019-08-06 | 北京搜狗科技发展有限公司 | A kind of data processing method, device and the device for data processing |
| CN110096693B (en) * | 2018-01-29 | 2024-05-28 | 北京搜狗科技发展有限公司 | Data processing method and device for data processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104572655B (en) | 2019-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103123604B (en) | The method of the internal memory use of tracking data disposal system | |
| CN106909372B (en) | Method and system for calculating purchase path of mobile terminal user | |
| CN104965757A (en) | Virtual machine live migration method, virtual machine migration management apparatus, and virtual machine live migration system | |
| CN104123171B (en) | Virtual machine migrating method and system based on NUMA architecture | |
| CN105224444A (en) | Daily record generation method and device | |
| CN106453444A (en) | Cache data sharing method and equipment | |
| CN104899218A (en) | Data reading and writing method and data reading and writing apparatus | |
| CN104781794A (en) | In-place change between transient and persistent state for data structures in non-volatile memory | |
| CN104657435A (en) | Storage management method for application data and network management system | |
| CN105094780A (en) | Information correlation method and device | |
| CN103136215A (en) | Data read-write method and device of storage system | |
| CN101546287A (en) | Code modification method and code modification equipment | |
| CN104281535A (en) | Method and device for processing mapping tables in memory | |
| GB2593408A (en) | Increasing data performance by transferring data between storage tiers using workload characteristics | |
| CN104572655A (en) | Data processing method, device and system | |
| CN104424240A (en) | Multi-table correlation method and system, main service node and computing node | |
| CN103336670A (en) | Method and device for automatic distribution of data blocks based on data temperature | |
| CN107544811A (en) | Ios platform hides method, storage medium, electronic equipment and the system of dylib files | |
| CN103970512A (en) | Multi-core processor and parallel replay method thereof | |
| CN102426601B (en) | Data deleting method and device | |
| CN106708445A (en) | Link selection method and device | |
| CN104408056A (en) | Data processing method and device | |
| CN105095105A (en) | Cache partitioning method and device | |
| CN107817972A (en) | Cache code process method, apparatus, storage medium and electronic equipment | |
| CN105488047A (en) | Metadata read-write method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |