CN105630972A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN105630972A CN105630972A CN201510990963.5A CN201510990963A CN105630972A CN 105630972 A CN105630972 A CN 105630972A CN 201510990963 A CN201510990963 A CN 201510990963A CN 105630972 A CN105630972 A CN 105630972A
- Authority
- CN
- China
- Prior art keywords
- user
- data
- logon
- day
- bitmap
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
Abstract
The invention discloses a data processing method and device, and relates to the technical field of databases. The data processing method comprises the steps that unique integral data ID is distributed to each user; according to the integral data ID of each user, user login data of a bitmap-based data structure is generated; the user login data of the bitmap-based data structure is stored in a database. According to the data processing method, the bitmap-based data structure is provided for recording the user login phenomenon. The inquiry statistical processing efficiency can be effectively improved on the basis of user login records stored in bitmap data.
Description
Technical field
The present invention relates to database technical field, in particular to a kind of data processing method and device.
Background technology
User logs in the link that situation analysis is usually most basic during each website data is analyzed. Especially, in online game, players log analysis generally includes number of persons logging every day, certain a period of time number of persons logging, player's retention ratio and the retention situation etc. in a certain kind player's a period of time. For the statistics of these indexs, realize typically by handles user login daily record at present. Traditional scheme is after collection user logs in daily record, each user is logged in daily record and stores in data base, when inquiry for analyzing each the statistical demand of personnel, one or more of standardized query statement will be converted into, then use database query interface to add up the index being intended to inquiry.
And popularizing along with online game, player gets more and more, when userbase reach ten million rank even higher time, traditional by storing user, to log in the data base query method of daily record very slow in statistical disposition data hourly velocity, cannot meet analysis personnel and immediately obtain the demand of statistical result.
Summary of the invention
In view of this, the present invention provides a kind of data processing method and device, it is possible to promote the treatment effeciency of data.
Other characteristics of the present invention and advantage will be apparent from by detailed description below, or partially by the practice of the present invention and acquistion.
According to an aspect of the present invention, it is provided that a kind of data processing method, including: distribute a unique integer data ID for each user; Described integer data ID according to each user, generates the user's logon data based on bitmap data structure; And described user's logon data based on bitmap data structure is stored in data base.
According to an embodiment of the present invention, described integer data ID from 1 or 0 open numbering and is incremented by one by one.
According to an embodiment of the present invention, the described integer data ID according to each user, generate the user's logon data based on bitmap data structure and include: the described integer data ID according to each user, it is determined that the sequence of each user; And the described sequence according to each user, generate described user's logon data based on bitmap data structure.
According to an embodiment of the present invention, described sequence from 1 or 0 open numbering and is incremented by one by one.
According to an embodiment of the present invention, described data base is HBase data base.
According to an embodiment of the present invention, described data processing method also includes: according to described user's logon data based on bitmap data structure, carries out user and logs in situation analysis.
According to an embodiment of the present invention, described user's logon data based on bitmap data structure includes: every day user's logon data, every day Adds User at least one in logon data, every server user's logon data and every tagging user logon data.
According to an embodiment of the present invention, according to described user's logon data based on bitmap data structure, carry out user and log in situation analysis and include: inquiry obtain user's logon data all of described every day in a period of time from described data base; The user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the first result; And to add up the number of 1 in described first result be the number of persons logging of user in described a period of time.
According to an embodiment of the present invention, according to described user's logon data based on bitmap data structure, carry out user and log in situation analysis and include: inquiry obtain described every server user's logon data of Add User described every day of a certain day logon data and a server from described data base; Described every server user's logon data of the logon data that Added User the described every day of this day and described server carries out the arithmetic operation of logical AND, obtains the second result; And to add up the number of 1 in described second result be the number that Adds User of this day described server.
According to an embodiment of the present invention, according to described user's logon data based on bitmap data structure, carry out user and log in situation analysis and also include: inquiry user's logon data all of described every day in a period of time after obtaining this day from described data base; The user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the 3rd result; Described second result and described 3rd result are carried out the arithmetic operation of logical AND, obtains the 4th result; And add up the retention number of users in described a period of time later of Adding User that the number of 1 in described 4th result is this day described server.
According to a further aspect in the invention, it is provided that a kind of data processing equipment, including: ID distribute module, is used for distributing a unique integer data ID for each user; Logon data generation module, for the described integer data ID according to each user, generates the user's logon data based on bitmap data structure; And logon data memory module, for described user's logon data based on bitmap data structure is stored in data base.
According to an embodiment of the present invention, described integer data ID from 1 or 0 open numbering and is incremented by one by one.
According to an embodiment of the present invention, described logon data generation module includes: submodule is determined in sequence, for the described integer data ID according to each user, it is determined that the sequence of each user; And data genaration submodule, for the described sequence according to each user, generate described user's logon data based on bitmap data structure.
According to an embodiment of the present invention, described sequence from 1 or 0 open numbering and is incremented by one by one.
According to an embodiment of the present invention, described data base is HBase data base.
According to an embodiment of the present invention, described data processing equipment also includes: user logs in analysis module, for according to described user's logon data based on bitmap data structure, carrying out user and log in situation analysis.
According to an embodiment of the present invention, described user's logon data based on bitmap data structure includes: every day user's logon data, every day Adds User at least one in logon data, every server user's logon data and every tagging user logon data.
According to an embodiment of the present invention, described user logs in analysis module and includes: the first inquiry obtains submodule, for inquiry from described data base and obtain user's logon data all of described every day in a period of time; First logical operation submodule, for user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the first result; And the first statistics submodule, for add up the number of 1 in described first result be the number of persons logging of user in described a period of time.
According to an embodiment of the present invention, described customer analysis module includes: the second inquiry obtains submodule, described every server user's logon data of Add User the described every day for inquiring about and obtain a certain day from described data base logon data and a server; Second logical operation submodule, carries out the arithmetic operation of logical AND, obtains the second result for described every server user's logon data of Added User described every day of this day logon data and described server; And the second statistics submodule, for add up the number of 1 in described second result be the number that Adds User of this day described server.
According to an embodiment of the present invention, described second inquiry obtains after submodule is additionally operable to from described data base inquiry and obtains this day user's logon data all of described every day in a period of time; Described customer analysis module also includes: the 3rd logical operation submodule, for user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the 3rd result; And described second result and described 3rd result are carried out the arithmetic operation of logical AND, obtain the 4th result; 3rd statistics submodule, for adding up the retention number of users in described a period of time later of Adding User that the number of 1 in described 4th result is this day described server.
Data processing method according to the present invention, it is provided that a kind of data structure based on bitmap, for recording the login situation of user. By being stored in the user's login record in data bitmap, it is possible to effectively promote the treatment effeciency of query statistic.
It addition, according to some embodiments, as required the user's logon data based on bitmap data structure is carried out logical operations, obtains statistical result by logical operations; Owing to the speed of logical operations is very fast, therefore the data processing method of the present invention can accelerate the processing speed of data, the instant statistical result returning needs.
It should be appreciated that above general description and details hereinafter describe and be merely illustrative of, the present invention can not be limited.
Accompanying drawing explanation
Its example embodiment being described in detail by referring to accompanying drawing, above-mentioned and other target of the present invention, feature and advantage will become apparent from.
Fig. 1 is the flow chart of a kind of data processing method according to an illustrative embodiments.
Fig. 2 is the schematic diagram of a kind of bitmap data structure according to an example.
Fig. 3 is the flow chart of the another kind of data processing method according to an illustrative embodiments.
Fig. 4 is the flow chart of another data processing method according to an illustrative embodiments.
Fig. 5 is the block diagram of a kind of data processing equipment according to an illustrative embodiments.
Fig. 6 is the block diagram of the another kind of data processing equipment according to an illustrative embodiments.
Fig. 7 is the block diagram of another data processing equipment according to an illustrative embodiments.
Detailed description of the invention
It is described more fully with example embodiment referring now to accompanying drawing. But, example embodiment can be implemented in a variety of forms, and is not understood as limited to example set forth herein; On the contrary, it is provided that these embodiments make the present invention will more fully and completely, and the design of example embodiment is conveyed to those skilled in the art all sidedly. Accompanying drawing is only the schematic illustrations of the present invention, is not necessarily drawn to scale. Accompanying drawing labelling identical in figure represents same or similar part, thus will omit repetition thereof.
Additionally, described feature, structure or characteristic can be combined in one or more embodiment in any suitable manner. In the following description, it is provided that many details are thus providing fully understanding embodiments of the present invention. It will be appreciated, however, by one skilled in the art that can put into practice technical scheme and omit in described specific detail is one or more, or other method, constituent element, device, step etc. can be adopted. In other cases, be not shown in detail or describe known features, method, device, realization or arithmetic operation to avoid, a presumptuous guest usurps the role of the host and each aspect of the present invention is thickened.
Fig. 1 is the flow chart of a kind of data processing method according to an illustrative embodiments. As it is shown in figure 1, data processing method 10 includes:
In step s 110, a unique integer data ID (Identification, identity) is distributed for each user.
The integer data ID that each user (such as can represent with the ID of its registration) is unique, and each user maps one by one with its integer data ID.
Integer data ID can from 1 open numbering, it is also possible to from 0 open numbering. Afterwards, integer data ID is incremented by, for instance the integer data ID of first user is 1, and the integer data ID of second user is 2, and the integer data of third party is 3, by that analogy. But the invention is not restricted to this, integer data ID can be incremented by with arbitrary form. As long as ensureing the integer data ID of each user uniquely.
In the step s 120, the integer data ID according to user, generate the user's logon data based on bitmap (bitmap) data structure.
Fig. 2 is the schematic diagram of a kind of bitmap data structure according to an example. As in figure 2 it is shown, bitmap data structure is a kind of data structure using bit (bit) to store information, each bit is " 0 " or " 1 ", represents the information of a dimension.
In the present invention, each bit corresponds to an integer data ID, thus representing the login situation of this integer data ID owning user.
Such as, with from 1 open numbering, the integer data ID being incremented by numbering one by one is example, as in this sky on June 1st, 2015, having integer data ID is that 3,5,8 these three users log in, then reflect that this day user logs in the data bitmap of situation and be: 00101001. Namely in data bitmap, the number of bits of data bitmap starts counting up from 1, the bit of corresponding 3rd, 5,8 is 1, and other bits are 0, thus representing that the user that integer data ID is 3,5,8 logged in this day.
And if integer data ID is from 0 open numbering, the still integer data ID to be incremented by numbering one by one, equally in this sky on June 1st, 2015, having integer data ID is that 3,5,8 these three users log in. And because integer data ID is from 0 open numbering, the number of bits of data bitmap also starts counting up from 0 accordingly, therefore reflects that this day user logs in the data bitmap of situation and can be: 000101001. Namely the bit on the 0th, 1,2,4,6,7 is 0, and the bit on the 3rd, 5,8 is 1, thus representing that the user that integer data ID is 3,5,8 logged in this day.
In step s 130, the user's logon data based on bitmap data structure is stored in data base.
In certain embodiments, this data base can be such as in HBase data base. HBase data base is a kind of non-relational database (NoSQL) based on Hadoop counting, it is possible to obtain data according to the KEY in data base from mass data rapidly. The operation of HBase data base is that this area counting personnel are known, and this present invention is repeated no more.
The data processing method 10 that present embodiment provides, it is provided that a kind of data structure based on bitmap, for recording the login situation of user. By being stored in the user's login record in data bitmap, it is possible to effectively promote the treatment effeciency of query statistic.
It will be clearly understood that present disclosure describe how to be formed and use particular example, but principles of the invention is not limited to any details of these examples. On the contrary, based on the instruction of present disclosure, these principles can be applied to numerous other embodiments.
Fig. 3 is the flow chart of the another kind of data processing method according to an illustrative embodiments. As it is shown on figure 3, data processing method 20 includes:
In step S210, distribute a unique integer data ID for each user.
The integer data ID that each user (such as can represent with the ID of its registration) is unique, and each user maps one by one with its integer data ID.
Integer data ID can from 1 open numbering, it is also possible to from 0 open numbering. Afterwards, integer data ID such as can be incremented by with a default exponent number.
Such as, for from 1 open numbering, this default exponent number is 1, then the integer data ID of first user is 1, and the integer data ID of second user is 2, and the integer data of third party is 3, by that analogy.
Or, this default exponent number can also be 3, still for from 1 open numbering, then the integer data ID of first user is 1, the integer data ID of second user is 1+ (2-1) * 3=4, the integer data ID of third party is 1+ (3-1) * 3=7 ..., the integer data ID of nth user is 1+ (n-1) * 3.
It should be noted that the above-mentioned merely illustrative explanation of incremental manner presetting exponent number with, and the unrestricted present invention. Those skilled in the art are it is to be appreciated that can be incremented by this integer data ID in actual applications in any way, as long as can determine that out the sequence of this user. Such as, above-mentioned from 1 open numbering and example that default exponent number is 1, start for user's sequence from 1, integer data ID be 1 user be ordered as 1, integer data ID be 2 user be ordered as 2, integer data ID be the user of 3 be ordered as 3, by that analogy. And it is above-mentioned from the example of 1 open numbering and default exponent number 3, start for user's sequence from 1, integer data ID be the user of 1 be ordered as 1, integer data is ID be 1+ (2-1) * 3 user be ordered as 2, integer data is ID be 1+ (3-1) * 3 user be ordered as 3, ..., integer data ID be the user of 1+ (n-1) * 3 be ordered as n.
Integer data ID in step S220, according to user, it is determined that the sequence of user.
Such as, above-mentioned from 1 open numbering and example that default exponent number is 1, if started for user's sequence from 1, then integer data ID be 1 user be ordered as 1, integer data ID be 2 user be ordered as 2, integer data ID be the user of 3 be ordered as 3, by that analogy. And if for user sequence start from 0, then integer data ID be 1 user be ordered as 0, integer data ID be 2 user be ordered as 1, integer data ID be the user of 3 be ordered as 2, by that analogy.
Or it is above-mentioned with in the example from 1 open numbering and default exponent number 3, if started for user's sequence from 1, then integer data ID be the user of 1 be ordered as 1, integer data is ID be 1+ (2-1) * 3 user be ordered as 2, integer data is ID be 1+ (3-1) * 3 user be ordered as 3, ..., integer data ID be the user of 1+ (n-1) * 3 be ordered as n. And if for user sequence start from 0, then integer data ID be the user of 1 be ordered as 0, integer data is ID be the user of 1+1*3 be ordered as 1, integer data is ID be the user of 1+2*3 be ordered as 2, ..., integer data ID be the user of 1+n*3 be ordered as n.
In step S230, the sequence according to user, generate the user's logon data based on bitmap data structure.
Such as, above-mentioned from 1 open numbering and example that default exponent number is 1, if started for user's sequence from 1, as in this sky on June 1st, 2015, having integer data ID is that 3,5,8 these three users log in, the sequence of the these three user determined respectively 3,5,8, then reflect that this day user logs in the data bitmap of situation and be: 00101001. Namely in data bitmap, the number of bits of data bitmap starts counting up from 1, the bit of the 3rd, 5,8 is 1, and other bits are 0, represents that the user determined according to integer data ID3,5,8 is ordered as the user of 3,5,8 login in this day.
And if for user sequence from 0 start example, as in this sky on June 1st, 2015, having integer data ID is that 3,5,8 these three users log in, it is determined that the sequence of the these three user gone out respectively 2,4,7, then reflect that this day user logs in the data bitmap of situation and be: 00101001. Namely in data bitmap, the number of bits correspondence of data bitmap starts counting up from 0, bit on 0th, 1,5,6 is 0, and the bit on the 2nd, 4,7 is 1, thus representing that the user determined according to integer data ID3,5,8 is ordered as the user of 2,4,7 login in this day.
Or, above-mentioned from 1 open numbering and example that default exponent number is 3, if started for user's sequence from 1, as in this sky on June 1st, 2015, having integer data ID is that 1,7,13 these three users log in, because of 1=1+ (1-1) * 3,7=1+ (3-1) * 3,13=1+ (5-1) * 3, it is ordered as 1,3,5 according to the user that integer data ID1,7,13 are determined. Therefore reflect that this day user logs in the data bitmap of situation and be: 10101. Namely in data bitmap, the number of bits of data bitmap starts counting up from 1, the bit of the 1st, 3,5 is 1, and other bits are 0, represents that the user determined according to integer data ID1,7,13 is ordered as the user of 1,3,5 login in this day.
And if for user sequence start from 0, as in this sky on June 1st, 2015, having integer data ID equally is that 1,7,13 these three users log in, and because of 1=1+0*3,7=1+2*3,13=1+4*3, is ordered as 0,2,4 according to the user that integer data ID1,7,13 are determined. And because user sorts from 0 open numbering, the number of bits of data bitmap also starts counting up from 0 accordingly, therefore reflects that this day user logs in the data bitmap of situation and be still: 10101.
In step S240, the user's logon data based on bitmap data structure is stored in data base.
Similarly, this data base can be such as HBase data base.
The data processing method 20 that present embodiment provides, it is possible to determine the sequence of user the sequence according to user according to integer data ID, the login situation of user be recorded in data bitmap, such that it is able to effectively promote the treatment effeciency of query statistic.
Another embodiment according to the present invention, the user's logon data based on bitmap data structure generated in the step S120 of data processing method 10 or the step S230 of data processing method 20 may include that user's logon data every day, every day Add User at least one in logon data, every server user's logon data and every tagging user logon data.
Wherein Adding User user's every day logon data, every day logon data, every server user's logon data and every tagging user logon data can with Key-Value form record. As table 1 illustrates the tables of data of each user's logon data with Key-Value form record. Wherein " DayLogin_ date " represent user's logon data every day, " DayNew_ date " represents the logon data that Adds User every day, and " Server_ server ID " represents every server user's logon data, and " TAG " represents every tagging user logon data. Wherein it is likely to the user's logon data under the special tag used in some special analysis task.
Table 1
It is exemplified below and how to generate above-mentioned Key-Value value. It should be noted that in following example, all with integer ID from 1 open numbering, be incremented by one by one and be numbered example explanation.
Such as, on October 1st, 2015 is opened clothes in the game of certain money, opens two servers of 2 A and B, opens the login taking latter 4 days and newly-increased situation is as follows:
20151001: server A increases two users newly, and being respectively allocated integer data ID for the two user is 1 and 2, server B increases a user newly, and distributing integer data ID for this user is 3.
20151002: in server A, integer data ID is old user's login of 1, simultaneously newly-increased two users, and being respectively allocated integer data ID for the two user is 4 and 5; Server B does not Add User, and logs in also without user.
20151003: in server A integer data ID respectively 1,2,5 old user log in, a newly-increased user simultaneously, distributing integer data ID for this user is 6; Newly-increased two users in server B, being respectively allocated integer data ID for the two user is 7 and 8.
20151004: in server A, integer data ID is old user's login of 4, in server B, integer data ID is old user's login of 3, all without Adding User on two servers.
Then each user's logon data is as shown in table 2:
Table 2
| Key | Value |
| Server_A | 11011100 |
| Server_B | 00100011 |
| DayNew_20151001 | 111 |
| DayNew_20151002 | 00011 |
| DayNew_20151003 | 000001 |
| DayNew_20151004 | 000000 |
| DayLogin_20151001 | 111 |
| DayLogin_20151002 | 10011 |
| DayLogin_20151003 | 11001111 |
| DayLogin_20151004 | 00110000 |
Fig. 4 is the flow chart of another data processing method according to an illustrative embodiments. As shown in Figure 4, data processing method 30, after the step S130 of data processing method 10 or the step S240 of data processing method 20, also includes:
In step S310, according to the user's logon data based on bitmap data structure, carry out user and log in situation analysis.
Embodiment one: according to the user's logon data based on bitmap data structure, the number of persons logging in statistics a period of time, statistical method includes:
1. inquiry obtain user's logon data all every days in this period from data base.
Such as, inquire about and obtain on October 1st, 2015 to user logon data DayLogin_20151001��DayLogin_20151004 all every days in October 4.
2. user's logon data acquired all every day is carried out logic or arithmetic operation.
Such as, DayLogin_20151001ORDayLogin_20151002ORDayLogin_20151003O RDayLogin_20151004=11111111
3. determine logic or arithmetic operation after 1 number, be the number of persons logging in this period.
Such as, sum (11111111)=8
Then the number of persons logging in October 1 to October 4 in 2015 is 8.
Embodiment two: according to the user's logon data based on bitmap data structure, adding up the number that Adds User on certain server of a certain day, statistical method includes:
1. from data base, inquire about and obtain user's logon data of Add User logon data and this server of this day.
Such as, the logon data that Adds User inquiring about and obtaining on October 2nd, 2015 is DayNew_20151002, and user's logon data of server A is Server_A.
2., by the logon data user's logon data with this server that Adds User of this day, carry out the arithmetic operation of logical AND.
Such as, DayNew_20151002ANDServer_A=00011
3. determine after the arithmetic operation of logical AND 1 number, be the number that Adds User on this server of this day.
Such as, sum (00011)=2.
Then the number that Adds User in server A on October 2 is 2.
Embodiment three: according to the user's logon data based on bitmap data structure, adding up the number retention situation in following a period of time that Adds User on certain server of a certain day, statistical method includes:
1. inquiry obtain the logon data that Adds User of this day, user's logon data of this server and user's logon data all every days in this period from data base.
Such as, the logon data that Adds User on October 2nd, 2015 is DayNew_20151002=00011, user's logon data of server A is Server_A=11011100, user's logon data DayLogin_20151003 and DayLogin_20151004 every day in following 2 days (i.e. October 3 and October 4).
2., by user's logon data of Add User logon data and this server of this day, carry out the arithmetic operation of logical AND, carry out with user's logon data all every days more afterwards logic or after result carry out the arithmetic operation of logical AND.
Such as, DayNew_20151002ANDServer_AAND (DayLogin_20151003ORDayLogin_20151004)=00011
3. determine after above-mentioned logical operation 1 number, be the retention number of users Added User within next this period on this server of this day.
Such as, sum (00011)=2.
Then the retention number of users in two days on the 4th 3��October of October that Adds User in server A on October 2 is still 2, and retention ratio is 2/2=100%.
Owing to the speed of logical operations is very fast, probably in microsecond rank, therefore use this statistical query method can accelerate the processing speed of data, the instant statistical result returning needs.
Further, when above-mentioned user's logon data is stored in HBase data base, the speed inquiring about and obtaining user's logon data from HBase data base is also very fast, general is also microsecond rank, therefore use HBase database stores user logon data, then can promote the speed of statistical query further.
The data processing method 30 that present embodiment provides, carries out logical operations as desired by the user's logon data based on bitmap data structure, can quickly obtain the index needing statistics.
It should be noted that above-mentioned accompanying drawing is only schematically illustrating of the process included by the method for exemplary embodiment of the invention, rather than restriction purpose. It can be readily appreciated that above-mentioned process shown in the drawings is not intended that or limits the time sequencing of these process. It addition, be also easy to understand, these process can such as either synchronously or asynchronously perform in multiple modules.
Following for apparatus of the present invention embodiment, it is possible to be used for performing the inventive method embodiment. For the details not disclosed in apparatus of the present invention embodiment, refer to the inventive method embodiment.
Fig. 5 is the block diagram of a kind of data processing equipment according to an illustrative embodiments. As it is shown in figure 5, data processing equipment 40 includes: ID distributes module 410, logon data generation module 420 and logon data memory module 430.
ID distributes module 410 for distributing a unique integer data ID for each user.
Logon data generation module 420, for the integer data ID according to each user, generates the user's logon data based on bitmap data structure.
Logon data memory module 430 is for being stored in the user's logon data based on bitmap data structure in data base.
The data processing equipment 40 that present embodiment provides, it is provided that a kind of data structure based on bitmap, for recording the login situation of user. By being stored in the user's login record in data bitmap, it is possible to effectively promote the treatment effeciency of query statistic.
Fig. 6 is the block diagram of the another kind of data processing equipment according to an illustrative embodiments. As shown in Figure 6, data processing equipment 50 includes: ID distributes module 510, logon data generation module 520, logon data memory module 530 and user and logs in analysis module 540.
ID distributes module 510 for distributing a unique integer data ID for each user.
Wherein integer data ID is such as from 1 or 0 open numbering, and is incremented by one by one.
Logon data generation module 520, for the integer data ID according to each user, generates the user's logon data based on bitmap data structure.
User's logon data such as may include that user's logon data every day, every day Add User at least one in logon data, every server user's logon data and every tagging user logon data.
In certain embodiments, logon data generation module 520 includes: submodule 5210 and data genaration submodule 5220 are determined in sequence.
Sequence determines that submodule 5210 is for the integer data ID according to each user, it is determined that the sequence of each user.
Sequence such as can from 1 or 0 open numbering and be incremented by one by one.
Data genaration submodule 5220, for the sequence according to each user, generates the user's logon data based on bitmap data structure.
Logon data memory module 530 is for being stored in the user's logon data based on bitmap data structure in data base.
In certain embodiments, data base can be such as HBase data base.
User logs in analysis module 540 for according to the user's logon data based on bitmap data structure, carrying out user and log in situation analysis, including: the first inquiry obtains arithmetic operation submodule 5420 and the first statistics submodule 5430 of submodule the 5410, first logic.
First inquiry obtains submodule 5410 and for inquiry from data base and obtains user's logon data all of every day in a period of time.
First logical operation submodule 5420 for user's logon data every day of acquisition is carried out logic or arithmetic operation, obtain the first result.
It is the number of persons logging of user in this period that first statistics submodule 5430 is used for adding up the number of 1 in the first result.
The data processing equipment 50 that present embodiment provides, carries out logical operations as desired by the user's logon data based on bitmap data structure, can quickly obtain the index needing statistics.
Fig. 7 is the block diagram of another data processing equipment according to an illustrative embodiments. As it is shown in fig. 7, data processing equipment 60 includes: ID distributes module 610, logon data generation module 620, logon data memory module 630 and user and logs in analysis module 640.
ID distributes module 610 for distributing a unique integer data ID for each user.
Wherein integer data ID is such as from 1 or 0 open numbering, and is incremented by one by one.
Logon data generation module 620, for the integer data ID according to each user, generates the user's logon data based on bitmap data structure.
User's logon data such as may include that user's logon data every day, every day Add User at least one in logon data, every server user's logon data and every tagging user logon data.
In certain embodiments, logon data generation module 620 includes: submodule 6210 and data genaration submodule 6220 are determined in sequence.
Sequence determines that submodule 6210 is for the integer data ID according to each user, it is determined that the sequence of each user.
Sequence such as can from 1 or 0 open numbering and be incremented by one by one.
Data genaration submodule 6220, for the sequence according to each user, generates the user's logon data based on bitmap data structure.
Logon data memory module 630 is for being stored in the user's logon data based on bitmap data structure in data base.
In certain embodiments, data base can be such as HBase data base.
User logs in analysis module 640 for according to the user's logon data based on bitmap data structure, carrying out user and log in situation analysis, including: the second inquiry obtains submodule the 6410, second logical operation submodule 6420 and the second statistics submodule 6430.
Second inquiry obtains the submodule 6410 every server user's logon data for Add User the every day inquiring about and obtaining a certain day from data base logon data and a server.
Second logical operation submodule 6420 carries out the arithmetic operation of logical AND for every server user's logon data of Added User the every day of this day logon data and server, obtains the second result.
Second statistics submodule 6430 is for adding up the number that Adds User that the number of 1 in the second result is this server of this day.
In certain embodiments, the second inquiry obtains after submodule 6410 is additionally operable to from data base inquiry and obtains this day user's logon data all of every day in a period of time. User logs in analysis module 640 and also includes: the 3rd logical operation submodule 6440 and the 3rd statistics submodule 6450.
3rd logical operation submodule 6440 for user's logon data every day of acquisition is carried out logic or arithmetic operation, obtain the 3rd result; And the second result and the 3rd result are carried out the arithmetic operation of logical AND, obtain the 4th result;
3rd statistics submodule 6450 is for adding up the retention number of users in this period later of Adding User that the number of 1 in the 4th result is this server of this day.
The data processing equipment 60 that present embodiment provides, carries out logical operations as desired by the user's logon data based on bitmap data structure, can quickly obtain the index needing statistics.
Through the above description of the embodiments, those skilled in the art is it can be readily appreciated that example embodiment described herein can be realized by software, it is also possible to the mode being combined necessary hardware by software is realized. Therefore, technical scheme according to embodiment of the present invention can embody with the form of software product, it (can be CD-ROM that this software product can be stored in a non-volatile memory medium, USB flash disk, portable hard drive etc.) in or network on, including some instructions so that computing equipment (can be personal computer, server, mobile terminal or the network equipment etc.) performs the method according to embodiment of the present invention.
More than it is particularly shown and described the illustrative embodiments of the present invention. It should be appreciated that the invention is not restricted to detailed construction described herein, set-up mode or realize method; On the contrary, it is intended to various amendments and equivalence in containing the spirit and scope being included in claims are arranged.
Claims (20)
1. a data processing method, it is characterised in that including:
A unique integer data ID is distributed for each user;
Described integer data ID according to each user, generates the user's logon data based on bitmap data structure; And
Described user's logon data based on bitmap data structure is stored in data base.
2. data processing method according to claim 1, it is characterised in that described integer data ID is from 1 or 0 open numbering and is incremented by one by one.
3. data processing method according to claim 1, it is characterised in that the described integer data ID according to each user, generates the user's logon data based on bitmap data structure and includes:
Described integer data ID according to each user, it is determined that the sequence of each user; And
Described sequence according to each user, generates described user's logon data based on bitmap data structure.
4. data processing method according to claim 3, it is characterised in that described sequence is from 1 or 0 open numbering and is incremented by one by one.
5. data processing method according to claim 1, it is characterised in that described data base is HBase data base.
6. the data processing method according to any one of claim 1-5, it is characterised in that also include: according to described user's logon data based on bitmap data structure, carry out user and log in situation analysis.
7. data processing method according to claim 6, it is characterized in that, described user's logon data based on bitmap data structure includes: every day user's logon data, every day Adds User at least one in logon data, every server user's logon data and every tagging user logon data.
8. data processing method according to claim 7, it is characterised in that according to described user's logon data based on bitmap data structure, carries out user and logs in situation analysis and include:
Inquiry obtain user's logon data all of described every day in a period of time from described data base;
The user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the first result; And
Adding up the number of 1 in described first result is the number of persons logging of user in described a period of time.
9. data processing method according to claim 7, it is characterised in that according to described user's logon data based on bitmap data structure, carries out user and logs in situation analysis and include:
Add User the described every day inquiring about and obtaining a certain day from described data base described every server user's logon data of logon data and a server;
Described every server user's logon data of the logon data that Added User the described every day of this day and described server carries out the arithmetic operation of logical AND, obtains the second result; And
Adding up the number of 1 in described second result is the number that Adds User of this day described server.
10. data processing method according to claim 9, it is characterised in that according to described user's logon data based on bitmap data structure, carries out user and logs in situation analysis and also include:
Inquiry user's logon data all of described every day in a period of time after obtaining this day from described data base;
The user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the 3rd result;
Described second result and described 3rd result are carried out the arithmetic operation of logical AND, obtains the 4th result; And
Add up the retention number of users in described a period of time later of Adding User that the number of 1 in described 4th result is this day described server.
11. a data processing equipment, it is characterised in that including:
ID distributes module, for distributing a unique integer data ID for each user;
Logon data generation module, for the described integer data ID according to each user, generates the user's logon data based on bitmap data structure; And
Logon data memory module, for being stored in described user's logon data based on bitmap data structure in data base.
12. data processing equipment according to claim 11, it is characterised in that described integer data ID is from 1 or 0 open numbering and is incremented by one by one.
13. data processing equipment according to claim 11, it is characterised in that described logon data generation module includes:
Submodule is determined in sequence, for the described integer data ID according to each user, it is determined that the sequence of each user; And
Data genaration submodule, for the described sequence according to each user, generates described user's logon data based on bitmap data structure.
14. data processing equipment according to claim 13, it is characterised in that described sequence is from 1 or 0 open numbering and is incremented by one by one.
15. data processing equipment according to claim 11, it is characterised in that described data base is HBase data base.
16. according to the data processing equipment described in any one of claim 11-15, it is characterised in that also include: user logs in analysis module, for according to described user's logon data based on bitmap data structure, carrying out user and log in situation analysis.
17. data processing equipment according to claim 16, it is characterized in that, described user's logon data based on bitmap data structure includes: every day user's logon data, every day Adds User at least one in logon data, every server user's logon data and every tagging user logon data.
18. data processing equipment according to claim 17, it is characterised in that described user logs in analysis module and includes:
First inquiry obtains submodule, for inquiry from described data base and obtain user's logon data all of described every day in a period of time;
First logical operation submodule, for user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the first result; And
First statistics submodule, is the number of persons logging of user in described a period of time for adding up the number of 1 in described first result.
19. data processing equipment according to claim 17, it is characterised in that described customer analysis module includes:
Second inquiry obtains submodule, described every server user's logon data of Add User the described every day for inquiring about and obtain a certain day from described data base logon data and a server;
Second logical operation submodule, carries out the arithmetic operation of logical AND, obtains the second result for described every server user's logon data of Added User described every day of this day logon data and described server; And
Second statistics submodule, is the number that Adds User of this day described server for adding up the number of 1 in described second result.
20. data processing equipment according to claim 19, it is characterised in that described second inquiry obtains after submodule is additionally operable to from described data base inquiry and obtains this day user's logon data all of described every day in a period of time; Described customer analysis module also includes:
3rd logical operation submodule, for user's logon data described every day obtained is carried out logic or arithmetic operation, obtain the 3rd result; And described second result and described 3rd result are carried out the arithmetic operation of logical AND, obtain the 4th result;
3rd statistics submodule, for adding up the retention number of users in described a period of time later of Adding User that the number of 1 in described 4th result is this day described server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510990963.5A CN105630972A (en) | 2015-12-24 | 2015-12-24 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510990963.5A CN105630972A (en) | 2015-12-24 | 2015-12-24 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105630972A true CN105630972A (en) | 2016-06-01 |
Family
ID=56045905
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510990963.5A Pending CN105630972A (en) | 2015-12-24 | 2015-12-24 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105630972A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106971121A (en) * | 2017-04-10 | 2017-07-21 | 深圳乐信软件技术有限公司 | Data processing method, device, server and storage medium |
| CN107294784A (en) * | 2017-07-10 | 2017-10-24 | 环球智达科技（北京）有限公司 | It is a kind of that statistical method is enlivened based on bit arithmetic |
| CN108415978A (en) * | 2018-02-09 | 2018-08-17 | 北京腾云天下科技有限公司 | User tag storage method, user's portrait computational methods and computing device |
| CN108665121A (en) * | 2017-03-27 | 2018-10-16 | Tcl集团股份有限公司 | A kind of method and system of computational intelligence terminal user retention ratio |
| WO2019024060A1 (en) * | 2017-08-03 | 2019-02-07 | 华为技术有限公司 | Data storage method and device, and storage medium |
| CN109656948A (en) * | 2018-11-15 | 2019-04-19 | 金蝶软件（中国）有限公司 | Bitmap data processing method, device, computer equipment and storage medium |
| CN110523083A (en) * | 2019-09-05 | 2019-12-03 | 网易（杭州）网络有限公司 | Processing method, device and the server device of log-in events |
| CN110727878A (en) * | 2019-09-19 | 2020-01-24 | 上海易点时空网络有限公司 | Distance calculation method and device for collaborative filtering, and collaborative filtering recommendation method and device |
| CN111080241A (en) * | 2019-12-04 | 2020-04-28 | 贵州非你莫属人才大数据有限公司 | Internet platform-based data-based talent management analysis system |
| CN111415200A (en) * | 2020-03-20 | 2020-07-14 | 腾讯音乐娱乐科技（深圳）有限公司 | Data processing method and device |
| CN112532748A (en) * | 2020-12-24 | 2021-03-19 | 北京百度网讯科技有限公司 | Message pushing method, device, equipment, medium and computer program product |
| CN112527803A (en) * | 2020-12-30 | 2021-03-19 | 北京百家科技集团有限公司 | Data processing method and device and computer equipment |
| CN113032470A (en) * | 2021-03-16 | 2021-06-25 | 深圳市麦谷科技有限公司 | Method and system for counting equipment data of Internet of vehicles |
| CN114138138A (en) * | 2022-01-30 | 2022-03-04 | 北京金堤科技有限公司 | Operation feedback method and device based on bitmap, storage medium and electronic equipment |
| CN114356972A (en) * | 2021-12-03 | 2022-04-15 | 四川科瑞软件有限责任公司 | Data processing method, and event time-based retrieval method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1514669A (en) * | 2003-07-23 | 2004-07-21 | 腾讯科技（深圳）有限公司 | Method and system of instant communication number state control |
| CN103593453A (en) * | 2013-11-20 | 2014-02-19 | 北京国双科技有限公司 | Method and device for calculating user retention ratio |
| CN103746851A (en) * | 2014-01-17 | 2014-04-23 | 中国联合网络通信集团有限公司 | Method and device for realizing counting of independent user number |
-
2015
- 2015-12-24 CN CN201510990963.5A patent/CN105630972A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1514669A (en) * | 2003-07-23 | 2004-07-21 | 腾讯科技（深圳）有限公司 | Method and system of instant communication number state control |
| CN103593453A (en) * | 2013-11-20 | 2014-02-19 | 北京国双科技有限公司 | Method and device for calculating user retention ratio |
| CN103746851A (en) * | 2014-01-17 | 2014-04-23 | 中国联合网络通信集团有限公司 | Method and device for realizing counting of independent user number |
Non-Patent Citations (1)
| Title |
|---|
| AVICHAL: "Fast,easy,realtime metrics using Redis bitmaps", 《HTTP://BLOG.GETSPOOL.COM/2011/11/29/FAST-EASY-REALTIME-METRICS-USING-REDIS-BITMAPS/》 * |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108665121A (en) * | 2017-03-27 | 2018-10-16 | Tcl集团股份有限公司 | A kind of method and system of computational intelligence terminal user retention ratio |
| CN106971121B (en) * | 2017-04-10 | 2021-01-01 | 深圳乐信软件技术有限公司 | Data processing method, device, server and storage medium |
| CN106971121A (en) * | 2017-04-10 | 2017-07-21 | 深圳乐信软件技术有限公司 | Data processing method, device, server and storage medium |
| CN107294784A (en) * | 2017-07-10 | 2017-10-24 | 环球智达科技（北京）有限公司 | It is a kind of that statistical method is enlivened based on bit arithmetic |
| WO2019024060A1 (en) * | 2017-08-03 | 2019-02-07 | 华为技术有限公司 | Data storage method and device, and storage medium |
| US11249969B2 (en) | 2017-08-03 | 2022-02-15 | Huawei Techologies Co., Ltd. | Data storage method and apparatus, and storage medium |
| CN108415978A (en) * | 2018-02-09 | 2018-08-17 | 北京腾云天下科技有限公司 | User tag storage method, user's portrait computational methods and computing device |
| CN108415978B (en) * | 2018-02-09 | 2021-04-09 | 北京腾云天下科技有限公司 | User tag storage method, user portrait calculation method and calculation equipment |
| CN109656948A (en) * | 2018-11-15 | 2019-04-19 | 金蝶软件（中国）有限公司 | Bitmap data processing method, device, computer equipment and storage medium |
| CN109656948B (en) * | 2018-11-15 | 2021-01-22 | 金蝶软件（中国）有限公司 | Bitmap data processing method and device, computer equipment and storage medium |
| CN110523083A (en) * | 2019-09-05 | 2019-12-03 | 网易（杭州）网络有限公司 | Processing method, device and the server device of log-in events |
| CN110727878A (en) * | 2019-09-19 | 2020-01-24 | 上海易点时空网络有限公司 | Distance calculation method and device for collaborative filtering, and collaborative filtering recommendation method and device |
| CN111080241A (en) * | 2019-12-04 | 2020-04-28 | 贵州非你莫属人才大数据有限公司 | Internet platform-based data-based talent management analysis system |
| CN111415200B (en) * | 2020-03-20 | 2023-05-26 | 腾讯音乐娱乐科技（深圳）有限公司 | Data processing method and device |
| CN111415200A (en) * | 2020-03-20 | 2020-07-14 | 腾讯音乐娱乐科技（深圳）有限公司 | Data processing method and device |
| CN112532748A (en) * | 2020-12-24 | 2021-03-19 | 北京百度网讯科技有限公司 | Message pushing method, device, equipment, medium and computer program product |
| CN112527803A (en) * | 2020-12-30 | 2021-03-19 | 北京百家科技集团有限公司 | Data processing method and device and computer equipment |
| CN112527803B (en) * | 2020-12-30 | 2024-01-26 | 北京百家科技集团有限公司 | Data processing method and device and computer equipment |
| CN113032470A (en) * | 2021-03-16 | 2021-06-25 | 深圳市麦谷科技有限公司 | Method and system for counting equipment data of Internet of vehicles |
| CN114356972B (en) * | 2021-12-03 | 2022-08-23 | 四川科瑞软件有限责任公司 | Data processing method, and event time-based retrieval method and device |
| CN114356972A (en) * | 2021-12-03 | 2022-04-15 | 四川科瑞软件有限责任公司 | Data processing method, and event time-based retrieval method and device |
| CN114138138A (en) * | 2022-01-30 | 2022-03-04 | 北京金堤科技有限公司 | Operation feedback method and device based on bitmap, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105630972A (en) | Data processing method and device | |
| CN110457672B (en) | Keyword determination method and device, electronic equipment and storage medium | |
| CN101079033A (en) | Integrative searching result sequencing system and method | |
| CN101968806A (en) | Data storage method, querying method and device | |
| CN107918618B (en) | Data processing method and device | |
| US20210191927A1 (en) | Secure aggregate function computation system, secure computation apparatus, secure aggregate function computation method, and program | |
| CN104268295A (en) | Data query method and device | |
| US20180307743A1 (en) | Mapping method and device | |
| CN108427736B (en) | Method for querying data | |
| CN112650743A (en) | Funnel data analysis method and system, electronic device and storage medium | |
| CN104199977A (en) | Method for creating information search based on data in database | |
| CN105512156A (en) | Method and device for generation of click models | |
| CN105279198A (en) | Data table storage method, data table modification method, data table query method and data table statistical method | |
| US11240012B2 (en) | Secure aggregate order system, secure computation apparatus, secure aggregate order method, and program | |
| CN114741368A (en) | Log data statistical method based on artificial intelligence and related equipment | |
| US8407255B1 (en) | Method and apparatus for exploiting master-detail data relationships to enhance searching operations | |
| CN104915435A (en) | Data query method and device | |
| CN106844713A (en) | A kind of method and device of data cube generation | |
| Haigh et al. | Colossus and programmability | |
| CN114398883B (en) | Presentation generation method and device, computer readable storage medium and server | |
| CN109145307A (en) | User's face sketch recognition method, method for pushing, device, equipment and storage medium | |
| CN112328641B (en) | Multi-dimensional data aggregation method and device and computer equipment | |
| CN104156492A (en) | Method and device for prompting search content | |
| CN112182107B (en) | List data acquisition method, device, computer equipment and storage medium | |
| CN115186188A (en) | Product recommendation method, device and equipment based on behavior analysis and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160601 |
|
| RJ01 | Rejection of invention patent application after publication |