CN106202374A - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN106202374A CN106202374A CN201610534513.XA CN201610534513A CN106202374A CN 106202374 A CN106202374 A CN 106202374A CN 201610534513 A CN201610534513 A CN 201610534513A CN 106202374 A CN106202374 A CN 106202374A
- Authority
- CN
- China
- Prior art keywords
- data
- user behavior
- behavior data
- preset
- amount
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data processing method and device, for improving the warehouse-in mechanism of user behavior data.Method includes: when there is user access activity, using N number of thread to read user behavior data simultaneously, and user behavior data writes the first data base；When the data volume of the user behavior data in the first data base reaches the first preset data amount, the user behavior data of the first preset data amount is write the second data base；When the user behavior data in the second data base meets and presets statistical condition, carry out statistical analysis to meeting the user behavior data presetting statistical condition；Wherein, N is the integer more than 1, and the second preset data amount is more than the first preset data amount.Multiple data base can be used when this technical scheme makes to put user behavior data in storage to process respectively, thus avoid the defect problem of data management in tradition warehouse-in mechanism, the warehouse-in achieving mass users behavioral data processes, the warehouse-in mechanism of perfect user behavior data.
Description
Technical field
The present invention relates to technical field of data storage, particularly relate to a kind of data processing method and device.
Background technology
User behavior analysis, refers in the case of obtaining website visiting amount master data, relevant data are added up,
Analyze, therefrom find that user accesses the rule of website, and these rules are combined with net marketing strategy etc., thus find mesh
Problem that may be present in front network marketing activity, and provide foundation for revising or reformulate net marketing strategy further.
At big data age, along with mobile Internet business and the quick growth of number of users, traditional warehouse-in mechanism is
Through being difficult to use the stock management demand of mass users behavioral data.Therefore, the most effectively mass users behavioral data is entered
Row stock management becomes problem the most in the urgent need to address.
Summary of the invention
The embodiment of the present invention provides a kind of data processing method and device, for improving the warehouse-in machine of user behavior data
System.
A kind of data processing method, comprises the following steps:
When there is user access activity, use N number of thread to read user behavior data simultaneously, and by described user behavior
Data write the first data base；
When the data volume of the user behavior data in described first data base reaches the first preset data amount, by described
The user behavior data of one preset data amount writes the second data base；
When the user behavior data in described second data base meets and presets statistical condition, meet default statistics to described
The user behavior data of condition carries out statistical analysis；
Wherein, described N is the integer more than 1, and described second preset data amount is more than described first preset data amount.
Some beneficial effects of the embodiment of the present invention may include that
Use the technical scheme in the embodiment of the present invention, it is possible to when there is user access activity, initially with multithreading
Mode reads user behavior data, and the user behavior data that multithreading reads is write the first data base, compared to existing skill
For using the mode of single-threaded reading data in art, this technical scheme makes the reading of user behavior data until warehouse-in efficiency
Higher；Secondly, it is possible to when the data volume of the user behavior data in the first data base reaches the first preset data amount, by first
The user behavior data of preset data amount writes the second data base, and the user behavior data in the second data base meets default
During statistical condition, user behavior data is carried out statistical analysis so that many numbers when user behavior data is put in storage, can be used
Process respectively according to storehouse, thus avoid the defect problem of data management in tradition warehouse-in mechanism, it is achieved that mass users behavioral data
Warehouse-in process, the warehouse-in mechanism of perfect user behavior data.
In one embodiment, described described user behavior data is write the first data base after, described method is also wrapped
Include:
Add up the data volume of user behavior data in described first data base；This step includes:
In described first data base, often write a user behavior data, then generate this user behavior data corresponding
From increasing number；
Judge the described multiple whether reaching described first preset data amount from the value of increasing number；
When the described value from increasing number reaches the multiple of described first preset data amount, determine in described first data base
The data volume of user behavior data reach the first preset data amount.
In this embodiment, it is possible to generate corresponding from increasing number for every user behavior data, and according to from increasing number
Whether value reaches the multiple of the first preset data amount judges whether the data volume of the user behavior data in the first data base reaches
To the first preset data amount so that the determination of the data volume of user behavior data is more simple and efficient.
In one embodiment, described second data base includes at least two subregion, and wherein, each subregion is respectively used to deposit
The second preset data amount in storage preset range is from the user behavior data of increasing number；
The described user behavior data by described first preset data amount writes the second data base, including:
According to the presetting from increasing number corresponding from increasing number and described each subregion that described user behavior data is corresponding
Scope, determines the subregion that described user behavior data is corresponding in described second data base；
Described user behavior data is write in the subregion of its correspondence.
In this embodiment, it is possible to according to the subregion determining this data place from increasing number that user behavior data is corresponding,
And then user behavior data is write in the subregion of its correspondence so that more bar when user behavior data is write the second data base
Physics and chemistry, sharpening, so when making subsequent calls user behavior data convenient accurately.
In one embodiment, the described user behavior data by described first preset data amount write the second data base it
After, described method also includes:
Judge to be currently written into whether the data volume of the user behavior data in the current bay of data reaches described second pre-
If data volume；
When the data volume of the user behavior data in described current bay reaches described second preset data amount, determine institute
State the user behavior data in current bay and reach described default statistical condition；
The described user behavior data to described satisfied default statistical condition carries out statistical analysis, including:
User behavior data in described current bay is carried out statistical analysis.
In this embodiment, whether the data volume of the user behavior data being currently written in the current bay of data by judgement
Reach the second preset data amount, and when reaching the second preset data amount, the user behavior data in current bay is added up
Analyze so that this technical scheme can determine whether to carry out user behavior data statistical analysis exactly, and which is used
Family behavioral data carries out statistical analysis.
In one embodiment, the data volume of the user behavior data that described judgement is currently written in the current bay of data
Whether reach described second preset data amount, including:
Judge to write the user behavior data in described current bay corresponding whether reach described the from the value of increasing number
The multiple of two preset data amounts；
Preset when the value from increasing number that the user behavior data write in described current bay is corresponding reaches described second
During the multiple of data volume, determine that the user behavior data in described current bay reaches described default statistical condition.
In this embodiment, it is possible to according to user behavior data corresponding whether reach the second preset data from the value of increasing number
The multiple of amount judges whether the data volume of the user behavior data in the second data base reaches the second preset data amount so that use
The determination of the data volume of family behavioral data is more simple and efficient.
In one embodiment, described first data base is Redis data base, and described second data base is MySQL data
Storehouse.
A kind of data processing equipment, including:
Read module, for when there is user access activity, using N number of thread to read user behavior data simultaneously, and
Described user behavior data is write the first data base；
Writing module, for reaching the first preset data when the data volume of the user behavior data in described first data base
During amount, the user behavior data of described first preset data amount is write the second data base；
Analyze module, for when the user behavior data in described second data base meets and presets statistical condition, to institute
State the satisfied user behavior data presetting statistical condition and carry out statistical analysis；
Wherein, described N is the integer more than 1, and described second preset data amount is more than described first preset data amount.
In one embodiment, described device also includes:
Statistical module, after described user behavior data is write the first data base, adds up described first data base
In the data volume of user behavior data；
Described statistical module includes:
Signal generating unit, for often writing a user behavior data in described first data base, then generates this user
Behavioral data corresponding from increasing number；
Judging unit, for judging the described multiple whether reaching described first preset data amount from the value of increasing number；
First determines unit, is used for when the described value from increasing number reaches the multiple of described first preset data amount, really
The data volume of the user behavior data in fixed described first data base reaches the first preset data amount.
In one embodiment, described second data base includes at least two subregion, and wherein, each subregion is respectively used to deposit
The second preset data amount in storage preset range is from the user behavior data of increasing number；
Said write module includes:
Second determines unit, for according to corresponding corresponding from increasing number and described each subregion of described user behavior data
The preset range from increasing number, determine the subregion that described user behavior data is corresponding in described second data base；
Writing unit, in the subregion that described user behavior data writes its correspondence.
In one embodiment, described device also includes:
Judge module, after the user behavior data of described first preset data amount is write the second data base, sentences
Whether the data volume of the user behavior data in the disconnected current bay being currently written into data reaches described second preset data amount；
Determine module, for reaching described second present count when the data volume of the user behavior data in described current bay
During according to amount, determine that the user behavior data in described current bay reaches described default statistical condition；
Described analysis module, is additionally operable to the user behavior data in described current bay is carried out statistical analysis.
Other features and advantages of the present invention will illustrate in the following description, and, partly become from description
Obtain it is clear that or understand by implementing the present invention.The purpose of the present invention and other advantages can be by the explanations write
Structure specifically noted in book, claims and accompanying drawing realizes and obtains.
Below by drawings and Examples, technical scheme is described in further detail.
Accompanying drawing explanation
Accompanying drawing is for providing a further understanding of the present invention, and constitutes a part for description, with the reality of the present invention
Execute example together for explaining the present invention, be not intended that limitation of the present invention.In the accompanying drawings:
Fig. 1 is the flow chart of a kind of data processing method in the embodiment of the present invention；
Fig. 2 is the flow chart of the method for the data volume of a kind of counting user behavioral data in the embodiment of the present invention；
Fig. 3 is the flow chart of step S12 in a kind of data processing method in the embodiment of the present invention；
Fig. 4 is a kind of stream judging whether user behavior data meets the method presetting statistical condition in the embodiment of the present invention
Cheng Tu；
Fig. 5 is the block diagram of a kind of data processing equipment in the embodiment of the present invention；
Fig. 6 is the block diagram of statistical module in a kind of data processing equipment in the embodiment of the present invention.
Detailed description of the invention
Below in conjunction with accompanying drawing, the preferred embodiments of the present invention are illustrated, it will be appreciated that preferred reality described herein
Execute example be merely to illustrate and explain the present invention, be not intended to limit the present invention.
Fig. 1 is the flow chart of a kind of data processing method in the embodiment of the present invention.As it is shown in figure 1, the method includes following
Step S11-S13:
Step S11, when there is user access activity, uses N number of thread to read user behavior data simultaneously, and by user
Behavioral data writes the first data base.
In one embodiment, the first data base can be Redis data base, owing to Redis data store internal mechanism is
Single-threaded operation, therefore uses N number of thread read user behavior data and use single-threaded by user behavior data write
Redis data base, can either improve the reading efficiency of user behavior data, is avoided that again dirty reading situation during warehouse-in.Concrete,
N number of thread loops reads user behavior data, and each thread often processes the first preset data user behavior data and then puts one in storage
Secondary (i.e. write Redis data base), sequence warehouse-in mechanism therein can use the incrBy order of Redis.
Step S12, when the data volume of the user behavior data in the first data base reaches the first preset data amount, by
The user behavior data of one preset data amount writes the second data base.
Step S13, when the user behavior data in the second data base meets and presets statistical condition, presets statistics to meeting
The user behavior data of condition carries out statistical analysis；
Wherein, N is the integer more than 1, and the second preset data amount is more than the first preset data amount.Under normal circumstances, second
Preset data amount is far longer than the first preset data amount, and such as, the second preset data amount is 1,000,000, and the first preset data amount is 1
Ten thousand.
Use the technical scheme in the embodiment of the present invention, it is possible to when there is user access activity, initially with multithreading
Mode reads user behavior data, and the user behavior data that multithreading reads is write the first data base, compared to existing skill
For using the mode of single-threaded reading data in art, this technical scheme makes the reading of user behavior data until warehouse-in efficiency
Higher；Secondly, it is possible to when the data volume of the user behavior data in the first data base reaches the first preset data amount, by first
The user behavior data of preset data amount writes the second data base, and the user behavior data in the second data base meets default
During statistical condition, user behavior data is carried out statistical analysis so that many numbers when user behavior data is put in storage, can be used
Process respectively according to storehouse, thus avoid the defect problem of data management in tradition warehouse-in mechanism, it is achieved that mass users behavioral data
Warehouse-in process, the warehouse-in mechanism of perfect user behavior data.
In one embodiment, perform after step S11, the further comprising the steps of A1 of said method: step A1, statistics the
The data volume of the user behavior data in one data base.Concrete, this step A1 can be embodied as step S21-described in Fig. 2
S24:
Step S21, often writes a user behavior data in the first data base, then generates this user behavior data pair
Answer from increasing number.
Wherein, from increasing the sequence number that serial number increases gradually with unit sequence number.Such as, when one user behavior data of write
Time, if previous bar is written of increasing serial number 20, the then user behavior being currently written certainly that user behavior data is corresponding
What data were corresponding increases serial number 21 certainly, next certainly increasing serial number 22 corresponding by being written of user behavior data.
Step S22, it is judged that whether reach the multiple of the first preset data amount from the value of increasing number.If from the value of increasing number
Reach the multiple of the first preset data amount, then perform step S23；If being not up to the first preset data amount from the value of increasing number
Multiple, then perform step S24.
Step S23, determines that the data volume of the user behavior data in the first data base reaches the first preset data amount.
Step S24, determines that the data volume of the user behavior data in the first data base is not up to the first preset data amount.
Often reach the first preset data amount owing to writing the data volume of the user behavior data of the first data base, will be write
Enter the second data base, and determine the standard of the data volume of user behavior data be its correspondence from increasing number, therefore, this enforcement
The user behavior judging in the first data base with the multiple whether reaching the first preset data amount from increasing number for standard in example
Whether the data volume of data reaches the first preset data amount.Such as, the first preset data amount is 10,000, the use in the first data base
What family behavioral data was corresponding has reached 10,000 from increasing number, illustrates now have 10,000 user behavior datas in the first data base, this
Time these 10,000 user behavior datas can be write the second data base.Meanwhile, the first data base continues be written into new user's row
For data, and correspondence from increasing number from the beginning of 10001, until being written of corresponding the reaching from increasing number of user behavior data
20000 (2 times of the i.e. first preset data amount), are now written into 10,000 user behavior datas again in the first data base, by this 1
Ten thousand user behavior datas write the second data base, continue to write new user behavior data in the first data base simultaneously, as
This circulation is carried out, until all of user behavior data is successfully put in storage.
In this embodiment, it is possible to generate corresponding from increasing number for every user behavior data, and according to from increasing number
Whether value reaches the multiple of the first preset data amount judges whether the data volume of the user behavior data in the first data base reaches
To the first preset data amount so that the determination of the data volume of user behavior data is more simple and efficient.
In one embodiment, the second data base includes at least two subregion, and wherein, it is pre-that each subregion is respectively used to storage
If the second preset data amount in scope is from the user behavior data of increasing number.Now, step S12 can perform as such as Fig. 3 institute
Step S31-S32 shown:
Step S31, according to the presetting from increasing number corresponding from increasing number and each subregion that user behavior data is corresponding
Scope, determines the subregion that user behavior data is corresponding in the second data base.
Step S32, writes user behavior data in the subregion of its correspondence.
In this embodiment, each subregion can use Digital ID.Such as, the first subregion for memory range be [1,100 ten thousand)
1,000,000 from the user behavior data of increasing number, the second subregion for memory range be [1,000,000,2,000,000) 1,000,000
From the user behavior data of increasing number, etc..
In this embodiment, it is possible to according to the subregion determining this data place from increasing number that user behavior data is corresponding,
And then user behavior data is write in the subregion of its correspondence so that more bar when user behavior data is write the second data base
Physics and chemistry, sharpening, so when making subsequent calls user behavior data convenient accurately.
In one embodiment, performing after step S12, whether the user behavior data needing to judge in the second data base
Meet and preset statistical condition, and determination methods can perform as step S41-S43 as shown in Figure 4:
Step S41, it is judged that whether the data volume of the user behavior data being currently written in the current bay of data reaches
Two preset data amounts；If the data volume of the user behavior data in current bay reaches the second preset data amount, then perform step
Rapid S42；If the data volume of the user behavior data in current bay is not up to the second preset data amount, then perform step S43.
Concrete, this step step S41 can perform as follows: judges the user behavior data in write current bay
The multiple of the corresponding second preset data amount that whether reaches from the value of increasing number；User behavior data in write current bay
When the corresponding value from increasing number reaches the multiple of the second preset data amount, determine that the user behavior data in current bay reaches
Preset statistical condition.
Owing to the data volume of the user behavior data in write current bay often reaches the second preset data amount, will be write
Enter in new subregion, and determine the standard of the data volume of user behavior data be its correspondence from increasing number, therefore, this enforcement
In example with whether reach the second preset data amount from increasing number multiple for standard to the user behavior number judging in current bay
According to data volume whether reach the second preset data amount.Such as, the second preset data amount is 1,000,000, the user in current bay
What behavioral data was corresponding has reached 1,000,000 from increasing number, illustrates now have 1,000,000 user behavior datas in current bay, this
Time these 1,000,000 user behavior datas can be carried out statistical analysis.The user behavior data continued to write to then is written to down
In one subregion, when next subregion is written of corresponding the reaching 2,000,000 from increasing number (i.e. second is pre-of user behavior data
If the 2 of data volume times) time, this next one subregion is written into 1,000,000 user behavior datas, now can be to these 1,000,000 use
Family behavioral data carries out statistical analysis, continues to write user behavior data in new subregion simultaneously.
Step S42, determines that the user behavior data in current bay reaches to preset statistical condition.
Step S43, determines that the user behavior data in current bay does not arrives and presets statistical condition.
In this embodiment, whether the data volume of the user behavior data being currently written in the current bay of data by judgement
Reach the second preset data amount, and when reaching the second preset data amount, the user behavior data in current bay is added up
Analyze so that this technical scheme can determine whether to carry out user behavior data statistical analysis exactly, and which is used
Family behavioral data carries out statistical analysis.Furthermore it is possible to according to user behavior data corresponding whether reach from the value of increasing number
The multiple of two preset data amounts judges whether the data volume of the user behavior data in the second data base reaches the second present count
According to amount so that the determination of the data volume of user behavior data is more simple and efficient.
In one embodiment, the second data base may also include current storehouse and history library.Current storehouse refers to currently write
The data base of access customer behavioral data, history library is then used for storing least one set the second predetermined number user behavior data.
In this case, the subregion that above-mentioned described each subregion is in history library.In the specific implementation, write in the second data base
During user behavior data, can first write data in the current storehouse of the second data base, when current storehouse is written of user behavior
The data volume of data reaches the second preset data amount, then the user behavior data write of the second preset data amount in current storehouse gone through
In the subregion of Shi Ku.Concrete which subregion that writes then can determine according to above-mentioned mentioned mode, i.e. according to user behavior
The preset range from increasing number corresponding from increasing number and each subregion corresponding to data determines that user behavior data is in history
Subregion corresponding in storehouse.User behavior data in history library for follow-up statistics, analyze, the operation such as lookup.
In any of the above-described embodiment, the first data base can be Redis data base, and the second data base can be MySQL data
Storehouse.
Fig. 5 is the block diagram of a kind of data processing equipment in the embodiment of the present invention.As it is shown in figure 5, this device includes:
Read module 51, for when there is user access activity, using N number of thread to read user behavior data simultaneously,
And described user behavior data is write the first data base；
Writing module 52, for reaching the first preset data amount when the data volume of the user behavior data in the first data base
Time, the user behavior data of the first preset data amount is write the second data base；
Analyze module 53, for when the user behavior data in the second data base meets and presets statistical condition, to meeting
The user behavior data presetting statistical condition carries out statistical analysis；
Wherein, N is the integer more than 1, and the second preset data amount is more than the first preset data amount.
In one embodiment, as shown in Figure 6, said apparatus also includes:
Statistical module 54, after user behavior data is write the first data base, adds up the use in the first data base
The data volume of family behavioral data；
Statistical module 54 includes:
Signal generating unit 541, for often writing a user behavior data in the first data base, then generates this user's row
For data corresponding from increasing number；
Judging unit 542, for judging whether to reach the multiple of the first preset data amount from the value of increasing number；
First determines unit 543, for when reaching the multiple of the first preset data amount from the value of increasing number, determines first
The data volume of the user behavior data in data base reaches the first preset data amount.
In one embodiment, the second data base includes at least two subregion, and wherein, it is pre-that each subregion is respectively used to storage
If the second preset data amount in scope is from the user behavior data of increasing number；
Writing module 52 includes:
Second determines unit, for according to user behavior data corresponding from increasing number and each subregion corresponding from increasing
Number preset range, determine the subregion that user behavior data is corresponding in the second data base；
Writing unit, in the subregion that user behavior data writes its correspondence.
In one embodiment, said apparatus also includes:
Judge module, after writing the second data base by the user behavior data of the first preset data amount, it is judged that just
Whether the data volume of the user behavior data in the current bay of write data reaches the second preset data amount；
Determine module, be used for when the data volume of the user behavior data in current bay reaches the second preset data amount,
Determine that the user behavior data in current bay reaches to preset statistical condition；
Analyze module, be additionally operable to the user behavior data in current bay is carried out statistical analysis.
In one embodiment, it is judged that module is additionally operable to:
Judge to write the user behavior data in current bay corresponding whether reach the second present count from the value of increasing number
Multiple according to amount；
The value from increasing number corresponding when the user behavior data in write current bay reaches the second preset data amount
During multiple, determine that the user behavior data in current bay reaches to preset statistical condition.
In one embodiment, the first data base is Redis data base, and the second data base is MySQL database.
Use the technical scheme in the embodiment of the present invention, it is possible to when there is user access activity, initially with multithreading
Mode reads user behavior data, and the user behavior data that multithreading reads is write the first data base, compared to existing skill
For using the mode of single-threaded reading data in art, this technical scheme makes the reading of user behavior data until warehouse-in efficiency
Higher；Secondly, it is possible to when the data volume of the user behavior data in the first data base reaches the first preset data amount, by first
The user behavior data of preset data amount writes the second data base, and the user behavior data in the second data base meets default
During statistical condition, user behavior data is carried out statistical analysis so that many numbers when user behavior data is put in storage, can be used
Process respectively according to storehouse, thus avoid the defect problem of data management in tradition warehouse-in mechanism, it is achieved that mass users behavioral data
Warehouse-in process, the warehouse-in mechanism of perfect user behavior data.
About the device in above-described embodiment, wherein modules performs the concrete mode of operation in relevant the method
Embodiment in be described in detail, explanation will be not set forth in detail herein.
Those skilled in the art are it should be appreciated that embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the reality in terms of the present invention can use complete hardware embodiment, complete software implementation or combine software and hardware
Execute the form of example.And, the present invention can use at one or more computers wherein including computer usable program code
The shape of the upper computer program implemented of usable storage medium (including but not limited to disk memory and optical memory etc.)
Formula.
The present invention is with reference to method, equipment (system) and the flow process of computer program according to embodiments of the present invention
Figure and/or block diagram describe.It should be understood that can the most first-class by computer program instructions flowchart and/or block diagram
Flow process in journey and/or square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
Instruction arrives the processor of general purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce
A raw machine so that the instruction performed by the processor of computer or other programmable data processing device is produced for real
The device of the function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame now.
These computer program instructions may be alternatively stored in and computer or other programmable data processing device can be guided with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in this computer-readable memory produces and includes referring to
Make the manufacture of device, this command device realize at one flow process of flow chart or multiple flow process and/or one square frame of block diagram or
The function specified in multiple square frames.
These computer program instructions also can be loaded in computer or other programmable data processing device so that at meter
Perform sequence of operations step on calculation machine or other programmable devices to produce computer implemented process, thus at computer or
The instruction performed on other programmable devices provides for realizing at one flow process of flow chart or multiple flow process and/or block diagram one
The step of the function specified in individual square frame or multiple square frame.
Obviously, those skilled in the art can carry out various change and the modification essence without deviating from the present invention to the present invention
God and scope.So, if these amendments of the present invention and modification belong to the scope of the claims in the present invention and equivalent technologies thereof
Within, then the present invention is also intended to comprise these change and modification.
Claims (10)
1. a data processing method, it is characterised in that including:
When there is user access activity, use N number of thread to read user behavior data simultaneously, and by described user behavior data
Write the first data base；
When the data volume of the user behavior data in described first data base reaches the first preset data amount, by described first pre-
If the user behavior data of data volume writes the second data base；
When the user behavior data in described second data base meets and presets statistical condition, meet default statistical condition to described
User behavior data carry out statistical analysis；
Wherein, described N is the integer more than 1, and described second preset data amount is more than described first preset data amount.
Method the most according to claim 1, it is characterised in that described described user behavior data is write the first data base
Afterwards, described method also includes:
Add up the data volume of user behavior data in described first data base；This step includes:
In described first data base, often write a user behavior data, then generate corresponding certainly the increasing of this user behavior data
Sequence number；
Judge the described multiple whether reaching described first preset data amount from the value of increasing number；
When the described value from increasing number reaches the multiple of described first preset data amount, determine the use in described first data base
The data volume of family behavioral data reaches the first preset data amount.
Method the most according to claim 2, it is characterised in that described second data base includes at least two subregion, its
In, the second preset data amount that each subregion is respectively used to store in preset range is from the user behavior data of increasing number；
The described user behavior data by described first preset data amount writes the second data base, including:
According to the preset range from increasing number corresponding from increasing number and described each subregion that described user behavior data is corresponding,
Determine the subregion that described user behavior data is corresponding in described second data base；
Described user behavior data is write in the subregion of its correspondence.
Method the most according to claim 3, it is characterised in that the described user behavior number by described first preset data amount
After writing the second data base, described method also includes:
Judge whether the data volume of the user behavior data being currently written in the current bay of data reaches described second present count
According to amount；
When the data volume of the user behavior data in described current bay reaches described second preset data amount, determine described working as
User behavior data in front subregion reaches described default statistical condition；
The described user behavior data to described satisfied default statistical condition carries out statistical analysis, including:
User behavior data in described current bay is carried out statistical analysis.
Method the most according to claim 4, it is characterised in that described judgement is currently written into the use in the current bay of data
Whether the data volume of family behavioral data reaches described second preset data amount, including:
Judge to write the user behavior data in described current bay corresponding whether reach described second pre-from the value of increasing number
If the multiple of data volume；
The value from increasing number corresponding when the user behavior data write in described current bay reaches described second preset data
During the multiple measured, determine that the user behavior data in described current bay reaches described default statistical condition.
6. according to the method described in any one of claim 1-4, it is characterised in that described first data base is Redis data base,
Described second data base is MySQL database.
7. a data processing equipment, it is characterised in that including:
Read module, for when there is user access activity, using N number of thread to read user behavior data simultaneously, and by institute
State user behavior data and write the first data base；
Writing module, for reaching the first preset data amount when the data volume of the user behavior data in described first data base
Time, the user behavior data of described first preset data amount is write the second data base；
Analyze module, for when the user behavior data in described second data base meets and presets statistical condition, to described full
Foot is preset the user behavior data of statistical condition and is carried out statistical analysis；
Wherein, described N is the integer more than 1, and described second preset data amount is more than described first preset data amount.
Device the most according to claim 7, it is characterised in that described device also includes:
Statistical module, after described user behavior data is write the first data base, adds up in described first data base
The data volume of user behavior data；
Described statistical module includes:
Signal generating unit, for often writing a user behavior data in described first data base, then generates this user behavior
Data corresponding from increasing number；
Judging unit, for judging the described multiple whether reaching described first preset data amount from the value of increasing number；
First determines unit, for when the described value from increasing number reaches the multiple of described first preset data amount, determines institute
The data volume stating the user behavior data in the first data base reaches the first preset data amount.
Device the most according to claim 8, it is characterised in that described second data base includes at least two subregion, its
In, the second preset data amount that each subregion is respectively used to store in preset range is from the user behavior data of increasing number；
Said write module includes:
Second determines unit, for according to described user behavior data corresponding from increasing number and described each subregion corresponding from
The preset range of increasing number, determines the subregion that described user behavior data is corresponding in described second data base；
Writing unit, in the subregion that described user behavior data writes its correspondence.
Device the most according to claim 8, it is characterised in that described device also includes:
Judge module, after writing the second data base by the user behavior data of described first preset data amount, it is judged that just
Whether the data volume of the user behavior data in the current bay of write data reaches described second preset data amount；
Determine module, for reaching described second preset data amount when the data volume of the user behavior data in described current bay
Time, determine that the user behavior data in described current bay reaches described default statistical condition；
Described analysis module, is additionally operable to the user behavior data in described current bay is carried out statistical analysis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610534513.XA CN106202374A (en) | 2016-07-07 | 2016-07-07 | A kind of data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610534513.XA CN106202374A (en) | 2016-07-07 | 2016-07-07 | A kind of data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106202374A true CN106202374A (en) | 2016-12-07 |
Family
ID=57472760
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610534513.XA Pending CN106202374A (en) | 2016-07-07 | 2016-07-07 | A kind of data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106202374A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018149078A1 (en) * | 2017-02-16 | 2018-08-23 | 平安科技（深圳）有限公司 | Data processing method, apparatus and device, and computer readable storage medium |
| CN108874798A (en) * | 2017-05-09 | 2018-11-23 | 北京京东尚科信息技术有限公司 | A kind of big data sort method and system |
| CN109299079A (en) * | 2018-09-11 | 2019-02-01 | 南京朝焱智能科技有限公司 | A kind of high-speed data library design method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101354713A (en) * | 2008-09-08 | 2009-01-28 | 大唐软件技术股份有限公司 | Method and system for storing data |
| CN102693307A (en) * | 2012-05-24 | 2012-09-26 | 上海克而瑞信息技术有限公司 | Website user access behavior recording and analyzing system |
| CN102946319A (en) * | 2012-09-29 | 2013-02-27 | 焦点科技股份有限公司 | System and method for analyzing network user behavior information |
| CN103617294A (en) * | 2013-12-17 | 2014-03-05 | 江苏名通信息科技有限公司 | User behavior analysis method under LINUX system |
| CN103873583A (en) * | 2014-03-24 | 2014-06-18 | 北京聚思信息咨询有限公司 | Method and system for analyzing behaviors of internet users based on cloud platform |
| CN103886068A (en) * | 2014-03-20 | 2014-06-25 | 北京国双科技有限公司 | Data processing method and device for Internet user behavior analysis |
-
2016
- 2016-07-07 CN CN201610534513.XA patent/CN106202374A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101354713A (en) * | 2008-09-08 | 2009-01-28 | 大唐软件技术股份有限公司 | Method and system for storing data |
| CN102693307A (en) * | 2012-05-24 | 2012-09-26 | 上海克而瑞信息技术有限公司 | Website user access behavior recording and analyzing system |
| CN102946319A (en) * | 2012-09-29 | 2013-02-27 | 焦点科技股份有限公司 | System and method for analyzing network user behavior information |
| CN103617294A (en) * | 2013-12-17 | 2014-03-05 | 江苏名通信息科技有限公司 | User behavior analysis method under LINUX system |
| CN103886068A (en) * | 2014-03-20 | 2014-06-25 | 北京国双科技有限公司 | Data processing method and device for Internet user behavior analysis |
| CN103873583A (en) * | 2014-03-24 | 2014-06-18 | 北京聚思信息咨询有限公司 | Method and system for analyzing behaviors of internet users based on cloud platform |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018149078A1 (en) * | 2017-02-16 | 2018-08-23 | 平安科技（深圳）有限公司 | Data processing method, apparatus and device, and computer readable storage medium |
| CN108874798A (en) * | 2017-05-09 | 2018-11-23 | 北京京东尚科信息技术有限公司 | A kind of big data sort method and system |
| CN109299079A (en) * | 2018-09-11 | 2019-02-01 | 南京朝焱智能科技有限公司 | A kind of high-speed data library design method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109815267A (en) | The branch mailbox optimization method and system, storage medium and terminal of feature in data modeling | |
| US9558852B2 (en) | Method and apparatus for defect repair in NAND memory device | |
| CN108388509B (en) | Software testing method, computer readable storage medium and terminal equipment | |
| CN113807046B (en) | Test excitation optimization regression verification method, system and medium | |
| CN107562851B (en) | Data updating method and device and electronic equipment | |
| CN107273195A (en) | A kind of batch processing method of big data, device and computer system | |
| CN107678972B (en) | Test case evaluation method and related device | |
| CN104516828A (en) | Method and device for removing caching data | |
| US11934696B2 (en) | Machine learning assisted quality of service (QoS) for solid state drives | |
| CN107229414A (en) | Memory space recovery method and device | |
| CN106897342A (en) | A kind of data verification method and equipment | |
| CN109885310A (en) | A kind of method and device reducing mobile phone games Shader module EMS memory occupation | |
| CN106202374A (en) | A kind of data processing method and device | |
| CN109033365B (en) | Data processing method and related equipment | |
| CN106294128B (en) | A kind of automated testing method and device exporting report data | |
| CN112466378A (en) | Solid state disk operation error correction method and device and related components | |
| CN110084476A (en) | Case method of adjustment, device, computer equipment and storage medium | |
| CN104778088A (en) | Method and system for optimizing parallel I/O (input/output) by reducing inter-progress communication expense | |
| CN108399266A (en) | Data pick-up method, apparatus, electronic equipment and computer readable storage medium | |
| CN117033181A (en) | Method, device and equipment for generating test cases | |
| CN107104829B (en) | Physical equipment matching distribution method and device based on network topology data | |
| CN105353982B (en) | A kind of data access processing method and device based on circulation array | |
| CN109522565A (en) | A kind of verification method, device and computer readable storage medium | |
| CN106648550B (en) | Method and device for concurrently executing tasks | |
| CN111143177B (en) | Method, system, device and storage medium for collecting RMF III data of IBM host |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161207 |
|
| RJ01 | Rejection of invention patent application after publication |