CN103034650B - A kind of data handling system and method - Google Patents
A kind of data handling system and method Download PDFInfo
- Publication number
- CN103034650B CN103034650B CN201110300725.9A CN201110300725A CN103034650B CN 103034650 B CN103034650 B CN 103034650B CN 201110300725 A CN201110300725 A CN 201110300725A CN 103034650 B CN103034650 B CN 103034650B
- Authority
- CN
- China
- Prior art keywords
- data
- behavior
- user
- result
- user behavior
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data handling system and method.Described system comprises: data acquisition module, for the data of acquisition are sent to resolution system; Resolution system parses user behavior data and non-user behavioral data from reception data; User behavior data stores with tree structure by user behavior data storehouse, obtains user behavior data tree; Non-user behavioral data stores by its structure attribute by network data base; Key assignments cache module to preserve in above-mentioned database preserve the inquiry key assignments of data; Data cache module preserves the data of inquiry key assignments and the correspondence of having inquired about; Enquiry module is resolved and search inquiry key assignments in key assignments cache module, does not exist, returns and illegally inquire about prompting, exists and then inquires about in data cache module and return to applications, or inquire about in described database.The technical program, improves the computing power of data handling system, and this system is easy to expansion, and automaticity is high, is more conducive to inquiry.
Description
Technical field
The present invention relates to Internet technical field, particularly relate to a kind of data handling system and method.
Background technology
In the epoch of current information explosion, usually need to carry out analyzing and processing to mass data.The data analysis of such as, large-scale website in internet, and the data calculating etc. in scientific research.
General Data Analysis Services, uses the relevant database of high configuration to carry out the storage of data, relies on the computing power of database itself to complete the cleaning of data.When this mode is owing to relying on relevant database, there is following shortcoming: system loading is large, easily causes the bottleneck of system, be not easy to upgrading and expansion; Poor for data computing power more than TB level; The carrying cost of raw data is high, is unfavorable for utilizing and migration; Intellectualized operation is few, and artificial participation is high; And in relation database table, between each bar record, relevance is poor, and namely multiple behavior needs of same user store as many records, makes the Relationship Comparison between user and behavior loose, be unfavorable for the situation of the multiple behavior of analytic statistics, be unfavorable for inquiring about single user behavior.
Summary of the invention
The invention provides a kind of data handling system, the computing power of this data handling system be strong, be easy to expansion, automaticity high, be beneficial to inquiry.
Present invention also offers a kind of data processing method, improve the computing power of data handling system, and make this system be easy to expansion, automaticity is high, is more conducive to inquiry.
For achieving the above object, technical scheme of the present invention is achieved in that
The invention discloses a kind of data handling system, this system comprises: data acquisition module, resolution system, user behavior data storehouse, network data base, data cache module, key assignments cache module and enquiry module;
Data acquisition module, for receiving the data of first kind peripheral system real time propelling movement, and monitors Equations of The Second Kind peripheral system, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data; Send the data to resolution system;
Resolution system, for parsing user behavior data and non-user behavioral data from received data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of described network data base;
User behavior data storehouse, for being stored with tree structure by triangular to described user, behavior and result corresponding relation, obtains user behavior data tree; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node;
Network data base, for storing described non-user behavioral data by described structure attribute;
Key assignments cache module, for preserve in user behavior data storehouse and network data base preserve the inquiry key assignments of data;
Data cache module, for preserving the data of inquiry key assignments and the correspondence of having inquired about;
Enquiry module, for receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments cache module and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data cache module, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse and network data base, the data inquired are returned to applications.
The invention also discloses a kind of data processing method, the method comprises:
Receive the data of first kind peripheral system real time propelling movement, and Equations of The Second Kind peripheral system is monitored, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data;
User behavior data and non-user behavioral data is parsed from data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of described network data base;
Triangular to described user, behavior and result corresponding relation is stored with tree structure, obtains user behavior data tree, and be saved in user behavior data storehouse; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node;
Described non-user behavioral data is stored in network data base by described structure attribute;
In key assignments buffer memory in cache user behavior database and network data base preserve the inquiry key assignments of data and the data of the inquiry key assignments that buffer memory had been inquired about in data buffer storage and correspondence;
When receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments buffer memory and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data buffer storage, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse and network data base, the data inquired are returned to applications.
From above-mentioned, data handling system of the present invention comprises: data acquisition module, for the data of acquisition are sent to resolution system; Resolution system parses user behavior data and non-user behavioral data from reception data; User behavior data stores with tree structure by user behavior data storehouse, obtains user behavior data tree; Non-user behavioral data stores by its structure attribute by network data base; Key assignments cache module to preserve in above-mentioned database preserve the inquiry key assignments of data; Data cache module preserves the data of inquiry key assignments and the correspondence of having inquired about; Enquiry module is resolved and search inquiry key assignments in key assignments cache module, does not exist, returns and illegally inquire about prompting, exists and then inquires about in data cache module and return to applications, or inquire about in described database.The technical program, improves the computing power of data handling system, and this system is easy to expansion, and automaticity is high, is more conducive to inquiry.
Accompanying drawing explanation
Fig. 1 is the schematic diagram of the Data Analysis Services system improved of the prior art;
Fig. 2 is the composition structural representation of a kind of data handling system in the embodiment of the present invention;
Fig. 3 is the schematic diagram of a data handling system, its peripheral system and data flow in the present invention;
Fig. 4 is the process flow diagram of a kind of data processing method in the present invention.
Embodiment
In enforcement process of the present invention, inventor has carried out systematic study for Data Analysis Services technical solution, for Data Analysis Services scheme, distributed cloud computing can be realized by adopting cheap PC, and use HIVE to be used for the operator scheme of simulative relation type database, specifically as shown in Figure 1.
Fig. 1 is a kind of schematic diagram of Data Analysis Services system of improvement.As shown in Figure 1:
1, the data received are transferred to inside the platform of hadoop by the timing of data access platform;
2, synchronizing information data be transmitted is to the application system of data platform.Adopt the protoBuffer communication modes notification data platform of google;
3, the application system of data platform is according to receiving after information reading information from dispatching system, judges whether that unlatching is called Hive statement and calculated, and the order called;
4, Hive system reads metadata according to the information that receives and associates with the data of hadoop platform from mysql relational database;
5, Hive reads the data of calculating hadoop platform storage and generates destination file;
6, Hive being calculated the destination file generated imports in mysql database;
7, from MyS QL database, extracting part divided data is put in memcache;
8, data exhibiting platform reads data exhibiting from memcache;
9, data exhibiting platform reads data exhibiting from MyS QL database.
Scheme shown in Fig. 1, although had certain improvement, the reason of thinking set, has inherited existing operator scheme, result in extendability difference; Data for real-time do not do special process, cause scheme to be only applicable to the certain fields of data mining business; Data query poor-performing, system automation degree is inadequate.
Give a kind of brand-new data handling system in the present invention, to overcome the defect that existing system exists for this reason.
In order to make the object, technical solutions and advantages of the present invention clearly, describe the present invention below in conjunction with the drawings and specific embodiments.
Fig. 2 is the composition structural representation of a kind of data handling system in the embodiment of the present invention.As shown in Figure 2, this system comprises: data routing module 201, data acquisition module 202, resolution system 203, user behavior data storehouse 204, network data base 205, data cache module 206, key assignments cache module 207 and enquiry module 208; Wherein,
Data acquisition module 202, for receiving the data of first kind peripheral system real time propelling movement, and monitors Equations of The Second Kind peripheral system, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data; Send the data to resolution system 203;
Specifically, data acquisition module 202 is the data being received first kind peripheral system real time propelling movement by data routing module 201, and by data routing module 201 from Equations of The Second Kind peripheral system active obtaining user behavior data;
Data routing module 201, sends data acquisition module 202 to after converting the data from first kind peripheral system and Equations of The Second Kind peripheral system to meet notebook data disposal system data;
Resolution system 203, for parsing user behavior data and non-user behavioral data from received data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of described network data base;
Wherein, comprise user behavior data in the data of first kind peripheral system real time propelling movement, the data layout of user behavior data is the data of " user, behavior and result " form; And be various from the data layout that Equations of The Second Kind peripheral system obtains, resolution system 203 needs to resolve the user behavior data of multiple format, obtains the data of " user, behavior and result " such consolidation form.
In one embodiment of the invention, described resolution system 203 is Hadoop group system, adopts greenplum account form.
User behavior data storehouse 204, for being stored with tree structure by triangular to described user, behavior and result corresponding relation, obtains user behavior data tree; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node;
Network data base 205, for storing described non-user behavioral data by described structure attribute;
Key assignments cache module 207, for preserve in user behavior data storehouse and network data base preserve the inquiry key assignments of data;
Data cache module 206, for preserving the data of inquiry key assignments and the correspondence of having inquired about;
Enquiry module 208, for receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments cache module 207 and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data cache module 206, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse 204 and network data base 205, the data inquired are returned to applications.
In the system shown in Fig. 2,
Described data acquisition module 202, receives the data of first kind peripheral system real time propelling movement by service interface mode;
Described data acquisition module 202, for monitoring, when these system idles, from its active obtaining data application log system, application system backup library and network crawler system.
As shown in Figure 2, this system also comprises data analysis module 209, for traveling through the structure attribute of the user behavior data tree in described user behavior data storehouse 204 and/or the non-user behavioral data in requester network database 205, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis is calculated.
Described data analysis mould 209 pieces comprises: user behavior quantity statistics unit 2091, for traveling through described user behavior data tree, inquire about and locate result corresponding to each behavior, adding up the quantity of the corresponding result of each behavior, and setting up behavior, corresponding relation between result and quantity;
Described user behavior data storehouse 204 also comprises: result storage unit, for behavior, corresponding relation between result and quantity being stored with tree structure, obtains user behavior quantity tree; Wherein, " behavior " is root node, and " result " is the branch node under root node, and " quantity " is the branch node under " result " node.Do not draw the result storage unit in user behavior data storehouse 204 in fig. 2.
In the system shown in Fig. 2,
Described data analysis module 209 also comprises: query analysis unit 2092, for when described enquiry module 208 does not inquire the data of described inquiry key assignments and correspondence in data buffer storage 206, the inquiry key assignments obtained is resolved according to described enquiry module 208, travel through the structure attribute of described user behavior data tree and/or inquiry non-user behavioral data, location preanalysis data; According to described inquiry request, analytical calculation is carried out to described preanalysis data; Analysis result is stored in user behavior data storehouse 204 and/or network data base 205; So that described enquiry module 208 is inquired about and is obtained Query Result and return to applications in described user behavior data storehouse 204 and/or network data base 205.
In the system shown in Fig. 2, described data analysis module 209, specifically comprise multiple data analysis submodule 2093, be arranged on multiple equipment in a distributed way, the structure attribute of each data analysis submodule 2093 for adopting distributed computing to travel through described user behavior data tree and/or inquiry non-user behavioral data, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis is calculated;
And/or described user behavior data storehouse 204 in a distributed manner storage mode stores data.
In one embodiment of the invention, in application such as to the analysis that the user access activity of large-scale website is added up, the data handling system provided in the present invention: service interface can be called to obtain real time data, application log system, application system backup library and network crawler system are monitored, when it is idle, its data of active obtaining.Resolution system adopts Hadoop cluster, and adopts greenplun to calculate.User behavior data storehouse adopts non-relational NOSQL database, and network data base adopts relevant database.Then such system can be as shown in Figure 3.
Fig. 3 is the schematic diagram of a data handling system, its peripheral system and data flow in the present invention.As shown in Figure 3, data can be obtained in the multiple data sources such as service interface, application daily record, backup library and web crawlers, and after the process such as data route, Hadoop cluster/greenplum calculate, treated data are placed in NOSQL database and relevant database respectively according to its type, and set up key assignments buffer memory and data buffer storage.SQL router resolves this request content after receiving applications request (as user inquires about the statistics etc. of certain index), and in key assignments buffer memory, inquire about whether there is this type of inquiry, if not, returns illegal inquiry.If existed, continue in database or data buffer storage, inquire about corresponding data (the SQL statement inquiry of such as standard).
In the system as shown in fig. 3:
1, except being thought by the mode of original each system propelling data, adding service interface and calling, application daily record, each application system backup library, the data source of the multiple channels such as web crawlers.Wherein, for application daily record, each application system backup library, the resource of the application server of web crawlers etc. is monitored, only have and just carry out the synchronous of data when application server is in idle, which avoid fighting for system resource between application, improve system availability; Call acquisition real time data by service interface, realize supporting the process of real time data, be data more accurately, more timely.
2, peripheral data is changed into system data available by data routing module.
3, after data enter Hadoop cluster, greenplum is used to carry out calculating faster than hive or the mapReduce speed of existing scheme 3-4 times.
A, by calculate the data acquisition tree structure of high scalability is put in NOSQL database, this scheme is compared to the scheme of existing employing relevant database, that does not show due to NOSQL database remembers with gratitude, do not need to be pre-created field, so rebuild table when there be new attribute to add fashionable needs, enhance extendability.And provide the function of User Defined attribute, make system more intelligent.
B, relevant database is entered for common counter result of calculation.
4, the inquiry key assignments that likely occurs of buffer memory is in key assignments buffer memory.The pressure that use the method can be avoided inquiring about non-existent data due to applications and be caused data handling system.And the data that buffer memory had been inquired about in data buffer storage, this reduce the pressure of database, and more than 10 times to be exceeded than the speed of data query from database, thus ensure that the support that the concurrent figure of height is shown.
5, because system have employed SQL router, this SQL router externally applies open interface, for receiving the querying condition of user, after SQL router receives the querying condition of user, the condition can resolving applications self-defined inquiry obtains inquiring about key assignments, first inquire about in key assignments buffer memory and whether there is this inquiry key assignments, if there is no directly return and illegally inquire about prompting, if existed, then in data buffer storage, realize concrete inquiry (in like manner, if do not find in data buffer storage, inquire about in NOSQL database/relevant database again), and Query Result is fed back to user.The pattern of the passive exploitation of demand is proposed relative to existing user, more intelligent, have more dirigibility.
By above-mentioned visible in data handling system of the present invention, owing to adopting tree construction to manage user behavior data, and data cache module is adopted to store the content of having inquired about, and, adopt greenplum account form, substantially increase arithmetic speed.Comprehensive above-mentioned many reasons, the present invention can support that querying condition is arranged flexibly, realizes quick position search according to querying condition, and calculates the Query Result obtaining and need where necessary at any time, and supported inquiry mode is more flexible.Solving in prior art only can the analysis result that sets of inquiry system, and inflexible problem, improves Consumer's Experience and application scenarios further.
Fig. 4 is the process flow diagram of a kind of data processing method in the present invention.As shown in Figure 4, the method comprises:
401, receive the data of first kind peripheral system real time propelling movement, and Equations of The Second Kind peripheral system is monitored, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data.
402, from data, parse user behavior data and non-user behavioral data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of described network data base.
403, triangular to described user, behavior and result corresponding relation is stored with tree structure, obtains user behavior data tree, and be saved in user behavior data storehouse; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node.
404, described non-user behavioral data is stored in network data base by described structure attribute.
405, in key assignments buffer memory in cache user behavior database and network data base preserve the inquiry key assignments of data and the data of the inquiry key assignments that buffer memory had been inquired about in data buffer storage and correspondence.
406, when receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments buffer memory and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data buffer storage, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse and network data base, the data inquired are returned to applications.
In the above-mentioned methods,
The data of described reception first kind peripheral system real time propelling movement comprise: the data being received first kind peripheral system real time propelling movement by service interface mode;
Described Equations of The Second Kind peripheral system comprises: application log system, application system backup library and network crawler system.
The method comprises further: the structure attribute traveling through described user behavior data tree and/or inquiry non-user behavioral data, inquires about and locates preanalysis data, calculate described preanalysis data analysis according to analysis demand;
The described user behavior data tree of described traversal, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis being calculated and comprises:
Travel through described user behavior data tree, inquire about and locate result corresponding to each behavior, adding up the quantity of the corresponding result of each behavior, and setting up behavior, corresponding relation between result and quantity, behavior, corresponding relation between result and quantity are stored with tree structure, obtains user behavior quantity tree; Wherein, " behavior " is root node, and " result " is the branch node under root node, and " quantity " is the branch node under " result " node.
In sum, data handling system of the present invention comprises: data acquisition module, for the data of acquisition are sent to resolution system; Resolution system parses user behavior data and non-user behavioral data from reception data; User behavior data stores with tree structure by user behavior data storehouse, obtains user behavior data tree; Non-user behavioral data stores by its structure attribute by network data base; Key assignments cache module to preserve in above-mentioned database preserve the inquiry key assignments of data; Data cache module preserves the data of inquiry key assignments and the correspondence of having inquired about; Enquiry module is resolved and search inquiry key assignments in key assignments cache module, does not exist, returns and illegally inquire about prompting, exists and then inquires about in data cache module and return to applications, or inquire about in described database.The technical program, improves the computing power of data handling system, and this system is easy to expansion, and automaticity is high, is more conducive to inquiry.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within the scope of protection of the invention.
Claims (10)
1. a data handling system, is characterized in that, this system comprises: data acquisition module, resolution system, user behavior data storehouse, network data base, data cache module, key assignments cache module and enquiry module;
Data acquisition module, for receiving the data of first kind peripheral system real time propelling movement, and monitors Equations of The Second Kind peripheral system, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data; Send the data to resolution system;
Resolution system, for parsing user behavior data and non-user behavioral data from received data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of described network data base;
User behavior data storehouse, for being stored with tree structure by triangular to described user, behavior and result corresponding relation, obtains user behavior data tree; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node;
Network data base, for storing described non-user behavioral data by described structure attribute;
Key assignments cache module, for preserve in user behavior data storehouse and network data base preserve the inquiry key assignments of data;
Data cache module, for preserving the data of inquiry key assignments and the correspondence of having inquired about;
Enquiry module, for receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments cache module and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data cache module, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse and network data base, the data inquired are returned to applications.
2. system according to claim 1, is characterized in that, this system comprises further: data routing module;
Described data acquisition module, for receiving the data of first kind peripheral system real time propelling movement, and by data routing module from Equations of The Second Kind peripheral system active obtaining user behavior data by data routing module;
Data routing module, sends data acquisition module to after converting the data from first kind peripheral system and Equations of The Second Kind peripheral system to meet notebook data disposal system data.
3. system according to claim 1 and 2, is characterized in that,
Described data acquisition module, receives the data of first kind peripheral system real time propelling movement by service interface mode;
Described data acquisition module, for monitoring, when these system idles, from its active obtaining data application log system, application system backup library and network crawler system.
4. system according to claim 1 and 2, it is characterized in that, described system also comprises data analysis module, for traveling through the structure attribute of described user behavior data tree and/or inquiry non-user behavioral data, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis is calculated.
5. system according to claim 4, is characterized in that,
Described data analysis module comprises: user behavior quantity statistics unit, for traveling through described user behavior data tree, inquire about and locating result corresponding to each behavior, adds up the quantity of each behavior correspondence result, and sets up behavior, corresponding relation between result and quantity;
Described user behavior data storehouse also comprises:
Result storage unit, for behavior, corresponding relation between result and quantity being stored with tree structure, obtains user behavior quantity tree; Wherein, " behavior " is root node, and " result " is the branch node under root node, and " quantity " is the branch node under " result " node.
6. system according to claim 4, is characterized in that,
Described data analysis module also comprises: query analysis unit, for when described enquiry module does not inquire the data of described inquiry key assignments and correspondence in data buffer storage, the inquiry key assignments obtained is resolved according to described enquiry module, travel through the structure attribute of described user behavior data tree and/or inquiry non-user behavioral data, location preanalysis data; According to described inquiry request, analytical calculation is carried out to described preanalysis data; Analysis result is stored in user behavior data storehouse and/or network data base; So that described enquiry module is inquired about and is obtained Query Result and return to applications in described user behavior data storehouse and/or network data base.
7. system according to claim 4, is characterized in that,
Described data analysis module, specifically comprise multiple data analysis submodule, be arranged on multiple equipment in a distributed way, the structure attribute of each data analysis submodule for adopting distributed computing to travel through described user behavior data tree and/or inquiry non-user behavioral data, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis is calculated;
And/or described user behavior data storehouse in a distributed manner storage mode stores data.
8. a data processing method, is characterized in that, the method comprises:
Receive the data of first kind peripheral system real time propelling movement, and Equations of The Second Kind peripheral system is monitored, when Equations of The Second Kind peripheral system is idle, from Equations of The Second Kind peripheral system active obtaining data;
User behavior data and non-user behavioral data is parsed from data; Resolve from described user behavior data and obtain user, behavior and result data; And set up the triangular corresponding relation of user, behavior and result; Wherein result is concrete content of the act; And described non-user behavioral data is resolved according to the structure attribute of network data base;
Triangular to described user, behavior and result corresponding relation is stored with tree structure, obtains user behavior data tree, and be saved in user behavior data storehouse; Wherein, " user " is root node, and " behavior " is the branch node under root node, and " result " is the branch node under " behavior " node;
Described non-user behavioral data is stored in network data base by described structure attribute;
In key assignments buffer memory in cache user behavior database and network data base preserve the inquiry key assignments of data and the data of the inquiry key assignments that buffer memory had been inquired about in data buffer storage and correspondence;
When receiving the inquiry request of applications, resolve the inquiry key assignments in inquiry request, inquire about in key assignments buffer memory and whether there is this inquiry key assignments, if there is no then externally application returns the prompting of illegal inquiry, if existed, in data buffer storage, inquire about the data that whether there is this inquiry key assignments and correspondence further, that the corresponding data in data buffer storage is returned to applications, otherwise inquire about in user behavior data storehouse and network data base, the data inquired are returned to applications.
9. method according to claim 8, its spy is,
The data of described reception first kind peripheral system real time propelling movement comprise: the data being received first kind peripheral system real time propelling movement by service interface mode;
Described Equations of The Second Kind peripheral system comprises: application log system, application system backup library and network crawler system.
10. method according to claim 8 or claim 9, it is characterized in that, the method comprises further: the structure attribute traveling through described user behavior data tree and/or inquiry non-user behavioral data, inquires about and locates preanalysis data, calculate described preanalysis data analysis according to analysis demand;
The described user behavior data tree of described traversal, inquire about according to analysis demand and locate preanalysis data, described preanalysis data analysis being calculated and comprises:
Travel through described user behavior data tree, inquire about and locate result corresponding to each behavior, adding up the quantity of the corresponding result of each behavior, and setting up behavior, corresponding relation between result and quantity, behavior, corresponding relation between result and quantity are stored with tree structure, obtains user behavior quantity tree; Wherein, " behavior " is root node, and " result " is the branch node under root node, and " quantity " is the branch node under " result " node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110300725.9A CN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110300725.9A CN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103034650A CN103034650A (en) | 2013-04-10 |
| CN103034650B true CN103034650B (en) | 2015-10-28 |
Family
ID=48021552
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110300725.9A Active CN103034650B (en) | 2011-09-29 | 2011-09-29 | A kind of data handling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103034650B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346345B (en) * | 2013-07-24 | 2019-03-26 | 上海中兴软件有限责任公司 | The storage method and device of data |
| CN103514273B (en) * | 2013-09-17 | 2016-08-17 | 宁波东冠科技有限公司 | DAM control system and the data processing method of this system |
| CN106484691B (en) * | 2015-08-24 | 2019-12-10 | 阿里巴巴集团控股有限公司 | data storage method and device of mobile terminal |
| CN105760548A (en) * | 2016-03-21 | 2016-07-13 | 武汉烽火众智数字技术有限责任公司 | Vehicle first appearance analysis method and system based on big data cross-domain comparison |
| CN107798037A (en) * | 2017-04-26 | 2018-03-13 | 平安科技（深圳）有限公司 | The acquisition methods and server of user characteristic data |
| CN112182340B (en) * | 2019-07-01 | 2024-06-07 | 中国移动通信集团浙江有限公司 | Internet of things information query method, subscription method, device and electronic equipment |
| CN114020850B (en) * | 2022-01-05 | 2022-04-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170426A (en) * | 2006-10-25 | 2008-04-30 | 马永利 | Personalized content distribution scheme based on user behavior (habit) analysis |
| CN101409690A (en) * | 2008-11-26 | 2009-04-15 | 北京学之途网络科技有限公司 | Method and system for obtaining internet user behaviors |
| CN102111453A (en) * | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101020206B1 (en) * | 2008-06-16 | 2011-03-08 | 성균관대학교산학협력단 | Method for recommendation to user and storage medium storing program for realizing the method |
-
2011
- 2011-09-29 CN CN201110300725.9A patent/CN103034650B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170426A (en) * | 2006-10-25 | 2008-04-30 | 马永利 | Personalized content distribution scheme based on user behavior (habit) analysis |
| CN101409690A (en) * | 2008-11-26 | 2009-04-15 | 北京学之途网络科技有限公司 | Method and system for obtaining internet user behaviors |
| CN102111453A (en) * | 2011-03-04 | 2011-06-29 | 创博亚太科技（山东）有限公司 | Method and system for extracting Internet user network behaviors |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103034650A (en) | 2013-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103034650B (en) | A kind of data handling system and method | |
| JP6617117B2 (en) | Scalable analysis platform for semi-structured data | |
| Gupta et al. | Cloud computing and big data analytics: what is new from databases perspective? | |
| US8918363B2 (en) | Data processing service | |
| Cambazoglu et al. | Scalability challenges in web search engines | |
| US20180285418A1 (en) | Executing queries for structured data and not-structured data | |
| CN103631870B (en) | System and method used for large-scale distributed data processing | |
| CN103678665A (en) | Heterogeneous large data integration method and system based on data warehouses | |
| CN108536778B (en) | Data application sharing platform and method | |
| CN104239572A (en) | System and method for achieving metadata analysis based on distributed cache | |
| CN103036921B (en) | A kind of user behavior analysis system and method | |
| US9229961B2 (en) | Database management delete efficiency | |
| CN104252536A (en) | Hbase-based internet log data inquiring method and device | |
| CN107343021A (en) | A kind of Log Administration System based on big data applied in state's net cloud | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| US11853301B1 (en) | Sharing compiled code for executing queries across query engines | |
| CN103646051A (en) | Big-data parallel processing system and method based on column storage | |
| CN113641862A (en) | Method and system for integrating multi-source heterogeneous data based on uniform access distribution | |
| Mostajabi et al. | A systematic review of data models for the big data problem | |
| Zou et al. | From a stream of relational queries to distributed stream processing | |
| US20140258264A1 (en) | Management of searches in a database system | |
| Zhou et al. | Sfmapreduce: An optimized mapreduce framework for small files | |
| Arputhamary et al. | A review on big data integration | |
| KR101828522B1 (en) | System of Parallel Distributed Processing System for Heterogeneous Data Processing | |
| CN108804502A (en) | Big data inquiry system, method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 810, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building A block 5 layer Patentee before: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. |