CN108959337A - Big data acquisition methods, device, equipment and storage medium - Google Patents
Big data acquisition methods, device, equipment and storage medium Download PDFInfo
- Publication number
- CN108959337A CN108959337A CN201810242151.6A CN201810242151A CN108959337A CN 108959337 A CN108959337 A CN 108959337A CN 201810242151 A CN201810242151 A CN 201810242151A CN 108959337 A CN108959337 A CN 108959337A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- memory module
- attribute
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 239000000284 extract Substances 0.000 claims abstract description 17
- 238000004590 computer program Methods 0.000 claims description 15
- 238000013500 data storage Methods 0.000 claims description 5
- 238000013075 data extraction Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000013524 data verification Methods 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to big data technical field, in particular to a kind of big data acquisition methods, device, equipment and storage medium.Big data acquisition methods include the bottom data for obtaining data acquisition cluster acquisition；Owning user in User Information Database is matched according to the service attribute of the bottom data, and by the bottom data classified storage in corresponding memory module；Obtain user access request, the user access request is parsed to obtain parsing data and send the parsing data and user's checking request to the User Information Database, when the parsing data pass through verifying, extracts the data in corresponding memory module and shown.Bottom data can be quickly respectively stored in corresponding memory module by technical solution of the present invention by different usage types, and externally provided unified access interface and carried out data transmission.
Description
Technical field
The present invention relates to big data technical fields more particularly to a kind of big data acquisition methods, device, equipment and storage to be situated between
Matter.
Background technique
Existing big data is exported to interconnected system using mainly relevant database is exported to by Sqoop, for outside
System uses.Sqoop has imported and exported important work as the bridge between Hadoop and traditional database, for data
With.Sqoop is the abbreviation of SQL-to-Hadoop, be mainly used for Hadoop (Hive) and traditional database (mysql,
Postgresql... the transmitting that data are carried out between), the data in a relevant database can be led and enter Hadoop's
In HDFS, the data of HDFS can also be led and be entered in relevant database.Application requires to export data Sqoop each time
Appointing system will increase the workload of operator and influence operation accuracy rate when data volume is big and number of operations more.Cause
This, big data acquisition methods do not have the function of high-efficiency transfer and accurate delivery in the prior art, so that working efficiency and standard
True rate is not high.
Summary of the invention
The purpose of the present invention is to provide a kind of big data acquisition methods, device, equipment and storage medium, can be realized by
Data storage is respectively stored in corresponding memory module by different usage types, is externally provided unified access interface and is carried out data biography
It is defeated, improve the transmission rate and transmission data accuracy of data acquisition.
The invention is realized in this way first aspect present invention provides a kind of big data acquisition methods, comprising:
Cluster, which is acquired, by data acquires bottom data；
Owning user in User Information Database is matched according to the service attribute of the bottom data, and by the bottom number
According to classified storage in corresponding memory module；
User access request is obtained, the user access request is parsed and obtains parsing data and to the user information data
Library sends the parsing data and user's checking request, when the parsing data pass through verifying, extracts in corresponding memory module
Data and shown.
Second aspect of the present invention provides a kind of big data acquisition device, and the big data acquisition device includes:
Data acquisition module, for acquiring the bottom data of cluster acquisition by data；
Data Matching memory module, for matching institute in User Information Database according to the service attribute of the bottom data
Belong to user, and by the bottom data classified storage in corresponding memory module；
Data extraction module, for obtaining user access request, parsing the user access request and believing to the user
It ceases database and sends user's checking request, User Token is obtained when being verified and correspondence is extracted according to the User Token and is deposited
It stores up data in module and is shown.
Third aspect present invention provides a kind of terminal device, including memory, processor and is stored in the memory
In and the computer program that can run on the processor, the processor realize such as this hair when executing the computer program
The step of bright first aspect the method.
Fourth aspect present invention provides a kind of computer readable storage medium, and the computer-readable recording medium storage has
Computer program, when the computer program is executed by processor realize as described in the first aspect of the invention method the step of.
The present invention provides a kind of big data acquisition methods, device, equipment and storage medium, passes through data and acquires cluster acquisition
Bottom data is obtained, the owning user in User Information Database is matched according to the service attribute of bottom data, and by data
Usage type is separately stored in corresponding memory module, and when getting user access request, parsing user access request is obtained
Must parse data and to User Information Database send parsing data and user's checking request, when parse data verification by when mention
It takes the data in the identity attribute file of family in corresponding memory module and is shown, realized data by usage type
Difference is respectively stored in corresponding memory module, defines bottom data information, saves user and bottom data owner relationship, right
The unified access interface of outer offer, external system is verified by User Information Database to be extracted corresponding bottom data and returns to display,
The workload for reducing operator, improves message transmission rate and accuracy.
Detailed description of the invention
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to embodiment or description of the prior art
Needed in attached drawing be briefly described, it should be apparent that, the accompanying drawings in the following description is only of the invention some
Embodiment for those of ordinary skill in the art without any creative labor, can also be according to these
Attached drawing obtains other attached drawings.
Fig. 1 is a kind of flow chart for big data acquisition methods that an embodiment of the present invention provides；
Fig. 2 is the specific flow chart of the step S20 in a kind of big data acquisition methods that an embodiment of the present invention provides；
Fig. 3 is the detailed process of the step S203 in a kind of big data acquisition methods that an embodiment of the present invention provides
Figure；
Fig. 4 is the specific flow chart of the step S30 in a kind of big data acquisition methods that an embodiment of the present invention provides；
Fig. 5 is a kind of structural schematic diagram for big data acquisition device that second embodiment of the invention provides；
Fig. 6 is the Data Matching memory module 32 in a kind of big data acquisition device that second embodiment of the invention provides
Structural schematic diagram；
Fig. 7 is the structural schematic diagram for the terminal device that third embodiment of the invention provides.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
In order to illustrate technical solution of the present invention, the following is a description of specific embodiments.
Embodiment 1
First embodiment of the invention provides a kind of big data acquisition methods, as shown in Figure 1, big data acquisition methods include:
Step S10 acquires cluster by data and acquires bottom data.
In step slo, data acquisition cluster refers to the data cluster of data in acquisition database, for example, data acquire
Cluster can be Spark Streaming cluster, and bottom data refers to various businesses data and the user stored in the database
Information data, for example, bottom data can be the inventory of foreground real-time statistic analysis, summarize data or based on the unique id of user
All detail user informations etc..Bottom data can also be divided into real time data and off-line data, and real time data refers to by data
The data that the database of acquisition cluster acquisition is written in real time, off-line data are that the database to go offline of data acquisition cluster acquisition is synchronous
Data.By the way that this step is arranged, may be implemented to carry out Quick Acquisition to the bottom data information in database.
Specifically, being first passed through in Spark Streaming cluster acquisition database before obtaining user access request
Bottom data, the data being written in acquisition database in real time, for example, user A has handled A business, database is in business personnel
Operation under typing user A handle the related data of A business, data acquisition cluster acquires the correlation that user A handles A business simultaneously
Data, if the personal information of user A includes at least the ID of user A, the password of user A, the data of A business include the number of A business
According to attribute and service attribute.Data acquire the data of offline synchronization in cluster acquisition database, for example, user A is in transacting business
Have changed the partial information of personal information afterwards, it, can be first synchronous with ETL data when the new individual data of database write access customer A
Tool pre-processes the new individual data of user A, then is obtained by acquiring cluster acquisition by data.
Step S20 matches owning user in User Information Database according to the service attribute of the bottom data, and by institute
Bottom data classified storage is stated in corresponding memory module.
In step S20, service attribute refers to business information associated by bottom data and user information, for example, business
Attribute includes data attribute and storage attribute, and data attribute includes user information data and data type, and storage attribute includes number
According to the corresponding relationship between type and storage location, User Information Database refers to the database of storage user related information, uses
Family relevant information includes at least the business handled of User ID, user password and user, and owning user, which refers to, to be possessed and bottom data
Service attribute identical services attribute user, owning user contains at least one user, is also possible to multiple users, bottom number
Refer to that the different classifications of the usage type by data, such as bottom data are segmented into statistical data, inventory number according to classified storage
Accordingly and key value data according to, total amount, memory module refers to the storage medium of the bottom data of storage acquisition, for example, depositing
Storage module can be Elasticsearch cluster, Hbase cluster or Redis cluster.By the way that this step is arranged, may be implemented
Bottom data information is defined, saves user and bottom data owner relationship, and store by data usage type different classifications.
Specifically, for example, getting data acquisition cluster collects the related data that user A handles A business, including user
The ID of A, the password of user A, A business service attribute, parsing obtain service attribute, match and possess in User Information Database
The user of identical services attribute, when successful match, it is determined that be the owning user of data, for example, in user identity attribute text
Matching user is carried out in part, is matched to user A and is also possessed A service attribute, confirmation user A is owning user, by related data point
At statistical data, listings data and summarize data, and is respectively stored in different storage mediums.
Step S30 obtains user access request, parses the user access request and obtains parsing data and to the user
Information database sends the parsing data and user's checking request, when the parsing data pass through verifying, extracts correspondence and deposits
It stores up the data in module and is shown.
In step s 30, user access request refers to that user obtains the access request of data, for example, access request can be with
It is the request of data that user is sent by web services, parsing data refer to the data that parsing user access request obtains, for example,
Data include User ID, password and request data type, and User Information Database refers to the database of storage user related information,
User related information includes at least the business and memory module that User ID, user password and user handle, and user's checking request is
Refer to verifying User ID whether correctly requested with password, memory module can be Elasticsearch cluster, Hbase cluster or
Person's Redis cluster.By the way that this step is arranged, it may be implemented to verify access user, quickly be mentioned from different memory modules
It takes the data of corresponding user's requested service and shows.
Specifically, for example, obtaining the access request " the handling situation of inquiry A business " of user A, parsing access request " is looked into
Ask A business and handle situation " ID of user A is obtained, the password and request data type of user A is A business, by the ID of user A,
The password and request data type of user A is sent to User Information Database request for A business and the request of the user's checking of generation
Verifying, as the ID and all identical password with the user A prestored in User Information Database, user's checking passes through, and obtains user
User A identity attribute file in information database extracts data according to the memory module of A business in user's A identity attribute file
And it is shown.
As an embodiment of the present embodiment, bottom data is acquired for acquiring cluster by data in step S10,
Include:
The business incremental data generated by Spark Streaming cluster monitor database, and obtain the business and increase
Measure data.
In this step, Spark Streaming cluster refers to the data acquisition cluster of acquisition database data, Spark
Streaming cluster can be acquired processing to real-time stream, and database, which refers to, stores various businesses data and user
The storage medium of information data, business incremental data refer to the data information that database is written under the operation of operator, example
Such as, business incremental data can be the new business data information that user handles, and is also possible to user and modifies personal information data letter
Breath.By the way that this step is arranged, may be implemented to acquire the bottom data information in database in real time.
Specifically, database typing user A under the operation of business personnel handles A for example, user A has handled A business
The related data of business, Spark Streaming cluster acquires the related data that user A handles A business in real time, such as user A
Personal information includes at least the ID of user A, and the password of user A, the data of A business include the data attribute and business category of A business
Property.Data acquire the data of offline synchronization in cluster acquisition database, for example, user A has changed personal money after transacting business
The partial information of material can be first with ETL data synchronization means to the new of user A when the new individual data of database write access customer A
Personal information is pre-processed, then by being obtained by the acquisition of Spark Streaming cluster.
The present invention provides a kind of big data acquisition methods, device, equipment and storage medium, passes through data and acquires cluster acquisition
Bottom data is obtained, the owning user in User Information Database is matched according to the service attribute of bottom data, and by data
Usage type is separately stored in corresponding memory module, and when getting user access request, parsing user access request is obtained
Must parse data and to User Information Database send parsing data and user's checking request, when parse data verification by when mention
It takes the data in the identity attribute file of family in corresponding memory module and is shown, realized data by usage type
Difference is respectively stored in corresponding memory module, defines bottom data information, saves user and bottom data owner relationship, right
The unified access interface of outer offer, external system is verified by User Information Database to be extracted corresponding bottom data and returns to display,
The workload for reducing operator, improves message transmission rate and accuracy.
As an embodiment of the present embodiment, as shown in Fig. 2, in step S20 according to the bottom data
Service attribute matches owning user in User Information Database, and by the bottom data classified storage in corresponding memory module
In, comprising:
Step S201, the service attribute for parsing the bottom data obtain data attribute and storage attribute, wherein the number
It include user information data and data type according to attribute, the storage attribute includes corresponding between data type and storage location
Relationship.
Step S202 updates the user identity in User Information Database according to the data attribute and the storage attribute
Property file, the user identity property file include User ID, password, data type information and memory module information.
Step S203 classifies the bottom data according to the data type, and will according to the storage attribute
The bottom data is stored in corresponding memory module.
In above-mentioned steps S201 into step S203, service attribute refers to business information associated by bottom data and user
Information, service attribute include data attribute and storage attribute, and data attribute includes user information data and data type, and storage belongs to
Property includes the corresponding relationship between data type and storage location, and user identity property file is depositing for multiple subscriber identity informations
File is stored up, user identity property file includes User ID, password, data type and memory module, updates user identity attribute
File refer to update the subscriber identity information of owning user being matched in user identity property file, data type information and
Corresponding memory module information, user ID data information include User ID and user password, and data type refers to bottom data
Type, for example, data type can be statistical data type, listings data type, summarize data type and key value
Data type, service data information include traffic data type and memory module, and memory module refers to the bottom number of storage acquisition
According to storage medium, for example, memory module can be Elasticsearch cluster, Hbase cluster or Redis cluster.Pass through
This step is set, may be implemented to define bottom data information, saves user and bottom data owner relationship, and use class by data
The storage of type different classifications.
As an embodiment of the present embodiment, as shown in figure 3, for will according to the data type in step S203
The bottom data is classified, and the bottom data is stored in corresponding memory module according to the storage attribute,
Include:
The bottom data is divided into statistical data, inventory according to the data type in the data attribute by step S2031.
Data, total amount are accordingly and key value data.
Step S2032. is by the statistical data, the listings data and described summarize data storage and arrives
In Elasticsearch cluster, and by key value data storage into Hbase cluster.
In above step S2031 and step S2032, data type refers to the usage type of data, according to data type
Statistical data, listings data, total amount can be splitted data into accordingly and key value data, statistical data refer to that user believes
The statistical data of breath, listings data refer to every business list data that user handles, summarize data and refer to user information data
With the data that summarize of business datum, key value data refer to user information data, such as based on all bright of the unique id of user
Thin user information, Elasticsearch cluster are storage statistical data, listings data and the memory module for summarizing data,
Hbase cluster is the memory module for storing key value data.
As an embodiment of the present embodiment, as shown in figure 4, for obtaining user access request in step S30, solution
The user access request is analysed to obtain parsing data and test to the User Information Database transmission parsing data and user
Card request is extracted the data in corresponding memory module and is shown when the parsing data pass through verifying, comprising:
Step S301. obtains user access request, parses the user access request and obtains User ID, password and number of request
According to type and user's checking request is generated, and sends the User ID to the User Information Database, the password, described ask
Data type and the user's checking is asked to request.
Step S302. includes the user according to the user's checking request detection to the user identity property file
When ID, the password and the request data type, determine that the parsing data pass through verifying, and extract the user identity
Data in the corresponding memory module of request data type described in property file are simultaneously shown.
In above step S301 and step S302, user access request refers to that user obtains the access request of data, example
Such as, access request can be the request of data that user is sent by web services, can pass through the web services and use in interface layer
Family information database provides unified interface, and web services provide the displaying of data for user, and industry is presented for user in a manner of Web
Business related content, web services provide interactive interface, receive the access request of user, user name and service request are sent to use
Family information database, while receiving the related data of return and showing user, parsing data refer to parsing user access request
Obtained data, for example, data include User ID, password and request data type, User Information Database refers to storage user
The database of identity attribute file, subscriber identity information property file refer to the storage file of multiple subscriber identity informations, user
Identity attribute file includes User ID, password, data type information and memory module information, and user's checking request refers to verifying
Whether User ID correctly requests with password, when user identity property file includes the User ID, the password and described
When request data type, i.e., verifying User ID and password are correct, determine that the parsing data by verifying, extract the user
The corresponding memory module information of request data type described in identity attribute file, memory module can be Elasticsearch
Cluster, Hbase cluster or Redis cluster, and extract data into the memory module and shown.By the way that this step is arranged
Suddenly, it may be implemented to verify access user, the data of corresponding user's requested service quickly extracted from different memory modules
And it is shown.
As an embodiment of the present embodiment, for, when the parsing data pass through verifying, being mentioned in step S302
It takes the data in corresponding memory module and is shown, comprising:
When memory module is that Redis is cached, is directly cached from the Redis and extract data and shown；
When memory module is not Redis caching, from the ElasticSearch cluster or the HBase cluster
It extracts data and is stored in Redis caching.
It, should according to the service attribute of data when data are written in real time by sparkstreaming in above step
Data are included into user under one's name, and are stored in different memory modules according to data type, when user is sent by web services
When request of data, input user name, password and service request type, by User Information Database rights management, when password just
The storage attribute that the corresponding data attribute of the user name and data are obtained when really, according to user name in subscriber identity information category
The storage location of its accessible data is searched in property file, and then is inquired to the storage location, the business datum
Storage location can be ElasticSearch cluster, HBase cluster or Redis caching, when the Redis buffer memory business
Called data directly is cached from Redis when data, when Redis caching does not store the business datum, from ElasticSearch collection
It called data and is buffered in redis in group and HBase cluster, result is exported and is returned.By the way that this step is arranged, may be implemented
To the data rapidly extracting inquired, low latency inquiry is realized.
Embodiment 2
Second embodiment of the invention provides a kind of big data acquisition device, as shown in figure 5, big data acquisition device 3 includes:
Data acquisition module 31, for obtaining the bottom data of data acquisition cluster acquisition；
Data Matching memory module 32, for being matched in User Information Database according to the service attribute of the bottom data
Owning user, and by the bottom data classified storage in corresponding memory module；
Data extraction module 33 parses the user access request and obtains parsing data for obtaining user access request
And the parsing data and user's checking request are sent to the User Information Database, when the parsing data pass through verifying
When, it extracts the data in corresponding memory module and is shown.
Further, the industry that data acquisition module 31 is generated especially by Spark Streaming cluster monitor database
Business incremental data, and obtain the business incremental data.
As shown in fig. 6, the Data Matching memory module 32 includes:
Parsing module 321, the service attribute for parsing the bottom data obtain data attribute and storage attribute,
In, the data attribute includes user information data and data type, and the storage attribute includes data type and storage location
Between corresponding relationship；
Update module 322, for being updated in User Information Database according to the data attribute and the storage attribute
User identity property file, the user identity property file include User ID, password, data type information and memory module
Information；
Classification storage module 323, for the bottom data to be classified according to the data type, and according to described
The bottom data is stored in corresponding memory module by storage attribute.
The classification storage module 323 is specifically used for:
The bottom data is divided into statistical data according to the data type in the data attribute, listings data, is summarized
Data and key value data；
The statistical data, the listings data and the data that summarize are stored into Elasticsearch cluster,
And by key value data storage into Hbase cluster.
The specific work process of module in above-mentioned big data acquisition device 3 can be obtained with reference to big data in previous embodiment 1
The corresponding process of method is taken, details are not described herein.
Embodiment 3
Third embodiment of the invention provides a computer readable storage medium, is stored on the computer readable storage medium
Computer program, the computer program realize one of above-described embodiment 1 big data acquisition methods, are when being executed by processor
It avoids repeating, which is not described herein again.Alternatively, the computer program realizes that one kind is big in above-described embodiment 2 when being executed by processor
The function of each module/unit in data acquisition facility, to avoid repeating, which is not described herein again.
Embodiment 4
Fig. 7 is the schematic diagram of terminal device 4 in the present embodiment.As shown in fig. 7, terminal device 4 includes processor 43, storage
Device 41 and it is stored in the computer program 42 that can be run in memory 41 and on processor 43.Processor 43 executes computer
Realize a kind of each step of big data acquisition methods in above-described embodiment 1 when program 42, such as step S10 shown in FIG. 1,
S20 and S30.Alternatively, processor 43 realizes that a kind of big data acquisition device is each in above-described embodiment 2 when executing computer program 42
Module/unit function, data acquisition module 31 as shown in Figure 5, Data Matching memory module 32 and data extraction module 33.
Computer program 42 can be divided into one or more module/units, one or more module/unit is deposited
Storage executes in memory 41, and by processor 43, to complete the present invention.One or more module/units can be can be complete
At 42 instruction segment of series of computation machine program of specific function, the instruction segment is for describing computer program 42 in terminal device 4
In implementation procedure.
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of big data acquisition methods, which is characterized in that the big data acquisition methods include:
Cluster, which is acquired, by data acquires bottom data；
Owning user in User Information Database is matched according to the service attribute of the bottom data, and the bottom data is divided
Class is stored in corresponding memory module；
User access request is obtained, the user access request is parsed and obtains parsing data and sent out to the User Information Database
It send the parsing data and user's checking to request, when the parsing data pass through verifying, extracts the number in corresponding memory module
According to and shown.
2. big data acquisition methods as described in claim 1, which is characterized in that described to acquire cluster acquisition bottom by data
Data, including；
The business incremental data generated by Spark Streaming cluster monitor database, and obtain the business incremental number
According to.
3. big data acquisition methods as described in claim 1, which is characterized in that according to the service attribute of the bottom data
With owning user in User Information Database, and by the bottom data classified storage in corresponding memory module, comprising:
The service attribute for parsing the bottom data obtains data attribute and storage attribute, wherein the data attribute includes using
Family information data and data type, the storage attribute include the corresponding relationship between data type and storage location；
The user identity property file in User Information Database is updated according to the data attribute and the storage attribute, it is described
User identity property file includes User ID, password, data type information and memory module information；
The bottom data is classified according to the data type, and is stored up the bottom data according to the storage attribute
There are in corresponding memory module.
4. big data acquisition methods as claimed in claim 3, which is characterized in that according to the data type by the bottom number
According to classifying, and the bottom data is stored in corresponding memory module according to the storage attribute, including；
The bottom data is divided into statistical data according to the data type in the data attribute, listings data, summarizes data
And key value data；
The statistical data, the listings data and the data that summarize are stored into Elasticsearch cluster, and will
The key value data storage is into Hbase cluster.
5. big data acquisition methods as described in claim 1, which is characterized in that obtain user access request, parse the use
Family access request obtains parsing data and sends the parsing data and user's checking request to the User Information Database, when
When the parsing data pass through verifying, extracts the data in corresponding memory module and is shown, comprising:
User access request is obtained, the user access request is parsed and obtains User ID, password and request data type and generate
User's checking request, and to the User Information Database send the User ID, the password, the request data type with
And the user's checking request；
According to the user's checking request detection to the user identity property file include the User ID, the password and
When the request data type, determine that the parsing data pass through verifying, and extract described in the user identity property file
Data in the corresponding memory module of request data type are simultaneously shown.
6. big data acquisition methods as claimed in claim 5, which is characterized in that extract institute in the user identity property file
It states the data in the corresponding memory module of request data type and is shown, further includes:
When memory module is that Redis is cached, is directly cached from the Redis and extract data and shown；
When memory module is not Redis caching, from the ElasticSearch cluster or the HBase cluster
It extracts data and is stored in the Redis caching.
7. a kind of big data acquisition device, which is characterized in that the big data acquisition device includes:
Data acquisition module acquires bottom data for acquiring cluster by data；
Data Matching memory module, for matching affiliated use in User Information Database according to the service attribute of the bottom data
Family, and by the bottom data classified storage in corresponding memory module；
Data extraction module parses the user access request and obtains parsing data and to institute for obtaining user access request
It states User Information Database and sends the parsing data and user's checking request, when the parsing data pass through verifying, extract
It corresponds to the data in memory module and is shown.
8. big data acquisition device as claimed in claim 7, which is characterized in that the data acquisition module especially by
The business incremental data that Spark Streaming cluster monitor database generates, and obtain the business incremental data.
9. a kind of terminal device, including memory, processor and storage are in the memory and can be on the processor
The computer program of operation, which is characterized in that the processor realizes such as claim 1 to 6 when executing the computer program
The step of any one the method.
10. a kind of computer readable storage medium, the computer-readable medium storage has computer program, which is characterized in that
It is realized when the computer program is executed by processor such as the step of any one of claim 1 to 6 the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810242151.6A CN108959337A (en) | 2018-03-22 | 2018-03-22 | Big data acquisition methods, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810242151.6A CN108959337A (en) | 2018-03-22 | 2018-03-22 | Big data acquisition methods, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108959337A true CN108959337A (en) | 2018-12-07 |
Family
ID=64495309
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810242151.6A Pending CN108959337A (en) | 2018-03-22 | 2018-03-22 | Big data acquisition methods, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108959337A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726205A (en) * | 2018-12-26 | 2019-05-07 | 国电南瑞科技股份有限公司 | A kind of power system security stability analysis data-storage system and storage method |
| CN110704445A (en) * | 2019-09-30 | 2020-01-17 | 石化盈科信息技术有限责任公司 | Industrial data lake system |
| CN110781238A (en) * | 2019-10-08 | 2020-02-11 | 中国建设银行股份有限公司 | Client view caching method and device based on combination of Redis and Hbase |
| CN111090882A (en) * | 2019-12-18 | 2020-05-01 | 北京浪潮数据技术有限公司 | Operation control method, device and equipment for redis database |
| CN111429137A (en) * | 2020-03-17 | 2020-07-17 | 深圳市随金科技有限公司 | Modular data acquisition method and device, server and storage medium |
| CN112380167A (en) * | 2020-11-17 | 2021-02-19 | 深圳市和讯华谷信息技术有限公司 | Batch data verification method and device, computer equipment and storage medium |
| CN112732749A (en) * | 2021-01-26 | 2021-04-30 | 长威信息科技发展股份有限公司 | Big data hybrid calculation and unified access method and system |
| CN113190870A (en) * | 2021-05-27 | 2021-07-30 | 新华三技术有限公司 | Redis database access authority control method and device |
| CN114125497A (en) * | 2021-11-19 | 2022-03-01 | 中国电信集团系统集成有限责任公司 | SFU architecture, media transmission method thereof and electronic equipment |
| CN114625320A (en) * | 2022-03-15 | 2022-06-14 | 江苏太湖慧云数据系统有限公司 | Hybrid cloud platform data management system based on characteristics |
| CN114637866A (en) * | 2022-05-09 | 2022-06-17 | 武汉汉灿科技有限公司 | Information management method and device for digital new media |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN105069158A (en) * | 2015-08-25 | 2015-11-18 | 携程计算机技术（上海）有限公司 | Data mining method and system |
| CN106202485A (en) * | 2016-07-18 | 2016-12-07 | 乐视控股（北京）有限公司 | Data manipulation method and system |
| CN106656979A (en) * | 2016-10-20 | 2017-05-10 | 北京集奥聚合科技有限公司 | Data interaction method and system for receiving and transmitting data |
| CN106980699A (en) * | 2017-04-14 | 2017-07-25 | 中国科学院深圳先进技术研究院 | A kind of data processing platform (DPP) and system |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN107798037A (en) * | 2017-04-26 | 2018-03-13 | 平安科技（深圳）有限公司 | The acquisition methods and server of user characteristic data |
-
2018
- 2018-03-22 CN CN201810242151.6A patent/CN108959337A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN105069158A (en) * | 2015-08-25 | 2015-11-18 | 携程计算机技术（上海）有限公司 | Data mining method and system |
| CN106202485A (en) * | 2016-07-18 | 2016-12-07 | 乐视控股（北京）有限公司 | Data manipulation method and system |
| CN106656979A (en) * | 2016-10-20 | 2017-05-10 | 北京集奥聚合科技有限公司 | Data interaction method and system for receiving and transmitting data |
| CN106980699A (en) * | 2017-04-14 | 2017-07-25 | 中国科学院深圳先进技术研究院 | A kind of data processing platform (DPP) and system |
| CN107798037A (en) * | 2017-04-26 | 2018-03-13 | 平安科技（深圳）有限公司 | The acquisition methods and server of user characteristic data |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726205A (en) * | 2018-12-26 | 2019-05-07 | 国电南瑞科技股份有限公司 | A kind of power system security stability analysis data-storage system and storage method |
| CN109726205B (en) * | 2018-12-26 | 2021-07-23 | 国电南瑞科技股份有限公司 | Storage system and storage method for safety and stability analysis data of power system |
| CN110704445A (en) * | 2019-09-30 | 2020-01-17 | 石化盈科信息技术有限责任公司 | Industrial data lake system |
| CN110704445B (en) * | 2019-09-30 | 2023-07-25 | 石化盈科信息技术有限责任公司 | Industrial data lake system |
| CN110781238A (en) * | 2019-10-08 | 2020-02-11 | 中国建设银行股份有限公司 | Client view caching method and device based on combination of Redis and Hbase |
| CN110781238B (en) * | 2019-10-08 | 2022-09-13 | 中国建设银行股份有限公司 | Client view caching method and device based on combination of Redis and Hbase |
| CN111090882B (en) * | 2019-12-18 | 2022-08-05 | 北京浪潮数据技术有限公司 | Operation control method, device and equipment for redis database |
| CN111090882A (en) * | 2019-12-18 | 2020-05-01 | 北京浪潮数据技术有限公司 | Operation control method, device and equipment for redis database |
| CN111429137A (en) * | 2020-03-17 | 2020-07-17 | 深圳市随金科技有限公司 | Modular data acquisition method and device, server and storage medium |
| CN112380167A (en) * | 2020-11-17 | 2021-02-19 | 深圳市和讯华谷信息技术有限公司 | Batch data verification method and device, computer equipment and storage medium |
| CN112732749A (en) * | 2021-01-26 | 2021-04-30 | 长威信息科技发展股份有限公司 | Big data hybrid calculation and unified access method and system |
| CN113190870A (en) * | 2021-05-27 | 2021-07-30 | 新华三技术有限公司 | Redis database access authority control method and device |
| CN114125497A (en) * | 2021-11-19 | 2022-03-01 | 中国电信集团系统集成有限责任公司 | SFU architecture, media transmission method thereof and electronic equipment |
| CN114125497B (en) * | 2021-11-19 | 2024-05-07 | 中电信数智科技有限公司 | SFU architecture, media transmission method thereof and electronic equipment |
| CN114625320A (en) * | 2022-03-15 | 2022-06-14 | 江苏太湖慧云数据系统有限公司 | Hybrid cloud platform data management system based on characteristics |
| CN114625320B (en) * | 2022-03-15 | 2024-01-02 | 江苏太湖慧云数据系统有限公司 | Hybrid cloud platform data management system based on characteristics |
| CN114637866A (en) * | 2022-05-09 | 2022-06-17 | 武汉汉灿科技有限公司 | Information management method and device for digital new media |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108959337A (en) | Big data acquisition methods, device, equipment and storage medium | |
| CN109033471B (en) | Information asset identification method and device | |
| CN112540811B (en) | Cache data detection method and device, computer equipment and storage medium | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN107895011B (en) | Session information processing method, system, storage medium and electronic equipment | |
| CN109074377A (en) | Managed function for real-time processing data stream executes | |
| US20230289828A1 (en) | Data processing method, computer device, and readable storage medium | |
| US20210092160A1 (en) | Data set creation with crowd-based reinforcement | |
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| CN109063138A (en) | For in block chain, that is, service platform search data method, equipment and storage medium | |
| CN112686717B (en) | Data processing method and system for advertisement recall | |
| CN108647357A (en) | The method and device of data query | |
| CN109614262A (en) | Business method of calibration, device and computer readable storage medium | |
| CN112003940B (en) | Payment network state processing method and server based on block chain and online service | |
| CN112541009A (en) | Data query method and device, electronic equipment and storage medium | |
| CN110109978A (en) | Data analysing method, device, server and readable storage medium storing program for executing based on index | |
| CN105491078A (en) | Data processing method and device in SOA system, and SOA system | |
| CN113918526A (en) | Log processing method and device, computer equipment and storage medium | |
| CN108268468A (en) | The analysis method and system of a kind of big data | |
| CN109308290A (en) | A kind of efficient data cleaning conversion method based on CIM | |
| US9740601B2 (en) | Globalization testing management service configuration | |
| US10482268B1 (en) | Systems and methods for access management | |
| CN107871055A (en) | A kind of data analysing method and device | |
| CN110443044A (en) | Block chain client bug excavation method, device, equipment and storage medium | |
| CN112860954A (en) | Real-time computing method and real-time computing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181207 |
|
| RJ01 | Rejection of invention patent application after publication |