CN108647323B - Occupational competence data summarizing method - Google Patents
Occupational competence data summarizing method Download PDFInfo
- Publication number
- CN108647323B CN108647323B CN201810447297.4A CN201810447297A CN108647323B CN 108647323 B CN108647323 B CN 108647323B CN 201810447297 A CN201810447297 A CN 201810447297A CN 108647323 B CN108647323 B CN 108647323B
- Authority
- CN
- China
- Prior art keywords
- data
- user
- employment
- authentication
- integration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0815—Network architectures or network communication protocols for network security for authentication of entities providing single-sign-on or federations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0876—Network architectures or network communication protocols for network security for authentication of entities based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Abstract
The invention discloses a data gathering method of professional ability, which integrates acquired employment data from a transmission layer, a calculation layer and a storage layer respectively, and comprises identity integration and data integration; carrying out pressure load sharing on a network link outlet in a transmission layer, and carrying out pressure load sharing on a user request; integrating data in a computing layer through distributed distribution, layered caching and layered transfer, and performing identity integration on a user; hashed storage is performed in the storage layer. The invention is suitable for the transmission, calculation and storage of large-scale concurrent data, thereby realizing the efficient summarization of a large amount of employment data of each port.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data summarization method of occupational competence.
Background
With the rapid development of the internet, the information interoperability is more and more extensive. Employment data has the characteristics of wide dispersion, large data information amount and the like, and the employment data is generally collected from various ports and then summarized; when large employment data are summarized, the problem of high data concurrency needs to be faced.
In the existing data summarization mode, after a large number of users request the system, the I/O bottleneck of network bandwidth and WEB server can be caused; when a large number of user requests are received for calculation, bottlenecks of the service server and the service support server can be caused; the transmission layer and the calculation layer generate a large amount of data, and the data volume is suddenly increased, so that the bottlenecks of the database and the storage are caused; efficient data transmission, computation and storage cannot be performed in the face of a large number of user accesses and high concurrent requests.
Disclosure of Invention
In order to solve the problems, the invention provides a data summarization method of professional ability, which is suitable for transmission, calculation and storage of large-scale concurrent data, so that high-efficiency summarization of employment data of a large number of ports is realized.
In order to achieve the purpose, the invention adopts the technical scheme that: a data gathering method of professional ability integrates acquired employment data from a transmission layer, a calculation layer and a storage layer respectively, and comprises identity integration and data integration; carrying out pressure load sharing on a network link outlet in a transmission layer, and carrying out pressure load sharing on a user request; integrating data in a computing layer through distributed distribution, layered caching and layered transfer, and performing identity integration on a user; performing hash storage in the storage layer;
the method comprises the following steps:
s100, acquiring employment data from different terminals, transmitting and analyzing the employment data, classifying the employment data at a network link outlet, entering a transmission channel, and performing pressure load sharing on a user request;
s200, respectively carrying out identity integration and data integration on the employment data in a calculation layer by using the received employment data;
s300, storing the integrated employment data in an employment data area of the database, wherein the employment data area comprises an identity authentication area and a data storage area.
Further, the step S100 includes the steps of:
s101, acquiring employment data from each terminal;
s102, performing data analysis on employment data of different terminals to obtain classification attributes of the data;
s103, distributing the data of the same category to corresponding transmission channels according to the classification attributes;
and S104, performing pressure load sharing on the user request, and distributing the user request to different network servers.
Furthermore, the image data in the business data is separated from the page data, and a page access request server and an independent image server are established, so that the pressure of providing the page access request server is reduced.
Further, a user basic data database is arranged in an identity authentication area of the database, and basic authentication data are provided for identity authentication of employment data; the user basic data is uniformly stored in the user basic data database, so that the basic data can be provided for identity authentication, and the efficient operation of the user authentication is ensured.
Further, in the identity integration process, an authentication protocol is used for user identity authentication and single sign-on; unified operation and management of a large number of users can be met, and operation efficiency is improved.
Further, the authentication protocol structurally comprises a user side and an authentication center, the authentication center performs authentication work, and the user side processes an access request for accessing the limited resource of the user side; and when the user side detects that the user logs in and accesses, the user information enters an authentication center for user authentication.
Furthermore, in order to ensure the efficient operation of a large amount of parallel data on a computing layer, the data integration is realized by carrying out distributed distribution, layered caching and layered transfer on the data;
processing massive employment data through a distributed algorithm, and distributing the load of calculation;
caching partial calculated results to relieve a request for reading a storage layer;
data transfer is performed through the intermediate server, and a large number of requests are distributed to the internal servers for calculation.
Further, in order to meet the requirement of efficient operation of storage and retrieval of a large amount of concurrent data in a storage layer, employment data is stored in a data storage area of a database in a hash mode through a base table, and data in the database is stored in a scattered mode;
the database is separated by arranging various classification modules, and different classification modules correspond to different database tables; in a classification module, establishing user table hash on employment data according to a user ID; and storing and calling employment data according to the user table hash.
The beneficial effects of the technical scheme are as follows:
the invention can deal with data summarization in the aspects of a large number of user accesses and high concurrent requests; the method is suitable for transmission, calculation and storage of large-scale concurrent data, so that high-efficiency summarization of a large amount of employment data of each port is realized;
the invention provides unified identity authentication and single sign-on, and meets the requirements of a professional ability analysis big data service platform;
the invention can reduce the development complexity and improve the scalability of the system by the distributed modular processing of the transmission, calculation and storage of the concurrent data;
the invention can reduce the cost, improve the system performance, has good expansibility and can add servers at any time to supplement the system performance.
Drawings
FIG. 1 is a schematic flow chart of a data summarization method of professional ability according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described with reference to the accompanying drawings.
In this embodiment, referring to fig. 1, the present invention provides a data summarization method for professional ability, which integrates acquired employment data from a transmission layer, a calculation layer, and a storage layer, respectively, and includes identity integration and data integration; carrying out pressure load sharing on a network link outlet in a transmission layer, and carrying out pressure load sharing on a user request; integrating data in a computing layer through distributed distribution, layered caching and layered transfer, and performing identity integration on a user; performing hash storage in the storage layer;
the method comprises the following steps:
s100, acquiring employment data from different terminals, transmitting and analyzing the employment data, classifying the employment data at a network link outlet, entering a transmission channel, and performing pressure load sharing on a user request;
s200, respectively carrying out identity integration and data integration on the employment data in a calculation layer by using the received employment data;
s300, storing the integrated employment data in an employment data area of the database, wherein the employment data area comprises an identity authentication area and a data storage area.
As an optimization solution of the above embodiment, the step S100 includes the steps of:
s101, acquiring employment data from each terminal;
s102, performing data analysis on employment data of different terminals to obtain classification attributes of the data;
s103, distributing the data of the same category to corresponding transmission channels according to the classification attributes;
and S104, performing pressure load sharing on the user request, and distributing the user request to different network servers.
The image data in the business data is separated from the page data, and a page access request server and an independent image server are established to reduce the pressure of providing the page access request server.
As an optimization scheme of the embodiment, a user basic data database is arranged in an identity authentication area of the database, and basic authentication data are provided for identity authentication of employment data; the user basic data is uniformly stored in the user basic data database, so that the basic data can be provided for identity authentication, and the efficient operation of the user authentication is ensured.
In the identity integration process, an authentication protocol is used for user identity authentication and single sign-on; unified operation and management of a large number of users can be met, and operation efficiency is improved.
The authentication protocol structurally comprises a user side and an authentication center, the authentication center carries out authentication work, and the user side processes an access request for accessing limited resources of the user side; and when the user side detects that the user logs in and accesses, the user information enters an authentication center for user authentication.
In order to ensure the efficient operation of a large amount of parallel data in a computing layer, the data integration is realized by carrying out distributed distribution, layered caching and layered transfer on the data;
processing massive employment data through a distributed algorithm, and distributing the load of calculation;
caching partial calculated results to relieve a request for reading a storage layer;
data transfer is performed through the intermediate server, and a large number of requests are distributed to the internal servers for calculation.
As an optimization scheme of the embodiment, in order to meet the requirement of efficient operation of storage and retrieval of a large amount of concurrent data in a storage layer, employment data is stored in a data storage area of a database in a hash mode by using a base table, and data in the database is stored in a scattered mode;
the database is separated by arranging various classification modules, and different classification modules correspond to different database tables; in a classification module, establishing user table hash on employment data according to a user ID; and storing and calling employment data according to the user table hash.
The foregoing shows and describes the general principles and broad features of the present invention and advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (7)
1. A data summarization method of professional ability is characterized in that collected employment data are integrated from a transmission layer, a calculation layer and a storage layer respectively, wherein the integration comprises identity integration and data integration; carrying out pressure load sharing on a network link outlet in a transmission layer, and carrying out pressure load sharing on a user request; integrating data in a computing layer through distributed distribution, layered caching and layered transfer, and performing identity integration on a user; performing hash storage in the storage layer;
the method comprises the following steps:
s100, acquiring employment data from different terminals, transmitting and analyzing the employment data, classifying the employment data at a network link outlet, entering a transmission channel, and performing pressure load sharing on a user request;
the step S100 includes the steps of:
s101, acquiring employment data from each terminal;
s102, performing data analysis on employment data of different terminals to obtain classification attributes of the data;
s103, distributing the data of the same category to corresponding transmission channels according to the classification attributes;
s104, performing pressure load sharing on the user request, and distributing the user request to different network servers;
s200, respectively carrying out identity integration and data integration on the employment data in a calculation layer by using the received employment data;
s300, storing the integrated employment data in an employment data area of the database, wherein the employment data area comprises an identity authentication area and a data storage area.
2. The method for summarizing data of professional ability according to claim 1, wherein the picture data in the job data is separated from the page data, and a page access request server and a separate picture server are established.
3. The method for summarizing data of occupational abilities according to claim 1, wherein a user basic data database is arranged in an identity authentication area of the database, and basic authentication data are provided for identity authentication of employment data.
4. The method of claim 3, wherein during identity integration, user identity authentication and single sign-on are performed using an authentication protocol.
5. The method of claim 4, wherein the authentication protocol structurally comprises a user side and an authentication center, the authentication center performs authentication, and the user side processes an access request for accessing a restricted resource of the user side; and when the user side detects that the user logs in and accesses, the user information enters an authentication center for user authentication.
6. The method for summarizing data of professional ability according to claim 5, wherein the data integration is performed by distributed distribution, hierarchical caching and hierarchical transfer of data;
processing massive employment data through a distributed algorithm, and distributing the load of calculation;
caching partial calculated results to relieve a request for reading a storage layer;
data transfer is performed through the intermediate server, and a large number of requests are distributed to the internal servers for calculation.
7. The method for summarizing data of professional ability according to claim 1, wherein employment data is stored in a data storage area of a database by using a base table hash, and the data in the database is stored in a scattered manner;
the database is separated by arranging various classification modules, and different classification modules correspond to different database tables; in a classification module, establishing user table hash on employment data according to a user ID; and storing and calling employment data according to the user table hash.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810447297.4A CN108647323B (en) | 2018-05-11 | 2018-05-11 | Occupational competence data summarizing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810447297.4A CN108647323B (en) | 2018-05-11 | 2018-05-11 | Occupational competence data summarizing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108647323A CN108647323A (en) | 2018-10-12 |
| CN108647323B true CN108647323B (en) | 2021-03-16 |
Family
ID=63754510
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810447297.4A Active CN108647323B (en) | 2018-05-11 | 2018-05-11 | Occupational competence data summarizing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108647323B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102420872A (en) * | 2011-12-06 | 2012-04-18 | 公安部交通管理科学研究所 | Hyper text transfer protocol (HTTP)-data-stream-based transmission management framework |
| CN104468697A (en) * | 2014-10-10 | 2015-03-25 | 浙江广播电视集团 | Radio station data transmission load distribution method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120117022A1 (en) * | 2009-07-13 | 2012-05-10 | Jean-Michel Collomb | Method and system for verifying data accuracy |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN107343041B (en) * | 2017-06-29 | 2020-05-19 | 三峡大学 | Accurate poverty alleviation management system and method based on cloud computing |
-
2018
- 2018-05-11 CN CN201810447297.4A patent/CN108647323B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102420872A (en) * | 2011-12-06 | 2012-04-18 | 公安部交通管理科学研究所 | Hyper text transfer protocol (HTTP)-data-stream-based transmission management framework |
| CN104468697A (en) * | 2014-10-10 | 2015-03-25 | 浙江广播电视集团 | Radio station data transmission load distribution method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108647323A (en) | 2018-10-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20110040771A1 (en) | Distributed hardware-based data querying | |
| CN100440891C (en) | Method for balancing gridding load | |
| US11431827B2 (en) | Data center management system | |
| CN102209087A (en) | Method and system for MapReduce data transmission in data center having SAN | |
| WO2020186807A1 (en) | System and method for power data linking based on blockchain technology | |
| CN102024008A (en) | Limitation method of Web server concurrent number | |
| CN107193898A (en) | The inquiry sharing method and system of log data stream based on stepped multiplexing | |
| CN103561078A (en) | Telecom operation system and service implementation method | |
| US20230017300A1 (en) | Query method and device suitable for olap query engine | |
| CN110708256A (en) | CDN scheduling method, device, network equipment and storage medium | |
| CN111404932A (en) | Method for accessing medical institution system to smart medical cloud service platform | |
| WO2022156542A1 (en) | Data access method and system, and storage medium | |
| CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
| Zhao et al. | Dynamic replica creation strategy based on file heat and node load in hybrid cloud | |
| CN113676518A (en) | Distributed data scheduling collection platform based on blocks | |
| CN101014044A (en) | Network GIS system and data transmitting method thereof | |
| CN108647323B (en) | Occupational competence data summarizing method | |
| CN103106242A (en) | Phone bill query method and phone bill query system | |
| CN102902593A (en) | Protocol distribution processing system based on cache mechanism | |
| CN113553199B (en) | Method and device for processing multi-client access by using asynchronous non-blocking mode | |
| Dong et al. | Design of IoT Gateway for Crop Growth Environmental Monitoring Based on Edge-Computing Technology | |
| CN112073223B (en) | System and method for managing and controlling operation of cloud computing terminal and cloud server | |
| CN111813542B (en) | Load balancing method and device for parallel processing of large-scale graph analysis task | |
| CN103647712A (en) | Distributed route processing business method and distributed route processing business system | |
| CN112597173A (en) | Distributed database cluster system peer-to-peer processing system and processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |