CN110704209A - Data mining analysis platform based on PIPE communication - Google Patents
Data mining analysis platform based on PIPE communication Download PDFInfo
- Publication number
- CN110704209A CN110704209A CN201910891468.7A CN201910891468A CN110704209A CN 110704209 A CN110704209 A CN 110704209A CN 201910891468 A CN201910891468 A CN 201910891468A CN 110704209 A CN110704209 A CN 110704209A
- Authority
- CN
- China
- Prior art keywords
- module
- sub
- analysis
- data mining
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data mining analysis platform based on PIPE (product Internet protocol) communication, in particular to the field of computer information service, which comprises a main channel data mining analysis platform, wherein the main channel data mining analysis platform comprises a sub-communication module, a sub-analysis module and a main control platform module; the main channel data mining analysis platform obtains the relevant configuration of different sub-analysis modules by reading the configuration file; the sub-communication module is used for calculating and analyzing specific business processes, and when the sub-communication module is started, the sub-communication module reads own configuration file information; and after the main channel data mining analysis platform reads the configuration, starting a main control platform module. The invention provides a method for realizing mutual communication among different data mining modules in the same platform by using a PIPE communication basis, thereby reducing the coupling, realizing the information interaction among communication processes, more easily completing complex application scenes and improving the performance of the data mining platform.
Description
Technical Field
The invention relates to the technical field of computer information service, in particular to a data mining analysis platform based on PIPE communication.
Background
In a traditional data mining analysis platform, different data statistics and analysis modules are usually hosted in different computer processes, data statistics modes are complicated and various, and data analysis is completed respectively. The data mining analysis platform is lack of mutual information interaction functions, so that high coupling exists in certain application scenarios, and the workload of the data mining analysis platform is also caused.
The PIPE is a mode for providing communication public resources on the basis of shared resources, and different processes can access the same public resource; therefore, data exchange among the processes needs to pass through a public resource, a buffer area is opened up in the kernel, the process A writes the data into the public resource buffer area from a user space, the process B reads the data from the public resource buffer area, and then the process completes the service scene of the process;
therefore, it is necessary to develop a data mining analysis platform based on PIPE communication.
Disclosure of Invention
In order to overcome the above defects in the prior art, embodiments of the present invention provide a data mining analysis platform based on PIPE-in-PIPE (PIPE) communication, in which a main control platform module calls a PIPE-in-PIPE function method to obtain an ID and a description of a communication PIPE by reading an existing configuration file table of the platform, and at the same time, a channel is created to wait for interactive connection, a sub-analysis module reads its own configuration, obtains the set channel ID and description, and connects channels of the main control platform module; when the sub-analysis module data is used for independently analyzing the service data, the operation state data is written into a channel created by the PIPE, the main control platform module can write into the pipeline, the sub-analysis module can read from the pipeline, the pipeline is realized by using a circular queue, the data flows in from a writing end and flows out from a reading end, so that the process communication between the main control platform module and the sub-analysis module is realized, the mutual communication between different data mining modules in the same platform is realized, the coupling is reduced, the information interaction between communication processes is realized, the complex application scene is more easily completed, and the performance of the data mining platform is improved.
In order to achieve the purpose, the invention provides the following technical scheme: a data mining analysis platform based on PIPE communication comprises a main channel data mining analysis platform, wherein the main channel data mining analysis platform comprises a sub-communication module, a sub-analysis module and a main control platform module;
the main channel data mining analysis platform obtains the relevant configuration of different sub-analysis modules by reading the configuration file;
the sub-communication module is used for calculating and analyzing specific business processes, and when the sub-communication module is started, the sub-communication module reads own configuration file information;
after the main channel data mining analysis platform reads the configuration, starting a main control platform module, wherein the main control platform module is used for starting, stopping, timing or periodically starting and stopping the sub-analysis module;
after the main data mining analysis platform operates, a PIPE communication pipeline is established through a process, and a channel ID and a channel description are obtained to point to two ends of a channel;
the sub-analysis module is automatically loaded into the main data mining analysis platform after the main data mining analysis platform operates, meanwhile, self-configuration file information is read, information of an opened PIPE channel of the main data mining analysis platform is obtained, log information generated during operation of the sub-analysis module is obtained, the log information can be written into an opened PIPE communication pipeline through a process, and after the main data mining analysis platform reads data in the opened PIPE communication pipeline, the information is used for processing a service flow.
In a preferred embodiment, the relevant configurations of the different sub-analysis modules are specifically a channel ID, a channel frequency, and sub-module configuration information.
In a preferred embodiment, the PIPE is implemented by using a ring queue, and data flows into the channel from the write end and flows out of the channel from the read end, so as to implement mutual communication between different processes in the platform.
The invention also provides a data mining and analyzing method based on the PIPE, which comprises the following steps:
s1, after the main data mining analysis platform is started, reading the existing configuration file table of the platform, calling a PIPE function method by the main control platform module, acquiring the ID and description of the communication pipeline through the read configuration file table, and simultaneously establishing a channel to wait for interactive connection;
s2, after the main control platform module finishes loading, the sub-analysis module finishes loading, then reads the self-configuration, obtains the set channel ID and description, and connects the channel of the main control platform module; when the sub-analysis module data is used for analyzing the service data independently, the operation state data is written into a channel created by the PIPE;
s3, the main control platform module can write in the pipeline, the sub analysis module can read from the pipeline, the pipeline is realized by using a circular queue, data flows in from the write end and flows out from the read end, and therefore process communication between the main control platform module and the sub analysis module is realized.
In a preferred embodiment, in step S2, the main control platform module turns on fd [0] and fd [1], and the sub-analysis module also turns on fd [0] and fd [1], that is, the main control platform module and the sub-analysis module both turn on the read end and the write end of the pipeline.
In a preferred embodiment, in step S3, the main control platform module turns off fd [0], the sub-analysis module turns off fd [1], that is, the main control platform module turns off the pipe read end, and the sub-analysis module turns off the pipe write end.
The invention has the technical effects and advantages that:
1. the method comprises the steps that an existing configuration file table of a platform is read, a main control platform module calls a PIPE function method, the ID and description of a communication pipeline are obtained through the read configuration file table, a channel is created to wait for interactive connection, a sub-analysis module reads self-configuration, the set channel ID and description are obtained, and the channel of the main control platform module is connected; when the sub-analysis module data is used for independently analyzing the service data, the operation state data is written into a channel created by the PIPE, the main control platform module can write into a pipeline, the sub-analysis module can read from the pipeline, the pipeline is realized by using a circular queue, the data flows in from a writing end and flows out from a reading end, and thus the process communication between the main control platform module and the sub-analysis module is realized; according to the technical scheme, different business processes are realized among different processes, and own information can be sent to other processes in a PIPE transmission mode when specific needs are met, so that resources of a data analysis platform are integrated, the data analysis capability is improved, and complex business scenes are easier to realize;
2. the invention provides a method for realizing mutual communication among different data mining modules in the same platform by using a PIPE communication basis, thereby reducing the coupling, realizing the information interaction among communication processes, more easily completing complex application scenes and improving the performance of the data mining platform.
Drawings
FIG. 1 is a block diagram of a main channel data mining analysis platform according to the present invention.
FIG. 2 is a schematic diagram of a parent process creating a PIPE communication PIPE according to the present invention.
FIG. 3 is a diagram illustrating a parent process fork coming out of a child process and the child process inheriting the ID and description of the parent process.
FIG. 4 is a diagram illustrating a parent process and a child process completing PIPE communication according to the present invention.
The reference signs are: 1, a main channel data mining analysis platform, a 101 sub-communication module, a 102 sub-analysis module and a 103 main control platform module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the data mining analysis platform based on PIPE communication shown in fig. 1, the data mining analysis platform comprises a main channel data mining analysis platform 1, wherein the main channel data mining analysis platform 1 comprises a sub communication module 101, a sub analysis module 102 and a main control platform module 103;
the main channel data mining analysis platform 1 obtains the relevant configuration of different sub-analysis modules 102, specifically, channel ID, channel frequency, sub-module configuration information and the like, by reading the configuration file;
the sub-communication module 101 is used for calculating and analyzing a specific business process, and after the sub-communication module 101 is started, the sub-communication module 101 reads own configuration file information;
after the main channel data mining analysis platform 1 is configured by reading, starting a main control platform module 103, wherein the main control platform module 103 is used for starting, stopping, timing or periodically starting and stopping the sub-analysis module 102;
after the main data mining analysis platform operates, a PIPE communication pipeline is established through a process, and a channel ID and a channel description are obtained to point to two ends of a channel;
the sub-analysis module 102 is automatically loaded into the main data mining analysis platform after the main data mining analysis platform operates, meanwhile, reads the information of the self-configuration file, obtains the PIPE channel information opened by the main data mining analysis platform, and generates log information during the operation of the sub-analysis module 102, namely, the log information can be written into the opened PIPE communication pipeline through the process, and the main data mining analysis platform processes the service flow by reading the data in the existing PIPE communication pipeline.
Furthermore, the PIPE communication pipeline is realized by using a ring queue, data flows into a channel from a write end and flows out of the channel from a read end, and therefore mutual communication among different processes in the platform is realized.
According to the data mining analysis method based on PIPE shown in FIGS. 2-4, the specific steps are as follows:
s1, after the main data mining analysis platform is started, reading the existing configuration file table of the platform, calling a PIPE function method by the main control platform module 103, acquiring the ID and description of the communication pipeline through the read configuration file table, and simultaneously creating a channel to wait for interactive connection, as shown in FIG. 2 specifically;
s2, after the main control platform module 103 finishes loading, the sub-analysis module 102 finishes loading similarly, then reads self-configuration, obtains the set channel ID and description, and connects the channel of the main control platform module 103; when the sub-analysis module 102 performs the service data analysis on the data alone, the running state data is written into the channel created by the PIPE, the fd [0] and the fd [1] are opened by the main control platform module 103, and the fd [0] and the fd [1] are also opened by the sub-analysis module 102, that is, the main control platform module 103 and the sub-analysis module 102 both open the pipeline read end and the write end; the main control platform module 103 and the sub-analysis module 102 can write in the pipeline and read from the pipeline; the pipeline is realized by using a circular queue, data flows in from a write end and flows out from a read end, so that process communication between the main control platform module 103 and the sub-analysis module 102 is realized, and specifically, see fig. 3;
s3, the main control platform module 103 can write into the pipeline, the sub-analysis module 102 can read from the pipeline, the pipeline is realized by using a circular queue, data flows in from a write end and flows out from a read end, and thus process communication between the main control platform module 103 and the sub-analysis module 102 is realized; the main control platform module 103 turns off fd [0], the sub-analysis module 102 turns off fd [1], that is, the main control platform module 103 turns off the pipeline read end, and the sub-analysis module 102 turns off the pipeline write end, as shown in fig. 4.
According to the technical scheme, different business processes are realized among different processes, and own information can be sent to other processes in a PIPE transmission mode when specific needs are met, so that resources of a data analysis platform are integrated, the data analysis capability is improved, and complex business scenes are easier to realize.
The points to be finally explained are: first, in the description of the present application, it should be noted that, unless otherwise specified and limited, the terms "mounted," "connected," and "connected" should be understood broadly, and may be a mechanical connection or an electrical connection, or a communication between two elements, and may be a direct connection, and "upper," "lower," "left," and "right" are only used to indicate a relative positional relationship, and when the absolute position of the object to be described is changed, the relative positional relationship may be changed;
secondly, the method comprises the following steps: in the drawings of the disclosed embodiments of the invention, only the structures related to the disclosed embodiments are referred to, other structures can refer to common designs, and the same embodiment and different embodiments of the invention can be combined with each other without conflict;
and finally: the above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that are within the spirit and principle of the present invention are intended to be included in the scope of the present invention.
Claims (6)
1. A data mining analysis platform based on PIPE communication is characterized in that: the main channel data mining analysis platform comprises a main channel data mining analysis platform (1), wherein the main channel data mining analysis platform (1) comprises a sub communication module (101), a sub analysis module (102) and a main control platform module (103);
the main channel data mining analysis platform (1) obtains the relevant configuration of different sub-analysis modules (102) by reading the configuration file;
the sub-communication module (101) is used for calculating and analyzing a specific business process, and when the sub-communication module (101) is started, the sub-communication module (101) reads own configuration file information;
after the main channel data mining analysis platform (1) is configured by reading, starting a main control platform module (103), wherein the main control platform module (103) is used for starting, stopping, timing or periodically starting and stopping the sub-analysis module (102);
after the main data mining analysis platform operates, a PIPE communication pipeline is established through a process, and a channel ID and a channel description are obtained to point to two ends of a channel;
the sub-analysis module (102) is automatically loaded into the main data mining and analysis platform after the main data mining and analysis platform operates, meanwhile, the information of the self-configuration file is read, the PIPE channel information opened by the main data mining and analysis platform is obtained, the sub-analysis module (102) generates log information during operation, the log information can be written into an opened PIPE communication pipeline through a process, and the main data mining and analysis platform processes the service flow by reading the data in the existing PIPE communication pipeline.
2. The PIPE-based data mining analysis platform of claim 1, wherein: the related configuration of the different sub-analysis modules (102) is specifically channel ID, channel frequency and sub-module configuration information.
3. The PIPE-based data mining analysis platform of claim 1, wherein: the PIPE communication pipeline is realized by using a ring queue, data flows into a channel from a write end and flows out of the channel from a read end, and therefore mutual communication among different processes in the platform is realized.
4. A data mining analysis platform based on PIPE communication according to any of claims 1-3, characterized by: the method also comprises a data mining analysis method, and the specific steps are as follows:
s1, after the main data mining analysis platform is started, reading the existing configuration file table of the platform, calling a PIPE function method by the main control platform module (103), acquiring the ID and description of the communication pipeline through the read configuration file table, and simultaneously establishing a channel to wait for interactive connection;
s2, after the main control platform module (103) is loaded, the sub-analysis module (102) is loaded, then the self-configuration is read, the set channel ID and description are obtained, and the channel of the main control platform module (103) is connected; when the data of the sub-analysis module (102) is subjected to service data analysis independently, the running state data is written into a channel created by the PIPE;
s3, the main control platform module (103) can write in the pipeline, the sub analysis module (102) can read from the pipeline, the pipeline is realized by using a circular queue, data flows in from a write end and flows out from a read end, and thus process communication between the main control platform module (103) and the sub analysis module (102) is realized.
5. The PIPE-based data mining analysis method of claim 4, wherein: in step S2, the main control platform module 103 turns on fd [0] and fd [1], and the sub-analysis module 102 also turns on fd [0] and fd [1], that is, the main control platform module 103 and the sub-analysis module 102 both turn on the read end and the write end of the pipeline.
6. The PIPE-based data mining analysis method of claim 4, wherein: in step S3, the main control platform module (103) closes fd [0], and the sub-analysis module (102) closes fd [1], that is, the main control platform module (103) closes the pipeline read end, and the sub-analysis module (102) closes the pipeline write end.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910891468.7A CN110704209B (en) | 2019-09-20 | 2019-09-20 | Data mining analysis platform based on PIPE communication |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910891468.7A CN110704209B (en) | 2019-09-20 | 2019-09-20 | Data mining analysis platform based on PIPE communication |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110704209A true CN110704209A (en) | 2020-01-17 |
| CN110704209B CN110704209B (en) | 2022-05-06 |
Family
ID=69195796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910891468.7A Active CN110704209B (en) | 2019-09-20 | 2019-09-20 | Data mining analysis platform based on PIPE communication |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110704209B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666163A (en) * | 2020-05-15 | 2020-09-15 | 北京仁科互动网络技术有限公司 | Network arrangement system and method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20090054569A (en) * | 2007-11-27 | 2009-06-01 | 주식회사 만도 | Database management system of vehicle and method thereof |
| CN102508765A (en) * | 2011-11-17 | 2012-06-20 | 深圳市中兴移动通信有限公司 | Method for monitoring software log in real time by inter-process communication |
| CN104063375A (en) * | 2013-03-18 | 2014-09-24 | 深圳市腾讯计算机系统有限公司 | Pipeline communication method and device |
| CN108959966A (en) * | 2018-07-16 | 2018-12-07 | 九江学院 | Cloud platform auditing system based on big data administrative analysis |
-
2019
- 2019-09-20 CN CN201910891468.7A patent/CN110704209B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20090054569A (en) * | 2007-11-27 | 2009-06-01 | 주식회사 만도 | Database management system of vehicle and method thereof |
| CN102508765A (en) * | 2011-11-17 | 2012-06-20 | 深圳市中兴移动通信有限公司 | Method for monitoring software log in real time by inter-process communication |
| CN104063375A (en) * | 2013-03-18 | 2014-09-24 | 深圳市腾讯计算机系统有限公司 | Pipeline communication method and device |
| CN108959966A (en) * | 2018-07-16 | 2018-12-07 | 九江学院 | Cloud platform auditing system based on big data administrative analysis |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666163A (en) * | 2020-05-15 | 2020-09-15 | 北京仁科互动网络技术有限公司 | Network arrangement system and method |
| CN111666163B (en) * | 2020-05-15 | 2023-07-11 | 北京仁科互动网络技术有限公司 | Network arrangement system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110704209B (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20140327686A1 (en) | Drawing Method, Apparatus, and Terminal | |
| CN109542614B (en) | Resource allocation method, device, terminal and storage medium | |
| US20210026688A1 (en) | Method and Apparatus for Scheduling Processor Core, and Storage Medium | |
| US7017163B2 (en) | Method, server, and client used in client-server distributed system | |
| CN108055304B (en) | Remote data synchronization method, device, server, equipment and storage medium | |
| CN106559679A (en) | Method, server and mobile terminal that video is decoded | |
| WO2018157732A1 (en) | Data transmission method and apparatus | |
| CN111212142A (en) | Service processing method, integrated open docking platform and computer storage medium | |
| TWI444044B (en) | Device, method and computer program product for transmitting data within remote application | |
| CN109634764A (en) | Work-flow control method, apparatus, equipment, storage medium and system | |
| CN105592019B (en) | The method that two-way access is applied between dual execution environment | |
| US9451433B2 (en) | Device and method of driving multi-PDP data communication for mobile broadband device | |
| US20210233572A1 (en) | Video processing method, electronic device, and storage medium | |
| CN110704209B (en) | Data mining analysis platform based on PIPE communication | |
| CN101895531B (en) | Client equipment, multimedia data unloading system and unloading method | |
| CN112764877A (en) | Method and system for communication between hardware acceleration equipment and process in docker | |
| CN111782473A (en) | Distributed log data processing method, device and system | |
| CN112347401A (en) | House same-screen watching method, system, device and computer readable storage medium | |
| US20120079130A1 (en) | Agent service processing method and internet protocol agent terminal | |
| CN115269013A (en) | Asynchronous computing method and device supporting concurrent processing of multiple computation acceleration engines | |
| CN114490104A (en) | Information forwarding method, device, equipment and medium in Android system module | |
| CN105120532A (en) | RIL processing method based on multiplexing and wireless communication module | |
| US9436522B2 (en) | Multimedia data processing method | |
| EP4141644A1 (en) | Data transmission method and data transmission device | |
| CN112083914A (en) | Method and system for realizing soft bus of object model embedded operating system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220414 Address after: 210000 No. 12, Mazhou East Road, Mau Ling Street, Jiangning District, Nanjing, Jiangsu Applicant after: Jiangsu Future Wisdom Information Technology Co.,Ltd. Applicant after: Nanjing Chuanli New Energy Technology Co., Ltd Address before: 211111 No. 12, Mazhou East Road, Mau Ling Street, Jiangning District, Nanjing, Jiangsu Applicant before: Jiangsu Future Wisdom Information Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |