CN110309203B - Interactive and user-defined data modeling system based on big data - Google Patents
Interactive and user-defined data modeling system based on big data Download PDFInfo
- Publication number
- CN110309203B CN110309203B CN201910590331.8A CN201910590331A CN110309203B CN 110309203 B CN110309203 B CN 110309203B CN 201910590331 A CN201910590331 A CN 201910590331A CN 110309203 B CN110309203 B CN 110309203B
- Authority
- CN
- China
- Prior art keywords
- modeling
- client
- server
- user
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses an interactive and user-defined data modeling system based on big data, which comprises a client, a server and a computing end, wherein the client is connected with the server through a network; the client and the server can be deployed in any server, and the computing end is deployed in a distributed computing environment; the client provides various operators for modeling, the operators are pre-solidified in the system, and a user constructs a business-based modeling flow in a self-defined operator combination mode; the server is used for analyzing the modeling process and submitting the modeling process to the computing end for computing; the client provides a coding-free, visual and interactive self-defined modeling mode for a user, effectively promotes the rapid floor application of big data machine learning in the industry, greatly reduces the use threshold of machine learning, and enlarges the application range of machine learning.
Description
Technical Field
The invention relates to the field of data mining modeling, in particular to an interactive and user-defined data modeling system based on big data.
Background
Along with the continuous deepening of information-based construction, governments and enterprises accumulate mass data. Due to the lack of data mining analysis capability and teams, data value discovery through data analysis mining remains a difficult point for governments and enterprises. In view of the rapid increase of open source technologies and high-speed iteration, common enterprises are difficult to follow, and the existing big data technology innovation application is still dominated by internet enterprises. At present, domestic big data is still in a concept landing stage, and the existing product has the problems of complex installation configuration, difficulty in understanding and applying an algorithm, difficulty in deploying a model and the like in the landing application process. Therefore, how to quickly fall on the ground for machine learning application based on industry big data, the use threshold of machine learning is reduced, and the expansion of the machine learning application range is a problem to be solved urgently at present.
Disclosure of Invention
The invention designs a novel machine learning modeling system, namely an interactive and user-defined data modeling system based on big data, which comprises a client, a server and a computer. According to the system, rich modeling operators are solidified for the user at the client, and the user can freely establish a machine learning modeling process at the client in a dragging mode, so that business personnel with rich experience can avoid coding modeling, and the entry threshold of machine learning is greatly reduced.
To achieve the above object, the present application provides an interactive and custom data modeling system based on big data, the system comprising: the system comprises a client, a server and a computing end; the client and the server can be deployed in any server, and the computing end is deployed in a distributed computing environment; the client provides various operators for modeling, the operators are pre-solidified in the system, and a user constructs a business-based modeling flow in a self-defined operator combination mode; the server side is used for analyzing the modeling process and submitting the modeling process to the calculation side for calculation.
The interactive and user-defined data modeling system based on the big data can reduce the investment of government and enterprise infrastructure and manpower, quickly and accurately mine the commercial value behind the big data, and help the government and the enterprise to improve the efficiency of data value discovery.
Preferably, the client also provides functions of managing and expanding the modeling process; the server is also used for monitoring the state of the workflow calculation task at the calculation end and transmitting the workflow calculation task to the client.
Preferably, the client is used for interacting with a user, and comprises: the method comprises the steps of self-defining a modeling process, managing the modeling process, modeling interaction and expanding a modeling component.
Preferably, the custom modeling process includes: and the user combines the self-defined operators according to the business requirements, displays the business modeling flow through the self-defined workflow and trains the business model.
Preferably, the management modeling process includes: the client side manages the business workflow constructed by the user in a unified manner; the service workflows constructed by different users can be shared, constructed together and managed together through the client, and the service workflows constructed by the users and constructed together can be added, deleted, modified and viewed through the client.
Preferably, the modeling interaction comprises: the client provides a modeling interaction function for a user, and the user performs custom modeling and manages a modeling process through the client; and the client transmits the state of the modeling process and the model training progress to the user.
Preferably, the extended modeling component includes: and adding a personalized and self-defined modeling operator by the user by utilizing an extension interface provided by the client.
Preferably, the server is configured to analyze a modeling process and monitor a state of the computing end, and includes: analyzing the modeling process, monitoring the process state, submitting the task and collecting the log.
Preferably, the analytical modeling process includes: the server analyzes the constructed service workflow of the user into a task which can be calculated by the calculation end;
the monitoring process state comprises the following steps: the server monitors the state and the progress of a workflow calculation task at the calculation end in real time and transmits the state and the progress to the client in real time;
the submission task comprises the following steps: the server side assembles tasks according to the modeling flow analysis result and submits the tasks to the computing side for computing;
collecting the log includes: and the server collects the log of the computation end and transmits the log to the client.
Preferably, the calculation end is used for carrying actual calculation of the modeling process.
One or more technical solutions provided by the present application have at least the following technical effects or advantages:
by solidifying the machine learning operator in the system, a user can customize a modeling process at a client, a server analyzes the related modeling process and submits a calculation task to a calculation end for calculation, meanwhile, the server monitors the calculation state of the modeling process and transmits the calculation state to the client in real time, and the user can manage and check the self-constructed modeling process through the client; the invention effectively promotes the quick landing of the application of the big data machine learning industry, greatly reduces the use threshold of the machine learning and enlarges the application range of the machine learning.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention;
FIG. 1 is a block diagram of a big data based interactive and custom data modeling system framework;
FIG. 2 is a modeling flow timing diagram;
FIG. 3 is an analytical modeling flow chart;
FIG. 4 is a server-side task submission flow diagram.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflicting with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described and thus the scope of the present invention is not limited by the specific embodiments disclosed below.
Referring to fig. 1, the present application provides an interactive and custom data modeling system based on big data, which includes a client, a server and a computer; the client and the server can be deployed in any server, and the computing end is deployed in a distributed computing environment; the client provides various operators for modeling, the operators are solidified in the system in advance, and a user constructs a business-based modeling process in a self-defined operator combination mode; the server is used for analyzing the modeling process and submitting the modeling process to the calculation end for calculation; the client side also provides functions of managing and expanding the modeling process; the server is also used for monitoring the state of the workflow calculation task at the calculation end and transmitting the workflow calculation task to the client.
The client is used for interacting with a user and mainly comprises a self-defined modeling process, a management modeling process, a modeling interaction and an extended modeling component function; the client can be a WEB end or a mobile end.
And the user-defined modeling flow function is used for displaying the business modeling flow and training a business model through a user-defined workflow according to the combination mode of user-defined operators according to the business requirements.
Managing modeling process functions, including: the client side manages the business workflow constructed by the user in a unified manner; the service workflows constructed by different users can be shared, constructed together and managed together through the client, and the service workflows constructed by the users and constructed together can be added, deleted, modified and viewed through the client.
The user interaction function means that the client provides a modeling interaction function for a user, and the user can customize a modeling and manage a modeling process and the like through the client; the client may communicate the state of the modeling process, model training progress, etc. to the user.
Expanding the function of a modeling component, namely adding an individualized and customized modeling operator by a user through an expansion interface provided by a client; new functions, such as deep learning and notewood, can also be extended by the above functions.
And the server is used for analyzing the modeling process and monitoring the state of the computing end, and mainly comprises the functions of analyzing the modeling process, monitoring the state of the process, submitting tasks, collecting logs and the like.
And analyzing the modeling flow function, namely analyzing the constructed business workflow of the user into a task which can be calculated by the calculation end by the server end.
And the process state monitoring function refers to that the server side monitors the state (success and failure) and the progress of the workflow calculation task at the calculation end in real time and transmits the state and the progress to the client side in real time.
And the task submitting function refers to that the server side assembles a task according to the modeling flow analysis result and submits the task to the computing side for computing.
The log collection function means that the server side collects logs of the computing side and feeds the logs back to the client side, and therefore the user can conveniently control the modeling process state integrally.
And the calculating end is used for bearing actual calculation of the modeling process.
As shown in FIG. 1, the present invention provides a big data based interactive and custom data modeling system, which comprises 3 parts: the system comprises a client, a server and a computing side. The client is mainly used for interacting with a user and comprises theme functions such as a user-defined modeling process, a management modeling process, a user interaction and an extended modeling component; the user can visually, self-define and interactively construct a modeling process at a client, an operator required by modeling is solidified in a system in advance, and an individualized operator can be added through a modeling component expansion function; a modeling process constructed by a user at a client can be analyzed at a server to submit a task to a computing end, and the server monitors the computing state of the modeling process at the computing end and transmits the computing state to the client; and the calculation end is used for calculating a modeling process.
The whole modeling process is from the beginning of construction of a user to the completion of final calculation to obtain a model, and the model needs to be completed by matching of a client, a server and a calculation end, as shown in fig. 2. Firstly, a user builds a modeling process at a client, and the built modeling process is transmitted to a server in a file form; meanwhile, the client can manage and view the modeling process; secondly, the server analyzes the modeling file, analyzes the modeling flow into a task which can be calculated according to the analysis result, and submits the task to the calculation end; then, the computing end computes tasks and transmits the computing state to the server end; and finally, the server transmits the state to the client. Analyzing and submitting the modeling process are important methods in the system, as shown in fig. 3, the system judges whether the modeling process customized and constructed by the user is correct according to customized rules, and if the modeling process customized and constructed by the user is incorrect, the modeling process is reconstructed; if the modeling process is correct, the modeling process is saved as a workflow file, such as json or xml; the server analyzes the workflow file, and the workflow file is assembled into a computable task according to the analysis result, wherein the analysis method is a self-development method corresponding to the file content, if the assembly is successful, the analysis process is finished, otherwise, the analysis is carried out again.
The server converts the analysis result into a calculation task and submits the calculation task to the calculation end, as shown in fig. 4. And the server divides the tasks according to different calculations, submits the tasks to different calculation engines according to the sequence in the modeling process, if the tasks are successfully submitted, and otherwise, the tasks are submitted again.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A big data based interactive and custom data modeling system, the system comprising:
the system comprises a client, a server and a computing end; the client and the server can be deployed in any server, and the computing end is deployed in a distributed computing environment; the client provides various operators for modeling, the operators are pre-solidified in the system, and a user constructs a business-based modeling flow in a self-defined operator combination mode; the server is used for analyzing the modeling process and submitting the modeling process to the computing end for computing in the form of computing tasks;
the server side: judging whether the modeling process customized and constructed by the user is correct or not according to the customized rule, and if not, reconstructing the modeling process; if the modeling process is correct, saving the modeling process as a workflow file;
the server analyzes the workflow file, and the workflow file is assembled into a computable task according to the analysis result, wherein the analysis method is a self-development method corresponding to the file content, if the assembly is successful, the analysis process is finished, otherwise, the analysis is carried out again;
and the server converts the analysis result into a calculation task and submits the calculation task to the calculation end, the server divides the task according to different calculations, submits the task to different calculation engines according to the sequence in the modeling flow, if the submission is successful, the submission of the task is successful, otherwise, the submission is repeated.
2. The big data based interactive and custom data modeling system as claimed in claim 1, wherein said client further provides functions to manage and extend modeling processes; the server is also used for monitoring the state of the workflow calculation task at the calculation end and transmitting the workflow calculation task to the client.
3. A big data based interactive and custom data modeling system, as claimed in any of claims 1-2, wherein said client is configured to interact with a user, comprising: the method comprises the steps of self-defining a modeling process, managing the modeling process, modeling interaction and expanding a modeling component.
4. The big-data based interactive and custom data modeling system of claim 3, wherein the custom modeling process comprises: and the user combines the self-defined operators according to the business requirements, displays the business modeling flow through the self-defined workflow and trains the business model.
5. The big-data based interactive and custom data modeling system of claim 3, wherein the administrative modeling process comprises: the client side manages the business workflow constructed by the user in a unified manner; the service workflows constructed by different users can be shared, constructed together and managed together through the client, and the service workflows constructed by the users and constructed together can be added, deleted, modified and viewed through the client.
6. The big-data based interactive and custom data modeling system of claim 3, wherein modeling interactions include: the client provides a modeling interaction function for a user, and the user performs custom modeling and manages a modeling process through the client; and the client transmits the state of the modeling process and the model training progress to the user.
7. The big-data based interactive and custom data modeling system of claim 3, wherein the extended modeling component comprises: the user can add personalized and self-defined modeling operators by utilizing an extension interface provided by the client.
8. The interactive and custom data modeling system based on big data as claimed in any of claims 1-2, wherein said server is used to analyze modeling process and monitor state of the computing end, comprising: analyzing the modeling process, monitoring the process state, submitting the task and collecting the log.
9. The big-data based interactive and custom data modeling system of claim 8, wherein the analytical modeling process comprises: the server analyzes the constructed service workflow of the user into a task which can be calculated by the calculation end;
the monitoring process state comprises the following steps: the server monitors the state and the progress of a workflow calculation task at the calculation end in real time and transmits the state and the progress to the client in real time;
the submission task comprises the following steps: the server side assembles tasks according to the modeling flow analysis result and submits the tasks to the computing side for computing;
collecting the log includes: and the server collects the log of the computation end and transmits the log to the client.
10. The interactive and custom data modeling system based on big data as claimed in any of claims 1-2, wherein said computing side is used to carry the actual calculations of the modeling process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910590331.8A CN110309203B (en) | 2019-07-02 | 2019-07-02 | Interactive and user-defined data modeling system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910590331.8A CN110309203B (en) | 2019-07-02 | 2019-07-02 | Interactive and user-defined data modeling system based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309203A CN110309203A (en) | 2019-10-08 |
| CN110309203B true CN110309203B (en) | 2021-08-10 |
Family
ID=68079713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910590331.8A Active CN110309203B (en) | 2019-07-02 | 2019-07-02 | Interactive and user-defined data modeling system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309203B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111240662B (en) * | 2020-01-16 | 2024-01-09 | 同方知网(北京)技术有限公司 | Spark machine learning system and method based on task visual drag |
| CN111966705A (en) * | 2020-08-12 | 2020-11-20 | 北京海致网聚信息技术有限公司 | Interactive data modeling method |
| CN114385233B (en) * | 2022-03-24 | 2022-08-02 | 山东省计算中心（国家超级计算济南中心） | Cross-platform adaptive data processing workflow system and method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102624870A (en) * | 2012-02-01 | 2012-08-01 | 北京航空航天大学 | Intelligent optimization algorithm based cloud manufacturing computing resource reconfigurable collocation method |
| CN105843873A (en) * | 2016-03-18 | 2016-08-10 | 北京物思创想科技有限公司 | System and method for managing data modeling |
| CN106548210A (en) * | 2016-10-31 | 2017-03-29 | 腾讯科技（深圳）有限公司 | Machine learning model training method and device |
| CN107480365A (en) * | 2017-08-09 | 2017-12-15 | 华中科技大学 | A kind of stylized heterogeneous modeling |
| CN107943463A (en) * | 2017-12-15 | 2018-04-20 | 清华大学 | Interactive mode automation big data analysis application development system |
| CN107948307A (en) * | 2017-12-12 | 2018-04-20 | 华东交通大学 | A kind of intelligent vehicle-mounted device and its safety communicating method based on car networking |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104850405A (en) * | 2015-05-25 | 2015-08-19 | 武汉众联信息技术股份有限公司 | Intelligent configurable workflow engine and implementation method therefor |
| CN104933104A (en) * | 2015-05-29 | 2015-09-23 | 数据堂（北京）科技股份有限公司 | Method and system for collecting metadata |
| CN105046408A (en) * | 2015-06-25 | 2015-11-11 | 国网山东省电力公司 | Configurable workflow realization method and system |
| CN106487852B (en) * | 2015-08-31 | 2020-02-18 | 阿里巴巴集团控股有限公司 | Method, device, terminal equipment and system for realizing client file synchronization |
| CN106371976A (en) * | 2016-08-31 | 2017-02-01 | 福建天晴数码有限公司 | Method and system for monitoring thread by client and Web server |
| CN108804457B (en) * | 2017-04-28 | 2021-10-08 | 北京京东尚科信息技术有限公司 | Data synchronization and processing method and device, electronic equipment and computer readable medium |
| CN109558573A (en) * | 2017-12-21 | 2019-04-02 | 上海土木信息科技有限公司 | A kind of list flow engine based on user-defined m odel language |
| CN109542851A (en) * | 2018-11-30 | 2019-03-29 | 北京金山云网络技术有限公司 | File updating method, apparatus and system |
| CN111581113B (en) * | 2020-06-04 | 2023-10-27 | 网易（杭州）网络有限公司 | File updating method and device, storage medium, processor and electronic device |
| CN112508205A (en) * | 2020-12-04 | 2021-03-16 | 中国科学院深圳先进技术研究院 | Method, device and system for scheduling federated learning |
-
2019
- 2019-07-02 CN CN201910590331.8A patent/CN110309203B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102624870A (en) * | 2012-02-01 | 2012-08-01 | 北京航空航天大学 | Intelligent optimization algorithm based cloud manufacturing computing resource reconfigurable collocation method |
| CN105843873A (en) * | 2016-03-18 | 2016-08-10 | 北京物思创想科技有限公司 | System and method for managing data modeling |
| CN106548210A (en) * | 2016-10-31 | 2017-03-29 | 腾讯科技（深圳）有限公司 | Machine learning model training method and device |
| CN107480365A (en) * | 2017-08-09 | 2017-12-15 | 华中科技大学 | A kind of stylized heterogeneous modeling |
| CN107948307A (en) * | 2017-12-12 | 2018-04-20 | 华东交通大学 | A kind of intelligent vehicle-mounted device and its safety communicating method based on car networking |
| CN107943463A (en) * | 2017-12-15 | 2018-04-20 | 清华大学 | Interactive mode automation big data analysis application development system |
Non-Patent Citations (1)
| Title |
|---|
| "针对公安民警开展数据建模方法的研究和实践";黄河清;《警察技术》;20180630;3 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309203A (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309203B (en) | Interactive and user-defined data modeling system based on big data | |
| CN108388146B (en) | Three-dimensional assembly process design system based on information physical fusion and operation method | |
| CN106407570A (en) | An airplane assembling process online simulating optimization system | |
| US20160189066A1 (en) | Method and system for automatic evaluation of robustness and disruption management for commercial airline flight operations | |
| CN109410015A (en) | A kind of large-scale customization system based on cloud platform | |
| US11922589B2 (en) | Methods and systems for digital twin augmented reality replication of non-homogeneous elements in integrated environments | |
| Aichernig et al. | Learning and statistical model checking of system response times | |
| JP5582510B2 (en) | A system for estimating the cost of preparing a new industrial plant for operation. | |
| Nikakhtar et al. | Comparison of two simulation software for modeling a construction process | |
| Heithoff et al. | Digital twins for sustainable software systems | |
| US9064223B2 (en) | Method and system for real-time view of software product quality | |
| CN116614344B (en) | Fault diagnosis method and system based on digital twinning | |
| CN116523244A (en) | Testing manpower risk early warning method based on outsourcing resources | |
| KR102355787B1 (en) | Methods to gather screen requirements and to manage processes for the software development including web based services and mobile services | |
| García et al. | Bluejay: A cross-tooling audit framework for agile software teams | |
| Fernando et al. | The importance of software metrics: Perspective of a software development projects in sri lanka | |
| CN114138532A (en) | Remote fault diagnosis system based on project scheme simulation software architecture design | |
| CN114169723A (en) | Digital workshop implementation method and system | |
| Kotronis et al. | Extending SysML to Integrate Cost Analysis Into Model-Based Systems Engineering | |
| Mutschler et al. | An approach to quantify the costs of business process intelligence | |
| Yeung et al. | The role of simulation in digital twin construction | |
| CN116720973B (en) | Digital financial management method and system based on RPA | |
| Putra et al. | Flexible stage-based process performance mining for customer journey analysis | |
| CN116596685A (en) | Large enterprise financial data analysis and management method | |
| Li et al. | A SOA-Based ARIS Model for BPR |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 610000 No. 270, floor 2, No. 8, Jinxiu street, Wuhou District, Chengdu, Sichuan Patentee after: Chengdu shuzhilian Technology Co.,Ltd. Address before: 610000 No.2, 4th floor, building 1, Jule Road intersection, West 1st section of 1st ring road, Wuhou District, Chengdu City, Sichuan Province Patentee before: CHENGDU SHUZHILIAN TECHNOLOGY Co.,Ltd. |
|
| CP03 | Change of name, title or address |