WO2020223997A1 - 一种存算显全局可配置的数据分析软件架构设计方法 - Google Patents
一种存算显全局可配置的数据分析软件架构设计方法 Download PDFInfo
- Publication number
- WO2020223997A1 WO2020223997A1 PCT/CN2019/087192 CN2019087192W WO2020223997A1 WO 2020223997 A1 WO2020223997 A1 WO 2020223997A1 CN 2019087192 W CN2019087192 W CN 2019087192W WO 2020223997 A1 WO2020223997 A1 WO 2020223997A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- layer
- interface
- analysis
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/24—Object-oriented
Definitions
- the present invention relates to the technical field of software system architecture, in particular to a data analysis software architecture design method that is globally configurable for storage, calculation and display.
- the technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art, and provide a data analysis software architecture design method that can store, calculate and display globally and is configurable. Using this method, a software developer can flexibly expand data and algorithms. , And is friendly to software users, with good scalability, strong maintainability, and high availability data analysis software architecture.
- the technical solution adopted by the present invention is: a data analysis software architecture design method that is globally configurable for storage, calculation and display.
- the data analysis software architecture is divided into interface layer, analysis layer, data access layer and plug-in layer.
- Four levels, the sum of the data access layer, the analysis layer, and the interface layer is named the storage and calculation display layer;
- the interface layer provides the software user with a visual interface for interactive operation;
- the analysis layer is responsible for executing algorithm data analysis and processing the business logic of the software system;
- the data access layer is based on data analysis requirements Obtain data from the data storage medium and pass it to the analysis layer to provide data services for the analysis layer;
- the plug-in layer provides a way for software developers to configure data and algorithms, analyzes the configuration of the above data and algorithms, and provides an interface for the analysis results
- the interface is used to call the display
- the plug-in layer is configured with two external storage types: data storage and algorithm storage;
- the data storage is an external storage medium for data sets required for data analysis;
- the algorithm storage is an external algorithm code called when data analysis is executed Package, in the form of a function or a code collection of functions or function sets that input data sets and algorithm parameters and return analysis results;
- software developers configure data storage and algorithm storage for data analysis in the plug-in layer, and configure the specific data format and
- the semantics are defined in the software development and design phase, and the developers follow the configuration method defined in the design phase when configuring; the configuration of the data storage is used to describe the files in the file system where the data is stored, and describe each file or file in the data file or database.
- the configuration of the algorithm storage is used to describe the basic information of the algorithm plug-in, giving algorithm parameter metadata such as algorithm name, algorithm path, algorithm input parameter constraints, and algorithm output parameter constraints that can be displayed on the interface layer; in addition to the above metadata, Developers extend metadata according to the specific situation; when the software is deployed and running, the plug-in layer will automatically parse the above configuration as the return result of the display interface of the interface layer, the computing interface of the analysis layer, and the storage interface of the data access layer.
- the data access layer accesses the plug-in layer by calling a storage interface; when the data access layer obtains the data index generated by the interface layer and passed by the analysis layer, it calls the storage interface to obtain the storage of the relevant data index parsed by the plug-in layer Location and storage data type These data storage configuration information; the data access layer accesses the external data storage according to the result returned by the storage interface, obtains the required data and saves it as the corresponding data set object according to the storage data type, and uses the data set object as the data for data analysis Set back to the analysis layer.
- the analysis layer accesses the plug-in layer by calling a computing interface; after the analysis layer obtains the algorithm index and algorithm parameters passed in by the interface layer, it calls the computing interface to obtain the access path, input parameter constraints, The output parameters constrain these algorithms to store configuration information; the analysis layer invokes the data analysis engine in the analysis layer according to the results returned by the calculation interface, and passes in the algorithm parameters encapsulated according to the results returned by the calculation interface and the data set returned by the data access layer to the data analysis In the engine, the algorithm package code in the algorithm storage is executed, data analysis calculation is completed, and the analysis result is returned to the interface layer according to the output parameter constraints.
- the interface layer accesses the plug-in layer by calling the display interface; when the software architecture user accesses the visualization interface, the interface calls the display interface to obtain all the algorithm storage configuration information parsed by the plug-in layer, construct a dynamic algorithm selection interface, and the software architecture uses The personnel select the algorithm from this interface and enter the algorithm input parameters; at the same time, the display interface is called to obtain all the data storage configuration information analyzed by the plug-in layer, and a dynamic data set selection interface is constructed.
- the software architecture user selects the data set from this interface; the software architecture user
- the visualization interface of the obtained analysis results is also constructed based on the algorithm output parameter constraints obtained by the display interface, and the form of the results includes images, tables, texts and analysis process logs.
- the present invention provides a data analysis software architecture design method that is globally configurable for storage, computing and display, so that the designed software architecture is globally configurable.
- the developer can configure the new data storage or algorithm storage in the plug-in layer.
- the corresponding modules of the software can be obtained by calling the storage interface, computing interface, and display interface.
- the corresponding configuration analysis results of the plug-in layer can access the newly-added data storage and algorithm storage, quickly realize the data analysis under the new requirements, and meet the original intention of the software architecture design with friendly use, good scalability, strong maintainability, and high availability.
- Figure 1 is a schematic diagram of a software architecture provided by an embodiment of the present invention.
- FIG. 2 is a schematic diagram of software architecture modules provided by an embodiment of the present invention.
- Figure 3 is an example of the configuration of a data plug-in configurator provided by an embodiment of the present invention.
- FIG. 5 is a schematic diagram of core classes of software architecture provided by an embodiment of the present invention.
- FIG. 6 is a schematic diagram of a data analysis process of a software architecture provided by an embodiment of the present invention.
- a data analysis software architecture design method with a globally configurable storage, calculation and display of the present invention is used to design a data analysis software architecture.
- a design method for data analysis software architecture that is globally configurable for storage, calculation and display.
- the data analysis software architecture is divided into four levels: interface layer, analysis layer, data access layer, and plug-in layer, as shown in Figure 1, where the data access layer The sum of the three layers, the analysis layer and the interface layer, is named the storage and calculation display layer;
- the interface layer provides the software user with a visual interface for interactive operation;
- the software user selects the algorithm for data analysis and transmits the data to the analysis layer for analysis and calculation , And display the data analysis results returned by the analysis layer on the visual interface;
- the analysis layer is responsible for performing algorithmic data analysis and processing the business logic of the software system;
- the data access layer obtains and transmits data from the data storage medium according to data analysis requirements Enter the analysis layer to provide data services for the analysis layer;
- the plug-in layer provides a way for software developers to configure data and algorithms, analyze the configuration of the above data and algorithms, and provide an interface for the analysis results;
- the interface is used for the interface layer Display interface calls
- the plug-in layer is configured with two types of external storage: data storage and algorithm storage;
- the data storage is an external storage medium for data sets required for data analysis, and its form includes but is not limited to various databases or data file systems;
- the specific data format and semantics of the configuration are defined in the software development and design phase, and the developers follow the configuration method defined in the design phase when configuring;
- the configuration of the data storage is used to describe the files in the file system where the data is stored , Describe the file or table index number and access path of each file or database table in the data file or database.
- These data files or table metadata, as well as the attribute index number and attribute of each attribute and its sub-attributes in the file or database table Name and attribute data type are data attribute metadata; the configuration stored in the algorithm is used to describe the basic information of the algorithm plug-in, giving the algorithm name, algorithm path, algorithm input parameter constraints, and algorithm output parameter constraints that can be displayed on the interface layer.
- Algorithm parameter metadata in addition to the above metadata, developers can also extend the metadata according to specific conditions; when the software is deployed and running, the plug-in layer will automatically analyze the above configuration as the display interface of the interface layer, the calculation interface of the analysis layer and The return result of the storage interface of the data access layer.
- the data access layer accesses the plug-in layer by calling the storage interface; when the data access layer obtains the data index generated by the interface layer and passed by the analysis layer, it calls the storage interface to obtain the storage location and storage of the relevant data index parsed by the plug-in layer Data type These data storage configuration information; the data access layer accesses the external data storage according to the result returned by the storage interface, obtains the required data and saves it as the corresponding data set object according to the storage data type, and returns the data set object as the data set for data analysis Analysis layer.
- the analysis layer accesses the plug-in layer by calling the computing interface; after the analysis layer obtains the algorithm index and algorithm parameters passed in from the interface layer, it calls the computing interface to obtain the access path, input parameter constraints, and output parameter constraints of the relevant algorithms parsed by the plug-in layer
- These algorithms store configuration information; the analysis layer invokes the data analysis engine in the analysis layer according to the results returned by the computing interface, and passes in the algorithm parameters encapsulated according to the results returned by the computing interface and the data set returned by the data access layer to the data analysis engine.
- Execute the algorithm package code in the algorithm storage complete the data analysis calculation, and return the analysis result to the interface layer according to the output parameter constraints.
- the interface layer accesses the plug-in layer by calling the display interface; when the user of the software architecture accesses the visualization interface, the interface calls the display interface to obtain all the algorithm storage configuration information parsed by the plug-in layer, and build a dynamic algorithm selection interface.
- Interface selection algorithm input algorithm input parameters; at the same time call the display interface to obtain all the data storage configuration information analyzed by the plug-in layer, construct a dynamic data set selection interface, software architecture users select data sets from this interface; analysis obtained by software architecture users
- the visualization interface of the results is also constructed based on the algorithm output parameter constraints obtained by the display interface.
- the results are in the form of images, tables, texts and analysis process logs.
- the data analysis software architecture designed by the software architecture design method of the present invention has 5 modules in the storage, calculation and display layer, which are parameter selection module, result display module, data analysis engine, business logic module, and data access module. ; There are 4 modules in the plug-in layer, namely data plug-in configurator, algorithm plug-in configurator, data plug-in parser, and algorithm plug-in parser; the storage-computing display layer accesses the plug-in layer through display interface, computing interface, and storage interface, such as As shown in Figure 2.
- files in the file system are used as data storage in the software architecture, and an executable R language algorithm package is used as the algorithm storage in the software architecture.
- the data in the file is in the form of a two-dimensional table, where the abscissa is a number of data attributes, the ordinate is the date, and a row of data represents the value of several attributes under a certain date; by determining the file name and the attributes in the file
- the method of column name and date can determine the value of the attribute in a certain date.
- the R language algorithm package is a code file for the basic functions of the data analysis algorithm implemented in the R language. After the server is deployed with the R execution environment, the analysis results can be output after inputting the parameters required by the functions in the algorithm package.
- a data plug-in configurator for configuring data storage is implemented, and its description form is an XML file.
- the category tag is used to describe the file information in the file system, including file index number, file name, file path, data start date and data end date in the file (in this embodiment, a row of file data represents one day , There is a continuous and uninterrupted date between the upper and lower rows, so define the date of the first data row as the start date, and the date of the last data row as the end date.
- the data plug-in parser can parse the number of rows of the file);
- the attribute tag describes the attribute information of each column in the file, including attribute index number, attribute name, column number, and attribute data type.
- the sub-tag attributes of category is a collection of data attributes described by all attribute tags in the file.
- the configuration example of the data plug-in configurator for the "operational parameter" file (file index number 21) is shown in Figure 3.
- the start time and end time in the file represent the corresponding dates of the first row of data and the last row of data in the file.
- Each attribute tag represents the index, name, number of columns, and data type of each column of the file.
- an algorithm plug-in configurator for configuring algorithm storage is implemented, and its description form is an XML file.
- the algorithm plug-in configurator use the algorithm tag to describe the algorithm information of the R algorithm package, including the algorithm package index number, algorithm package name, algorithm package call function name, algorithm package path, algorithm package dependent library, and return chart type.
- the subtag parameters of the algorithm is a collection of all algorithm parameters of the algorithm.
- the description label of the algorithm parameter is parameter, including the algorithm parameter index number, algorithm parameter type and parameter constraints. If the algorithm parameter is a selection type, add the option subtag under the parameter tag to describe the specific option; while the input type algorithm parameter does not need the option subtag.
- the R language algorithm packages contained in the algorithm storage are categorized into three categories: "descriptive statistics", “statistical analysis” and “data mining”, which are described by the RPackage tag; in the three categories
- the algorithm is also divided into several sub-categories through the approach tag.
- Algorithms is a sub-tag under the approach tag, which represents a collection of algorithm tags describing the R algorithm package.
- software users can quickly locate and select the algorithm used through the major tags.
- the top of the interface is the algorithm selection area.
- the above three types of algorithms are the first-level menus in the algorithm selection area.
- the approach label is the second-level menu, and the algorithm label is the third-level menu.
- the configuration example of the algorithm plug-in configurator for the two "time series analysis” algorithms is shown in Figure 4.
- the univariate index prediction model (algorithm index number "3_5_1”) needs to import the third-party R library "lubridate”, and the output image type is For scatter plot, the input limit is "only one data attribute can be selected", and the number of months predicted by the model is an integer not less than 1, and cannot be empty;
- the univariate ARIMA prediction model (algorithm index number "3_5_2”) needs to be introduced into a third-party R library "Tseries" and "forecast”, the output image type is a scatter chart, the input limit is "only one data attribute can be selected", and the number of months predicted by the model is an integer not less than 1, and cannot be empty.
- the server in the storage, computing and display layer, the server is developed using Java Web technology, and the core class diagram of the server part is shown in FIG. 5.
- the AttributePrase class and RPackagePrase class are the data plug-in parser and algorithm plug-in parser in the architecture, respectively.
- the data plug-in parser and algorithm plug-in parser parse the data plug-in configurator and algorithm plug-in configurator described in XML when the software server is started, and deserialize them into memory objects.
- the objects parsed by the AttributePrase class are stored in the CategoriesPool class, and the objects parsed by the RPackagePrase class are stored in the AlgorithmsPool class.
- the other modules only need to call the member methods of the objects of the above classes to call the plug-in configuration information.
- the AttributesServlet class and the RPackageServlet class belong to the interface layer of the software architecture in this embodiment.
- these two classes respectively encapsulate the data of the CategoriesPool object and AlgorithmsPool object and send it to the client to complete Parameter selection module function.
- the DatasetPool class and the DatasetFactory class belong to the data access module of the data access layer.
- the role of the DatasetFactory class is to read the data in the file system, and its positioning of the data depends on the CategoriesPool object.
- the data read by the DatasetFactory class is saved in the form of an object of the DatasetPool class.
- the AnalyzeCore class is a business logic module class that parses client parameters before calling the data analysis engine in the data analysis process, obtains the algorithms and data sets required for data analysis, and encapsulates the engine's algorithm analysis results and returns to the client.
- the function of the data analysis engine is completed by the REngine class, which accepts the algorithm parameters and data sets of the AnalyzeCore object, calls the algorithm of the R execution algorithm package, and simply encapsulates the execution result as a Java object and returns it to AnalyzeCore.
- the core business process of the data analysis process is shown in Figure 6.
- the client After the software user opens the client, the client automatically obtains configuration information from the AttributesServlet class and the RPackageServlet class, and automatically constructs the interface parameter selection module based on the information.
- the software user first selects the algorithm in the client, then searches and selects the data attributes to be added to the data set (that is, selects the range of the data column in the data file), and then enters or selects the algorithm parameters (the input box or the selection box is passed by RPackageServlet Algorithm information generation), finally select the time range of the data (that is, select the range of data rows in the data file) and click the analysis button.
- the above information is then transmitted to the server, received by the ServiceServlet class and forwarded to the AnalyzeCore class.
- the AnalyzeCore class parses the information from the client, and first calls the data in the DatasetPool object according to the information, and the DatasetPool object obtains the data set from the file system through the method of the DatasetFactory class. After the AnalyzeCore class obtains the data, it calls the REngine class with the data set and algorithm information as parameters.
- the REngine class calls the R environment deployed in the server to execute the corresponding R algorithm package according to the incoming algorithm index, and starts the data analysis process.
- the REngine class After the data analysis is over, the REngine class returns the analysis result (successful analysis result or failed error code) to the AnalyzeCore class. After AnalyzeCore is encapsulated, the ServiceServlet class returns the analysis result to the client through the HTTP protocol.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
一种存算显全局可配置的数据分析软件架构设计方法，涉及软件系统架构技术领域。该方法将数据分析软件架构分为界面层、分析层、数据访问层和插件层，并将数据访问层、分析层、界面层三层的总和命名为存算显层；界面层为软件使用人员提供交互式操作的可视化界面；分析层负责执行算法数据分析，以及处理软件系统的业务逻辑；数据访问层根据数据分析需求从数据存储介质中获取数据并传入到分析层；插件层提供软件开发人员配置数据和算法的方式，解析上述数据和算法的配置，并提供解析结果的接口。所述方法使设计的软件架构满足使用友好、扩展性好、维护性强、可用性高的软件架构设计的初衷。
Description
本发明涉及软件系统架构技术领域，尤其涉及一种存算显全局可配置的数据分析软件架构设计方法。
如今软件规模越来越庞大、功能越来越复杂，软件架构的设计随着也越来越受到软件从业人员的重视。软件架构作为软件的骨架，是决定软件质量的重要原因。一个软件在架构设计阶段出现的失误，很难在实际开发阶段予以彻底弥补，最终会造成开发和维护过程中大量资源的浪费，甚至导致软件的彻底失败。
现有的数据分析软件原本大多是为特定分析目的而设计的，这使得数据分析软件在功能上只支持特定的算法种类与特定的数据参数。现有的软件架构设计方法所设计的软件架构不能轻易扩展数据分析所使用的算法、不能支持随着数据变化与分析目的变化造成的算法参数的动态变化。当分析目的和分析方法需要改变时，现有的软件难以扩展，常常需要开发一款新软件以用于新需求下的数据分析，从而使得原有软件被放弃，在满足数据分析人员的动态变化的分析需求的过程中造成了人力物力财力的大量浪费。
发明概述
问题的解决方案
本发明要解决的技术问题是针对上述现有技术的不足，提供一种存算显全局可配置的数据分析软件架构设计方法，使用该方法设计出一种能够让软件开发人员灵活扩展数据与算法，且对软件使用人员使用友好的，扩展性好、维护性强、可用性高的数据分析软件架构。
为解决上述技术问题，本发明所采取的技术方案是：一种存算显全局可配置的数据分析软件架构设计方法，将数据分析软件架构分为界面层、分析层、数据 访问层和插件层四个层次，其中将数据访问层、分析层、界面层三层的总和命名为存算显层；所述界面层为软件使用人员提供交互式操作的可视化界面；软件使用人员选择数据分析的算法和数据传送至分析层进行分析计算，并在可视化界面显示分析层返回的数据分析结果；所述分析层负责执行算法数据分析，以及处理软件系统的业务逻辑；所述数据访问层根据数据分析需求从数据存储介质中获取数据并传入到分析层，为分析层提供数据服务；所述插件层提供软件开发人员配置数据和算法的方式，解析上述数据和算法的配置，并提供解析结果的接口；所述接口用于界面层的显示接口调用、分析层的计算接口调用、以及数据访问层的存储接口调用。
优选地，所述插件层配置两种外部存储类型：数据存储和算法存储；所述数据存储为数据分析所需数据集的外部存储介质；所述算法存储为数据分析执行时调用的外部算法代码包，其形式是输入数据集和算法参数并返回分析结果的函数或函数集的代码集合；软件开发人员将用于数据分析的数据存储和算法存储配置于插件层中，配置的具体数据格式和语义在软件开发设计阶段定义，开发人员在配置时遵循设计阶段定义的配置方式；所述数据存储的配置用于描述数据所存放的文件系统中的文件，描述数据文件或数据库中每个文件或数据库表的文件或表索引号和访问路径这些数据文件或表元数据，以及在文件或数据库表中每个属性及其子属性的属性索引号、属性名、属性数据类型这些数据属性元数据；所述算法存储的配置用于描述算法插件的基本信息，给出可显示在界面层的算法名、算法路径、算法输入参数约束、算法输出参数约束这些算法参数元数据；除了上述元数据外，开发人员根据具体情况扩展元数据；当软件部署并运行后，插件层将自动解析上述配置，作为界面层的显示接口、分析层的计算接口和数据访问层的存储接口的返回结果。
优选地，所述数据访问层通过调用存储接口访问插件层；数据访问层在获得由界面层产生，由分析层传递的数据索引的情况下，调用存储接口获取插件层解析的相关数据索引的存储位置和存储数据类型这些数据存储配置信息；数据访问层根据存储接口返回结果访问外部数据存储，获取所需数据并根据存储数据类型保存为相应数据集对象，并将数据集对象作为数据分析的数据集返回至分 析层。
优选地，所述分析层通过调用计算接口访问插件层；分析层在获取了界面层传入的算法索引和算法参数后，调用计算接口获取插件层解析的相关算法的访问路径、输入参数约束、输出参数约束这些算法存储配置信息；分析层根据计算接口返回的结果，唤起分析层内的数据分析引擎，传入根据计算接口返回结果封装后的算法参数和数据访问层返回的数据集至数据分析引擎中，执行算法存储中的算法包代码，完成数据分析计算，并将分析结果根据输出参数约束返回到界面层。
优选地，所述界面层通过调用显示接口访问插件层；当软件架构使用人员访问可视化界面时，界面调用显示接口获取插件层解析的所有算法存储配置信息，构建动态的算法选择界面，软件架构使用人员从该界面选择算法，输入算法输入参数；同时调用显示接口获取插件层解析的所有数据存储配置信息，构建动态的数据集选择界面，软件架构使用人员从该界面选择数据集；软件架构使用人员获得的分析结果的可视化界面也由显示接口获取的算法输出参数约束为基础构建，结果的形式包括图像、表格、文字和分析过程日志。
发明的有益效果
采用上述技术方案所产生的有益效果在于：本发明提供的一种存算显全局可配置的数据分析软件架构设计方法，使设计的软件架构存算显全局可配置，在软件架构使用人员的需求发生变更时，开发人员可将新的数据存储或算法存储配置于插件层中，此时无需更改存算显层的代码，软件的相应模块即可通过调用存储接口、计算接口、显示接口，获取插件层相应的配置解析结果，从而访问新增的数据存储和算法存储，快速实现新需求下的数据分析，满足使用友好、扩展性好、维护性强、可用性高的软件架构设计的初衷。
对附图的简要说明
图1为本发明实施例提供的软件架构示意图；
图2为本发明实施例提供的软件架构模块示意图；
图3为本发明实施例提供的数据插件配置器配置实例；
图4为本发明实施例提供的算法插件配置器配置实例；
图5为本发明实施例提供的软件架构核心类示意图；
图6为本发明实施例提供的软件架构的数据分析流程示意图。
发明实施例
下面结合附图和实施例，对本发明的具体实施方式作进一步详细描述。以下实施例用于说明本发明，但不用来限制本发明的范围。
本实施例采用本发明的一种存算显全局可配置的数据分析软件架构设计方法设计一种数据分析软件架构。
一种存算显全局可配置的数据分析软件架构设计方法，将数据分析软件架构分为界面层、分析层、数据访问层和插件层四个层次，如图1所示，其中将数据访问层、分析层、界面层三层的总和命名为存算显层；所述界面层为软件使用人员提供交互式操作的可视化界面；软件使用人员选择数据分析的算法和数据传送至分析层进行分析计算，并在可视化界面显示分析层返回的数据分析结果；所述分析层负责执行算法数据分析，以及处理软件系统的业务逻辑；所述数据访问层根据数据分析需求从数据存储介质中获取数据并传入到分析层，为分析层提供数据服务；所述插件层提供软件开发人员配置数据和算法的方式，解析上述数据和算法的配置，并提供解析结果的接口；所述接口用于界面层的显示接口调用、分析层的计算接口调用、以及数据访问层的存储接口调用。
所述插件层配置两种外部存储类型：数据存储和算法存储；所述数据存储为数据分析所需数据集的外部存储介质，其形式包括但不限于各类数据库或数据文件系统；所述算法存储为数据分析执行时调用的外部算法代码包，其形式是输入数据集和算法参数并返回分析结果的函数或函数集的代码集合；软件开发人员将用于数据分析的数据存储和算法存储配置于插件层中，配置的具体数据格式和语义在软件开发设计阶段定义，开发人员在配置时遵循设计阶段定义的配置方式；所述数据存储的配置用于描述数据所存放的文件系统中的文件，描述数据文件或数据库中每个文件或数据库表的文件或表索引号和访问路径这些数 据文件或表元数据，以及在文件或数据库表中每个属性及其子属性的属性索引号、属性名、属性数据类型这些数据属性元数据；所述算法存储的配置用于描述算法插件的基本信息，给出可显示在界面层的算法名、算法路径、算法输入参数约束、算法输出参数约束这些算法参数元数据；除了上述元数据外，开发人员也可根据具体情况扩展元数据；当软件部署并运行后，插件层将自动解析上述配置，作为界面层的显示接口、分析层的计算接口和数据访问层的存储接口的返回结果。
所述数据访问层通过调用存储接口访问插件层；数据访问层在获得由界面层产生，由分析层传递的数据索引的情况下，调用存储接口获取插件层解析的相关数据索引的存储位置和存储数据类型这些数据存储配置信息；数据访问层根据存储接口返回结果访问外部数据存储，获取所需数据并根据存储数据类型保存为相应数据集对象，并将数据集对象作为数据分析的数据集返回至分析层。
所述分析层通过调用计算接口访问插件层；分析层在获取了界面层传入的算法索引和算法参数后，调用计算接口获取插件层解析的相关算法的访问路径、输入参数约束、输出参数约束这些算法存储配置信息；分析层根据计算接口返回的结果，唤起分析层内的数据分析引擎，传入根据计算接口返回结果封装后的算法参数和数据访问层返回的数据集至数据分析引擎中，执行算法存储中的算法包代码，完成数据分析计算，并将分析结果根据输出参数约束返回到界面层。
所述界面层通过调用显示接口访问插件层；当软件架构使用人员访问可视化界面时，界面调用显示接口获取插件层解析的所有算法存储配置信息，构建动态的算法选择界面，软件架构使用人员从该界面选择算法，输入算法输入参数；同时调用显示接口获取插件层解析的所有数据存储配置信息，构建动态的数据集选择界面，软件架构使用人员从该界面选择数据集；软件架构使用人员获得的分析结果的可视化界面也由显示接口获取的算法输出参数约束为基础构建，结果的形式包括图像、表格、文字和分析过程日志。
本实施例中，采用本发明的软件架构设计方法设计的数据分析软件架构在存算显层共有5个模块，分别为参数选择模块、结果展示模块、数据分析引擎、业务 逻辑模块、数据访问模块；在插件层有4个模块，分别为数据插件配置器、算法插件配置器、数据插件解析器、算法插件解析器；存算显层分别通过显示接口、计算接口、存储接口访问插件层，如图2所示。
本实施例使用文件系统中的文件作为软件架构中的数据存储，使用可执行的R语言算法包作为软件架构中的算法存储。文件系统中有若干文件，文件内数据的形式是二维表，其中横坐标是若干数据属性，纵坐标为日期，一行数据表示某个日期下若干属性的值；通过确定文件名、文件内属性列名和日期的方式可以确定该属性在某个日期内的值。R语言算法包是用R语言实现的数据分析算法基本功能的代码文件，当服务器部署了R执行环境后，输入算法包内的函数所需的参数后，即可输出分析结果。
本实施例在插件层中，实现了用于配置数据存储的数据插件配置器，其描述形式为XML文件。在数据插件配置器中，使用category标签描述文件系统中的文件信息，包括文件索引号、文件名、文件路径、文件内数据起始日期和数据结束日期(本实施例中，文件数据一行表示一天，上下行之间为连续不间断的日期，故定义第一数据行的日期为起始日期，最后一个数据行的日期为结束日期，数据插件解析器即可解析出文件的行数)；使用attribute标签描述文件内每列属性的信息，包括属性索引号、属性名、列数和属性数据类型。category的子标签attributes是该文件下所有attribute标签所描述的数据属性的集合。
数据插件配置器对于“操业参数”文件(文件索引号21)的配置实例如图3所示，该文件内的起始时间和结束时间表示文件第一行数据和最后一行数据的对应日期，每个attribute标签代表文件每列数据的索引、名称、列数和数据类型。
本实施例在插件层中，实现了用于配置算法存储的算法插件配置器，其描述形式为XML文件。在算法插件配置器中，使用algorithm标签描述R算法包的算法信息，包括算法包索引号、算法包名、算法包调用函数名、算法包路径、算法包依赖库、返回图表类型。algorithm的子标签parameters是该算法所有算法参数的集合，算法参数的描述标签为parameter，包括算法参数索引号、算法参数类型和参数约束。若算法参数是选择型，则在parameter标签下加上option子标签描述具体选项；而输入型的算法参数则无需option子标签。本实施例中，将算法存储 所含的R语言算法包进行了归类，分为“描述统计”、“统计分析”和“数据挖掘”三大类算法，以RPackage标签描述；在三大类算法下也通过approach标签分别划分为若干小类。而algorithms是approach标签下的子标签，表示描述R算法包的algorithm标签的集合，在界面层，软件使用人员能通过大类标签快速定位和选择所用算法。例如在本实施例的界面上，界面顶部是算法选择区，上述三大类算法是算法选择区的一级菜单，approach标签为二级菜单，algorithm标签为三级菜单，在选择算法时鼠标点击三级菜单进入相应具体三级算法的算法参数输入界面。
算法插件配置器对于两种“时间序列分析”算法的配置实例如图4所示，其中单变量指数预测模型(算法索引号“3_5_1”)需要引入第三方R库“lubridate”，输出图像类型为散点图，输入限制为“只能选择一个数据属性”，模型预测月数为不小于1的整数，不得为空；单变量ARIMA预测模型(算法索引号“3_5_2”)需要引入第三方R库“tseries”和“forecast”，输出图像类型为散点图，输入限制为“只能选择一个数据属性”，模型预测月数为不小于1的整数，不得为空。
本实施例在存算显层中，服务器端使用Java Web技术开发，服务器部分的核心类图如图5所示。在分析层中，AttributePrase类和RPackagePrase类分别为架构中的数据插件解析器和算法插件解析器。数据插件解析器和算法插件解析器在软件服务器启动时解析上述用XML描述的数据插件配置器和算法插件配置器，反序列化为内存对象。AttributePrase类解析的对象保存在CategoriesPool类中，RPackagePrase类解析的对象保存在AlgorithmsPool类中，其余模块调用插件配置信息只需调用上述类的对象的成员方法。AttributesServlet类和RPackageServlet类属于本实施例中软件架构的界面层，作用是当软件架构使用人员访问客户端时，这两个类分别将CategoriesPool对象和AlgorithmsPool对象的数据封装后发送之客户端，从而完成参数选择模块的功能。DatasetPool类和DatasetFactory类属于数据访问层的数据访问模块。DatasetFactory类的作用是读取文件系统中的数据，其对数据的定位与CategoriesPool对象存在依赖关系。DatasetFactory类读取的数据以DatasetPool类的对象形式保存。AnalyzeCore类是数据分析过程中调用数据分析引擎前解析客户端参数、获取数据分析所需算法和数据集并将引擎的算法分析结果封装并返回到客户端的业务逻辑模块类。数据分析引擎的功能由REngine类完 成，接受AnalyzeCore对象的算法参数和数据集，调用R执行算法包的算法，并将执行结果简易封装为Java对象返回到AnalyzeCore中。
本实施例中，数据分析流程的核心业务流程如图6所示。软件使用人员打开客户端后，客户端自动从AttributesServlet类和RPackageServlet类获取配置信息，并根据信息自动构建界面的参数选择模块。软件使用人员在客户端中首先选择算法，随后搜索并选择加入数据集的数据属性(即选择数据文件内数据的列的范围)，接着输入或选择算法参数(输入框或选择框由RPackageServlet传来的算法信息生成)，最后选择数据的时间范围(即选择数据文件内数据的行的范围)并点击分析按钮。上述信息随即传到服务器端，由ServiceServlet类接收并转发到AnalyzeCore类。AnalyzeCore类解析客户端传来的信息，首先根据信息调用DatasetPool对象中的数据，而DatasetPool对象又通过DatasetFactory类的方法从文件系统获取数据集。AnalyzeCore类获得数据后将数据集和算法信息作为参数调用REngine类，REngine类根据传入的算法索引，调用服务器中部署的R环境执行相应的R算法包，开始数据分析过程。数据分析结束后，REngine类将分析结果(成功的分析结果或失败的错误代号)返回AnalyzeCore类，AnalyzeCore进行封装后由ServiceServlet类通过HTTP协议返回分析结果到客户端。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明权利要求所限定的范围。
Claims (5)
- 一种存算显全局可配置的数据分析软件架构设计方法，其特征在于：将数据分析软件架构分为界面层、分析层、数据访问层和插件层四个层次，其中将数据访问层、分析层、界面层三层的总和命名为存算显层；所述界面层为软件使用人员提供交互式操作的可视化界面；软件使用人员选择数据分析的算法和数据传送至分析层进行分析计算，并在可视化界面显示分析层返回的数据分析结果；所述分析层负责执行算法数据分析，以及处理软件系统的业务逻辑；所述数据访问层根据数据分析需求从数据存储介质中获取数据并传入到分析层，为分析层提供数据服务；所述插件层提供软件开发人员配置数据和算法的方式，解析上述数据和算法的配置，并提供解析结果的接口；所述接口用于界面层的显示接口调用、分析层的计算接口调用、以及数据访问层的存储接口调用。
- 根据权利要求1所述的一种存算显全局可配置的数据分析软件架构设计方法，其特征在于：所述插件层配置两种外部存储类型：数据存储和算法存储；所述数据存储为数据分析所需数据集的外部存储介质；所述算法存储为数据分析执行时调用的外部算法代码包，其形式是输入数据集和算法参数并返回分析结果的函数或函数集的代码集合；软件开发人员将用于数据分析的数据存储和算法存储配置于插件层中，配置的具体数据格式和语义在软件开发设计阶段定义，开发人员在配置时遵循设计阶段定义的配置方式；所述数据存储的配置用于描述数据所存放的文件系统中的文件，描述数据文件或数据库中每个文件或数据库表的文件或表索引号和访问路径这些数据文件或表元数据，以及在文件或数据库表中每个属性及其子属性的属性索引号、属性名、属性数据类型这些数据属性元数据；所述算法存储的配置用于描述算法插件的基本信息，给出可显示在界面层的算法名、算法路径、算法输入参 数约束、算法输出参数约束这些算法参数元数据；除了上述元数据外，开发人员根据具体情况扩展元数据；当软件部署并运行后，插件层将自动解析上述配置，作为界面层的显示接口、分析层的计算接口和数据访问层的存储接口的返回结果。
- 根据权利要求1所述的一种存算显全局可配置的数据分析软件架构设计方法，其特征在于：所述数据访问层通过调用存储接口访问插件层；数据访问层在获得由界面层产生，由分析层传递的数据索引的情况下，调用存储接口获取插件层解析的相关数据索引的存储位置和存储数据类型这些数据存储配置信息；数据访问层根据存储接口返回结果访问外部数据存储，获取所需数据并根据存储数据类型保存为相应数据集对象，并将数据集对象作为数据分析的数据集返回至分析层。
- 根据权利要求1所述的一种存算显全局可配置的数据分析软件架构设计方法，其特征在于：所述分析层通过调用计算接口访问插件层；分析层在获取了界面层传入的算法索引和算法参数后，调用计算接口获取插件层解析的相关算法的访问路径、输入参数约束、输出参数约束这些算法存储配置信息；分析层根据计算接口返回的结果，唤起分析层内的数据分析引擎，传入根据计算接口返回结果封装后的算法参数和数据访问层返回的数据集至数据分析引擎中，执行算法存储中的算法包代码，完成数据分析计算，并将分析结果根据输出参数约束返回到界面层。
- 根据权利要求1所述的一种存算显全局可配置的数据分析软件架构设计方法，其特征在于：所述界面层通过调用显示接口访问插件层；当软件架构使用人员访问可视化界面时，界面调用显示接口获取插件层解析的所有算法存储配置信息，构建动态的算法选择界面，软件架构使用人员从该界面选择算法，输入算法输入参数；同时调用显示接口获取插件层解析的所有数据存储配置信息，构建动态的数据集选择界面，软件架构使用人员从该界面选择数 据集；软件架构使用人员获得的分析结果的可视化界面也由显示接口获取的算法输出参数约束为基础构建，结果的形式包括图像、表格、文字和分析过程日志。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367895.5A CN109976729B (zh) | 2019-05-05 | 2019-05-05 | 一种存算显全局可配置的数据分析软件架构设计方法 |
| CN201910367895.5 | 2019-05-05 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2020223997A1 true WO2020223997A1 (zh) | 2020-11-12 |
Family
ID=67072746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2019/087192 Ceased WO2020223997A1 (zh) | 2019-05-05 | 2019-05-16 | 一种存算显全局可配置的数据分析软件架构设计方法 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109976729B (zh) |
| WO (1) | WO2020223997A1 (zh) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112949061A (zh) * | 2021-03-01 | 2021-06-11 | 北京清华同衡规划设计研究院有限公司 | 基于可复用算子的村镇发展模型构建方法和系统 |
| CN114510471A (zh) * | 2022-02-16 | 2022-05-17 | 北京九栖科技有限责任公司 | 用于大数据平台实时状态计算的方法、服务器及存储介质 |
| CN114816374A (zh) * | 2021-01-28 | 2022-07-29 | 中国科学院沈阳自动化研究所 | 可视化数据分析流程建模方法及系统 |
| CN115774553A (zh) * | 2021-09-07 | 2023-03-10 | 中国电子科技集团公司第十四研究所 | 一种软件化雷达算法模块图形化建模方法 |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111309295B (zh) * | 2020-03-12 | 2023-04-18 | 超越科技股份有限公司 | 一种三维变密度约束地下界面反演可视化方法，设备及可读存储介质 |
| CN115794040B (zh) * | 2022-11-14 | 2024-02-06 | 深圳十沣科技有限公司 | 构造cae软件架构的方法、装置、设备及存储介质 |
| CN115729641B (zh) * | 2022-11-21 | 2023-08-25 | 中电金信软件有限公司 | 一种自定义组件的元数据流转方法、装置及电子设备 |
| CN116450698A (zh) * | 2023-01-17 | 2023-07-18 | 中国联合网络通信集团有限公司 | 记录型数据处理方法、装置、设备及存储介质 |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015100646A1 (zh) * | 2013-12-31 | 2015-07-09 | 北京新媒传信科技有限公司 | 软件架构的实现方法和实现平台 |
| US20170371647A1 (en) * | 2016-06-28 | 2017-12-28 | Quarkdata Software Co., Ltd | Terminal management framework and terminal management method |
| CN109325620A (zh) * | 2018-09-23 | 2019-02-12 | 上海木白网络科技有限公司 | 基于云计算的制造系统的调度服务平台及方法 |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8417715B1 (en) * | 2007-12-19 | 2013-04-09 | Tilmann Bruckhaus | Platform independent plug-in methods and systems for data mining and analytics |
| CN102929607A (zh) * | 2012-10-09 | 2013-02-13 | 曙光信息产业(北京)有限公司 | 一种基于云计算的数据挖掘系统功能层析架构 |
| CN106294439B (zh) * | 2015-05-27 | 2020-02-28 | 北京广通神州网络技术有限公司 | 一种数据推荐系统及其数据推荐方法 |
| CN106021378B (zh) * | 2016-05-11 | 2019-07-05 | 吕骏 | 基于数据抽取、数据可视化的查询分析方法及系统 |
| CN106570081A (zh) * | 2016-10-18 | 2017-04-19 | 同济大学 | 基于语义网的大规模离线数据分析框架 |
| CN107526600B (zh) * | 2017-09-05 | 2020-11-10 | 成都优易数据有限公司 | 一种基于hadoop和spark的可视化数据挖掘分析平台及其数据清洗方法 |
| CN109542960B (zh) * | 2018-10-18 | 2023-03-14 | 国网内蒙古东部电力有限公司信息通信分公司 | 一种数据分析域系统 |
-
2019
- 2019-05-05 CN CN201910367895.5A patent/CN109976729B/zh not_active Expired - Fee Related
- 2019-05-16 WO PCT/CN2019/087192 patent/WO2020223997A1/zh not_active Ceased
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015100646A1 (zh) * | 2013-12-31 | 2015-07-09 | 北京新媒传信科技有限公司 | 软件架构的实现方法和实现平台 |
| US20170371647A1 (en) * | 2016-06-28 | 2017-12-28 | Quarkdata Software Co., Ltd | Terminal management framework and terminal management method |
| CN109325620A (zh) * | 2018-09-23 | 2019-02-12 | 上海木白网络科技有限公司 | 基于云计算的制造系统的调度服务平台及方法 |
Non-Patent Citations (1)
| Title |
|---|
| CHEN YUKUN: "China Master’s Theses Full-Text Database", RESEARCH AND DESIGN OF A DATA MINING SYSTEM PLATFORM BASED ON PLUGIN ARCHITECTURE, 1 April 2009 (2009-04-01), pages 1 - 76, XP055752235, ISSN: 1674-0246 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114816374A (zh) * | 2021-01-28 | 2022-07-29 | 中国科学院沈阳自动化研究所 | 可视化数据分析流程建模方法及系统 |
| CN112949061A (zh) * | 2021-03-01 | 2021-06-11 | 北京清华同衡规划设计研究院有限公司 | 基于可复用算子的村镇发展模型构建方法和系统 |
| CN112949061B (zh) * | 2021-03-01 | 2023-11-10 | 北京清华同衡规划设计研究院有限公司 | 基于可复用算子的村镇发展模型构建方法和系统 |
| CN115774553A (zh) * | 2021-09-07 | 2023-03-10 | 中国电子科技集团公司第十四研究所 | 一种软件化雷达算法模块图形化建模方法 |
| CN114510471A (zh) * | 2022-02-16 | 2022-05-17 | 北京九栖科技有限责任公司 | 用于大数据平台实时状态计算的方法、服务器及存储介质 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109976729A (zh) | 2019-07-05 |
| CN109976729B (zh) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109976729B (zh) | 一种存算显全局可配置的数据分析软件架构设计方法 | |
| US8122050B2 (en) | Query processing visualization system and method of visualizing query processing | |
| US11526656B2 (en) | Logical, recursive definition of data transformations | |
| US20210342738A1 (en) | Machine learning-facilitated data entry | |
| US8037000B2 (en) | Systems and methods for automated interpretation of analytic procedures | |
| US7917815B2 (en) | Multi-layer context parsing and incident model construction for software support | |
| CN114625353A (zh) | 模型框架代码生成系统及方法 | |
| CN103778107A (zh) | 一种基于excel快速动态生成表单的方法与平台 | |
| US20110208692A1 (en) | Generation of star schemas from snowflake schemas containing a large number of dimensions | |
| JP5349581B2 (ja) | 照会処理視覚化システム、照会処理を視覚化する方法及びコンピュータ・プログラム | |
| US20240403194A1 (en) | Structuring and rich debugging of inputs and outputs to large language models | |
| CN101776999A (zh) | 软件系统开发实施平台 | |
| CN114924721A (zh) | 代码生成方法、装置、计算机设备及存储介质 | |
| CN113655996A (zh) | 一种基于需求模型的企业级系统生成方法 | |
| CN118586849A (zh) | 数据处理方法 | |
| CN120085847A (zh) | 面向非技术人员的脚本智能适配与可视化编排方法及系统 | |
| CN112527918B (zh) | 一种数据处理的方法和装置 | |
| Wang et al. | Intelligent spark agents: A modular langgraph framework for scalable, visualized, and enhanced big data machine learning workflows | |
| US20250363106A1 (en) | Database instance processing method and apparatus, electronic device, computer-readable storage medium, and computer program product | |
| CN113868314A (zh) | 一种基于vue框架的调控云运行报表库系统及处理方法 | |
| US20250045620A1 (en) | Computing object having machine learning value generation method | |
| JP4954674B2 (ja) | ソフトウェア開発支援方法、ソフトウェア開発支援装置、ソフトウェア開発支援プログラム、及び計算機システム | |
| Schoenenwald et al. | Collecting and visualizing data lineage of Spark jobs: Digesting Spark execution plans to surface lineage graphs via a full-stack application | |
| CN121257500B (zh) | 基于在线表格的业务数据处理方法及装置 | |
| US20250182034A1 (en) | Hierarchical process model representation with upward callability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19927997 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19927997 Country of ref document: EP Kind code of ref document: A1 |