CN114218309A - Data processing method, system and computer equipment - Google Patents
Data processing method, system and computer equipment Download PDFInfo
- Publication number
- CN114218309A CN114218309A CN202111310394.7A CN202111310394A CN114218309A CN 114218309 A CN114218309 A CN 114218309A CN 202111310394 A CN202111310394 A CN 202111310394A CN 114218309 A CN114218309 A CN 114218309A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- platform
- business
- warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/904—Browsing; Visualisation therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to a data processing method, a data processing system and computer equipment. The method comprises the following steps: responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse; analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset data visualization mode. In the method, the data warehouse stores all data of all businesses of the enterprise, so that comprehensive and complete target business data can be acquired based on the data warehouse. Therefore, the data analysis is carried out on the target business data, the internal association and the integral characteristics among the target business data can be determined, and thus the obtained data analysis result can provide an effective basis for enterprise decision making. In addition, a preset data visualization mode is adopted to comprehensively display data analysis results in a multi-dimensional mode, so that the data display effect is more visual and clear.
Description
Technical Field
The present application relates to the field of business intelligent data management technologies, and in particular, to a data processing method, system, and computer device.
Background
The Business Intelligence (BI) system is a technology and a method for rapidly analyzing data, and is used for effectively integrating existing data in an enterprise, rapidly and accurately providing reports, providing decision bases and helping the enterprise make intelligent Business operation decisions.
BI systems generally include components for data acquisition, data analysis, and data visualization. In the related technology, the BI system extracts multi-party business data from a database in an enterprise, analyzes and processes the business data of the enterprise by using a proper query and analysis tool, a data mining tool and the like on the basis, and finally displays the processed visual data to provide support for the enterprise decision making process.
However, the collected business data in the related art cannot reliably and efficiently provide a basis for enterprise decision making.
Disclosure of Invention
In view of the above, there is a need to provide a data processing method, system and computer device capable of comprehensively collecting, effectively analyzing and displaying business intelligence data in multiple dimensions, so that business data can reliably and efficiently provide basis for enterprise decision making.
In a first aspect, the present application provides a data processing method. The method comprises the following steps:
responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and displaying the data analysis result in a preset data visualization mode.
In one embodiment, the method further comprises:
acquiring service data in each data system in at least one data system accessed in advance;
storing the service data in each data system into a data warehouse; the data warehouse comprises a plurality of data storage layers, and business data processed in different processing modes are correspondingly stored in each data storage layer.
In one embodiment, storing the business data in each data system into a data warehouse comprises:
screening the service data in each data system to obtain service processing data;
dividing the service processing data according to the service types to obtain service data sets corresponding to the service types;
storing the business data in each data system to a first data storage layer in a data warehouse; storing the business processing data to a second data storage layer in the data warehouse; and storing the service data sets corresponding to the service types into a third data storage layer in the data warehouse.
In one embodiment, the screening processing of the service data in each data system to obtain service processing data includes:
and screening the service data in each data system through data cleaning and dimensional modeling to obtain service processing data.
In one embodiment, acquiring service data in each data system in at least one data system accessed in advance includes:
acquiring initial service data in each data system in at least one data system accessed in advance;
and carrying out data cleaning and data conversion on the initial service data in each data system to obtain the service data in each data system.
In one embodiment, the data visualization manner at least comprises the following steps: visual graphic report forms and a theme cockpit.
In one embodiment, the method further comprises:
and responding to the cross-platform sharing request, sending the data analysis result to the third-party platform, and indicating the third-party platform to display on the third-party platform in a visual mode.
In a second aspect, the present application further provides a data processing apparatus. The device includes:
the data acquisition module is used for responding to the data processing request and acquiring target service data corresponding to the data processing request from the data warehouse;
the data analysis module is used for analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and the data display module is used for displaying the data analysis result in a preset data visualization mode.
In a third aspect, the present application further provides a data processing system, including: a business intelligence platform and at least one data system; at least one data system is connected with the business intelligent platform; the business intelligent platform comprises a data warehouse, wherein data in the data warehouse are acquired by the business intelligent platform from at least one data system;
the business intelligent platform is used for responding to the data processing request, acquiring target business data corresponding to the data processing request from the data warehouse, and analyzing the target business data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset visual mode.
In one embodiment, the business intelligence platform comprises a cross-platform management module and a data presentation module;
the cross-platform management module is used for responding to the cross-platform sharing request, sending the data analysis result to the third-party platform and indicating the third-party platform to display on the third-party platform in a visual mode; the cross-platform sharing request is triggered by a user based on a visual data sharing link in a cross-platform management module;
and the data display module is used for displaying the data analysis result in a visual mode.
In a fourth aspect, the present application further provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the method of any of the above-described first aspect when executing the computer program.
In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any of the embodiments of the first aspect described above.
In a sixth aspect, the present application further provides a computer program product. The computer program product comprising a computer program that, when executed by a processor, performs the steps of the method of any of the embodiments of the first aspect described above.
According to the data processing method, the data processing system and the computer equipment, the target business data corresponding to the data processing request is acquired from the data warehouse by responding to the data processing request; analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset data visualization mode. In the method, the data warehouse stores the business data of all the businesses of the enterprise, so that comprehensive and complete target business data can be acquired based on the data warehouse. And then data analysis is carried out based on the target business data, internal association and integral characteristics among the business data can be analyzed more accurately, and thus, the obtained data analysis result can provide effective basis for enterprise decision making. In addition, the data analysis result is displayed in a preset data visualization mode, so that the data analysis result can be displayed visually, clearly and comprehensively, and a user can check the data analysis result conveniently.
Drawings
FIG. 1 is a diagram of an application environment of a data processing method in one embodiment;
FIG. 2 is a flow diagram illustrating a data processing method according to one embodiment;
FIG. 3 is a schematic flow chart diagram of a data warehouse construction method in one embodiment;
FIG. 4 is a flow chart illustrating a data processing method according to another embodiment;
FIG. 5 is a flow chart illustrating a data processing method according to another embodiment;
FIG. 6 is a schematic diagram of a business intelligence platform in one embodiment;
FIG. 7 is a flow diagram illustrating data processing according to one embodiment;
FIG. 8 is a flow chart illustrating a data processing method according to another embodiment;
FIG. 9 is a block diagram showing the structure of a data processing apparatus according to an embodiment;
FIG. 10 is a block diagram of a data processing system in one embodiment;
FIG. 11 is a schematic diagram of a data processing system in another embodiment;
FIG. 12 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data processing method provided by the application can be applied to the application environment shown in fig. 1, the computer equipment in the application environment can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, internet of things equipment and portable wearable equipment, and the internet of things equipment can be smart speakers, smart televisions, smart air conditioners, smart vehicle-mounted equipment and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like.
The internal structure of the computer device is shown in fig. 1, and a processor in the internal structure is used for providing data calculation and analysis functions. The memory in the internal structure includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database may be used to store business data for an enterprise. The network interface is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.
In the related art, data processing of an enterprise is realized by using a Business Intelligence (BI) platform. BI is a technique and method for rapidly analyzing data, including collecting, managing, and analyzing data, transforming the data into useful information, and then distributing the information throughout the enterprise. BI enables organizations to take more flexible business decisions and makes businesses more competitive. For example, a business may use a BI application or technique to infer information from relevant indicators of the external environment and to predict prospective trends of its business segments. BI applications can also be used to improve the timing and quality of information, enabling managers to better understand the market positioning of their companies relative to competitors.
Based on the advantages, the BI platform has good development prospect. Particularly in the aspect of finance, a BI tool provided in a BI platform can help an enterprise to quickly classify data sources to obtain related data such as sales, management, manpower and the like. Furthermore, business analysis is carried out by using the data, so that financial services are further optimized, and data support is provided for enterprise decisions.
However, the BI platform in the related art has the following drawbacks when performing data analysis:
(1) the BI tool does not support a business system directly accessed into the enterprise, and cannot acquire internal data from the business system;
(2) when mass data are processed, the data cannot be efficiently and comprehensively analyzed by using a single-machine database, and the data processing efficiency is low;
(3) the BI tool cannot realize cross-terminal operation, cannot access multiple terminals, and cannot further operate data analysis results at different terminals;
(4) the BI tool provides a single data visualization mode, so that a data analysis result cannot be comprehensively displayed in a multi-dimension mode, and the data visualization result cannot be shared among multiple platforms.
Based on the data processing method, the data processing system and the computer equipment, the BI platform is improved, and the data in the enterprise is extracted from the multi-data source based on the improved BI platform so as to construct a data warehouse. And the data is comprehensively analyzed based on the data warehouse, so that the data analysis result can provide reliable data basis for enterprise decision making. Therefore, from the aspect of data analysis, the data processing method provided by the application can convert the business data in the enterprise into knowledge and help the enterprise to make an intelligent business operation decision.
Next, the technical solutions of the embodiments of the present application, and how to solve the above technical problems will be specifically described in detail through embodiments and with reference to the accompanying drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, in the data processing method provided in the embodiment of the present application, the execution main body may be a computer device, specifically, a BI platform deployed and normally operating on the computer device; the device can also be a background business server of an enterprise, or a data processing device, and the device can be realized as a part or all of the processor in a software, hardware or software and hardware combination mode. It is to be understood that the embodiments described are only some of the embodiments of the present application and not all of them.
In one embodiment, as shown in FIG. 2, a data processing method is provided, which is illustrated by applying the method to the computer device deployed with the BI platform in FIG. 1, and includes the following steps:
step 210: and responding to the data processing request, and acquiring target business data corresponding to the data processing request from the data warehouse.
The Data Warehouse (DW) is a theme-oriented, integrated, relatively stable Data set reflecting historical changes, and is a strategic set providing all types of Data support for decision making processes of all levels of an enterprise.
The theme-oriented data is that data in a data warehouse is organized according to a certain theme domain, such as: customers, products, transactions, accounts, etc.; integration means that original dispersed database data is processed and sorted by a system to eliminate inconsistency in source data; the relatively stable data means that once a certain data enters a data warehouse, the data only needs to be loaded and refreshed regularly without being imported repeatedly; reflecting historical changes means that quantitative analysis prediction can be made on development history and future trends of enterprises through data information.
It should be noted that the main differences between the data warehouse and the database are as follows: the database is designed facing to affairs and stores general online transaction data; the data warehouse is designed facing to subject, and generally stores historical data so as to be convenient for performing integrated analysis on the historical data and provide a data base for application. The data warehouse is not used for replacing the database, and the data is comprehensively and efficiently summarized and analyzed so as to provide corresponding data services.
In a possible implementation manner, the data processing request carries at least one item of information used for describing the target service data, such as a service department, a service index, a data type, a data generation date, and the like. Therefore, after the user triggers the data processing request in the BI platform, the BI platform obtains the target service data corresponding to the data processing request from the data warehouse according to the data processing request.
Step 220: and analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result.
In one possible implementation manner, the BI platform analyzes the target business data by using an On-Line Analytical Processing (OLAP) technology, and obtains a data analysis result by performing multi-dimensional comprehensive analysis On the target business data to obtain the internal relation and the overall data characteristics of the target business data.
The OLAP can enable an analyst to quickly, consistently and interactively observe information from various aspects, so as to achieve the purpose of deeply understanding data. It features Fast Analysis of Shared Multi-dimensional Information (FASMI). F is rapidity, which means that OLAP can react to most analysis requirements of users within seconds; a is analyzability (Analysis), meaning that the user can define new specialized calculations without programming, as part of the Analysis, and give reports in the way the user wants; m is multidimensional (Multi-dimensional), which refers to a multidimensional view that provides analysis of data; i is informativeness (Information), which means that Information can be obtained in time and large-capacity Information is managed.
Step 230: and displaying the data analysis result in a preset data visualization mode.
The data visualization mode provided by the BI platform at least comprises the following steps: visual graphic report forms and a theme cockpit. Visual graphical reports include, but are not limited to: detail tables, indicator cards, bar charts, pie charts, line charts, rectangular tree diagrams, funnel diagrams, tree diagrams, dashboards, four-quadrant diagrams and mind maps; the theme cockpit adopts various visual graphic reports to visually display the total change condition of a certain theme.
Therefore, the preset data visualization mode may be one or more selected from the data visualization modes by the user, or may be all data visualization modes provided by the BI platform, which is not limited in the embodiment of the present application.
As one example, the subject cockpit may include: a high-rise cockpit for reflecting internal and external overall operation conditions and trends, a marketing cockpit for reflecting marketing overall change conditions, a product cockpit for reflecting product overall conditions and the like.
It should be noted that the BI platform in the present application provides a plurality of data visualization ways. In specific implementation, target business data can be obtained from data sets of different themes in a data warehouse according to user requirements, and an analysis report, a visual graphical report and a theme cockpit are made in a BI platform.
In the embodiment of the application, in response to a data processing request, target business data corresponding to the data processing request is acquired from a data warehouse; analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset data visualization mode. In the method, the data warehouse stores the business data of all the businesses of the enterprise, so that comprehensive and complete target business data can be acquired based on the data warehouse. And then data analysis is carried out based on the target business data, internal association and integral characteristics among the business data can be analyzed more accurately, and thus, the obtained data analysis result can provide effective basis for enterprise decision making. In addition, the data analysis result is displayed in a preset data visualization mode, so that the data analysis result can be displayed visually, clearly and comprehensively, and a user can check the data analysis result conveniently.
Based on the foregoing embodiments, in an embodiment, as shown in fig. 3, the present application further provides a data warehouse building method, which is also described by taking as an example that the method is applied to a computer device deployed with a BI platform in fig. 1, and includes the following steps:
step 310: and acquiring service data in each data system in at least one data system accessed in advance.
The method comprises the steps that at least one data system in an enterprise is accessed into a BI platform in advance, so that the BI platform can acquire all business data of the enterprise from the at least one data system when a data warehouse is created, and the comprehensiveness and integrity of the business data are guaranteed from a data source.
As one example, at least one data system includes: office Automation (OA) systems, application systems, third-party data provider systems, and proprietary data systems for enterprises. When the system operates, the generated service data is stored according to a certain rule.
As another example, to extend data collection dimensions and granularity, at least one data system may also include various fill-in type data generated within an enterprise. Such as: documents and tables which are imported by staff in the enterprise, financial statements generated in the business handling process, enterprise qualification honor libraries, human resource cost reports and the like.
Optionally, when the specific deployment is implemented, a form similar to a ledger in the work of an individual or a department in an enterprise can be made into a filling module through a BI platform, the filling module of the BI is opened instead of opening a form file every time, corresponding service data is filled, and the service data filled in real time is directly stored and updated in a data warehouse of the BI platform.
In a possible implementation manner, the implementation procedure of the step 310 may be: acquiring initial service data in each data system in at least one data system accessed in advance; and carrying out data cleaning and data conversion on the initial service data in each data system to obtain the service data in each data system.
As an example, the data extraction tool may be utilized to completely extract data in at least one data system, resulting in initial business data. Further, an Extract-Transform-Load (ETL) technology is adopted to perform data cleaning and data conversion on the initial service data, so as to obtain service data in each data system.
The data cleaning is to carry out consistency detection on the extracted initial service data according to a standard, and delete or supplement invalid values and missing values; the data conversion is to perform standard conversion on the cleaned service data according to requirements so as to unify the format of the service data.
Step 320: storing the service data in each data system into a data warehouse; the data warehouse comprises a plurality of data storage layers, and business data processed in different processing modes are correspondingly stored in each data storage layer.
In a possible implementation manner, the cleaned and converted service data is loaded into a unified storage space of a data warehouse, and then the service data is processed and stored layer by layer according to a data storage layer set in the data warehouse and a processing manner corresponding to each data storage layer, so that the warehousing operation of the service data in at least one data system is realized.
In this embodiment, the BI platform is directly connected to at least one data system in the enterprise, so that comprehensive and complete business data can be obtained from the at least one data system, and a rich data source is provided for creating a data warehouse. In addition, the business data acquired from at least one data system is preprocessed by adopting an ETL technology, so that the acquired business data are uniformly stored in a warehouse according to a standard format, and the data storage efficiency is improved.
Based on the data warehouse in the above embodiments, in one embodiment, the data warehouse includes three layers: the business data processing system comprises a first data storage layer, a second data storage layer and a third data storage layer, wherein the first data storage layer, the second data storage layer and the third data storage layer correspondingly store business data processed in different processing modes.
As an example, the data processing and storing are performed in a layer-by-layer progressive manner, that is, the data in the first data storage layer is processed first, the obtained data is stored in the second data storage layer, and then the data in the second data storage layer is processed, and the obtained data is stored in the third data storage layer.
Based on this, as shown in fig. 4, the step 320 of storing the business data in each data system into the data warehouse includes the following steps:
step 410: and screening the service data in each data system to obtain service processing data.
In one possible implementation, the business data in each data system is screened through data cleaning and dimensional modeling to obtain business processing data.
As an example, the cleaning is to delete or supplement invalid values and missing values of the business data in each data system according to time nodes, and the dimensional modeling is to classify and summarize the business data in each data system according to different dimensions.
Step 420: and dividing the service processing data according to the service types to obtain service data sets corresponding to the service types.
The service data set is authoritative data obtained by further processing the service processing data, and the service data set is used for providing a corresponding data basis for data application.
It should be noted that an enterprise includes multiple business departments, and different business departments have different business data to be called when making decisions. Therefore, in order to provide data services better, the business process data needs to be divided in the data warehouse according to at least one business corresponding to the business process data. Since there is an association between services, one service data may be divided into one service data set or may be divided into a plurality of service data sets.
That is, in step 420, the service processing data is divided according to the service types to obtain a plurality of service data sets, and each service data set corresponds to one topic.
As an example, the resulting multi-service data set comprises: the system comprises a human resource theme data set, an operation management theme data set, a financial management and control data set and the like, and the number of the divided service data sets is not limited in the embodiment of the application.
Step 430: storing the business data in each data system to a first data storage layer in a data warehouse; storing the business processing data to a second data storage layer in the data warehouse; and storing the service data sets corresponding to the service types into a third data storage layer in the data warehouse.
In one possible implementation, the first Data storage tier is an Operational Data Store (ODS) tier, the second Data storage tier is a Data Warehouse (DW) tier, and the third Data storage tier is a Data Mart (DM) tier.
That is, in step 430, the service data in each data system is stored in the ODS layer, the service data in the ODS layer is filtered, the filtered service processing data is stored in the DW layer, and further, the data in the DW layer is divided according to the service types to obtain a plurality of service data sets, each service data set corresponds to a service, and the plurality of data sets are stored in the DM layer.
The ODS layer stores original data after data conversion, temporary data of a data exchange process, and detail-level, current and near-real-time service data. The near real-time service data refers to service data within 6 months at present. The ODS layer is used as a data processing transition area for source data acquisition, shields the core layer of the data warehouse as much as possible for the difference of different services, reduces the complexity of ETL design, processing and scheduling, and improves the ETL performance.
It should be noted that the ODS layer does not provide an external large data volume query and retrieval function, but provides a large data volume query and retrieval function at the data warehouse level, thereby fully exerting the advantages of the data warehouse in terms of large data volume data retrieval performance.
In addition, the data of the DW layer is final authoritative data, and the data content is permanently stored. That is, the data of the DW layer has only new operations, and has no change or deletion operations. The DM layer is also used for generating a wide table with more fields according to the service data set corresponding to each service type, providing subsequent operations such as service query, OLAP analysis and data distribution, and providing data service for the application.
In this embodiment, the service data in each data system is processed and stored layer by layer to store the service data in each data system into a three-layer data warehouse, and a data warehouse wide table is generated based on a plurality of service data sets generated in the DM layer. Therefore, a comprehensive data set can be provided for a user through the data warehouse, and an efficient data query mode is provided, so that data application is supported more highly.
Based on the above embodiments, in an embodiment, as shown in fig. 5, after displaying the data analysis result in a preset data visualization manner, the data processing method provided by the present application further includes the following steps:
step 240: and responding to the cross-platform sharing request, sending the data analysis result to the third-party platform, and indicating the third-party platform to display on the third-party platform in a visual mode.
The BI platform in the application supports cross-platform sharing of data analysis results. Specifically, a visual data sharing link is set in the BI platform in advance and is used for providing a cross-platform sharing function for a user.
In one possible implementation manner, after the BI platform displays the data analysis result in a data visualization manner in the display interface, the BI platform monitors the cross-platform sharing operation of the user. And after the user triggers the visual data sharing link in the BI platform, the BI platform displays the supported third-party platform and is selected by the user. And after the user selects the target platform, sending the data analysis result displayed in the BI platform to the target platform.
It should be noted that, in the process of sharing data analysis results across platforms, the data visualization mode is not changed. That is, if the data analysis result is displayed in the preset data visualization manner in the BI platform, after the data analysis result is shared with the third party platform, the data analysis result is still displayed in the same display manner as that in the BI platform, and the display results viewed by the user in the BI platform and the third party platform are the same.
Based on the cross-platform sharing function, the user can adopt a Personal Computer (PC) terminal and a mobile terminal to check the data analysis result and further operate the data analysis result.
In this embodiment, after the data analysis result obtained by the analysis of the BI platform is displayed, the data analysis result can be shared among multiple platforms, so that the user can view the data analysis result from multiple ends, further operate the data analysis result, and improve user experience.
In combination with the above embodiments, the data processing method provided by the present application is explained in detail with reference to fig. 6 to 8.
As shown in FIG. 6, the BI platform provided by the present application includes three levels of data acquisition, data warehousing, and data application. Wherein the data collection comprises a plurality of data sources: system data/databases, documents, tables, and other data, the data sources to include as much as possible all business data within the enterprise.
Furthermore, for all the collected service data, real-time data calculation and offline data calculation can be performed on the data clusters formed by the collected service data. The calculated data can be stored in a data warehouse in a layered mode, can be used for task scheduling, and can further analyze data blood relationship and data quality. The embodiment of the application does not limit the data calculation mode in the big data cluster and the application scene of the calculated data.
In addition, the data warehouse may include a data application layer in addition to the ODS layer, the DW layer, and the DM layer. The data application layer may provide OLAP data analysis to support data applications. Among these, data applications include, but are not limited to: data services, multidimensional data analysis and data modeling, data visualization, machine learning/deep learning, and other applications.
In addition, referring to fig. 7, the present application also provides a schematic diagram of a data processing flow. For the created data warehouse, in the actual application, the target business data can be called from the data warehouse, the target business data is analyzed, and the data analysis result is displayed according to a preset data visualization mode.
Wherein, the data application may be: and creating different data models for the data of the DM layer according to different analysis requirements of business departments so as to obtain data results required by related businesses. Therefore, corresponding data preparation is generated in advance according to business departments for users to inquire and call, and then the data preparation is displayed in the BI platform.
Based on the BI platform and the data processing flow, the present application further provides another data processing method, which is described by taking the method as an example for being applied to the computer device in fig. 1, and referring to fig. 8, the method includes the following steps:
step 810: acquiring initial service data in each data system in at least one data system accessed in advance;
step 820: performing data cleaning and data conversion on the initial service data in each data system to obtain service data in each data system;
step 830: screening the service data in each data system to obtain service processing data;
step 840: dividing the service processing data according to the service types to obtain service data sets corresponding to the service types;
step 850: storing the business data in each data system to a first data storage layer in a data warehouse; storing the business processing data to a second data storage layer in the data warehouse; storing the service data sets corresponding to the service types to a third data storage layer in a data warehouse;
step 860: responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
step 870: analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
step 880: displaying the data analysis result in a preset data visualization mode;
step 890: and responding to the cross-platform sharing request, sending the data analysis result to the third-party platform, and indicating the third-party platform to display on the third-party platform in a visual mode.
The implementation principle and technical effect of each step in the data processing method provided in this embodiment are similar to those in the foregoing embodiments, and are not described herein again.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a data processing apparatus for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more embodiments of the data processing device provided below may refer to the limitations on the data processing method in the above description, and are not described herein again.
In one embodiment, as shown in fig. 9, there is provided a data processing apparatus 900 comprising: a data acquisition module 910, a data analysis module 920, and a data presentation module 930, wherein:
a data obtaining module 910, configured to, in response to a data processing request, obtain target service data corresponding to the data processing request from a data warehouse;
the data analysis module 920 is configured to analyze the target service data based on a data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and a data display module 930, configured to display the data analysis result in a preset data visualization manner.
In one embodiment, the apparatus 900 further comprises:
the data obtaining module 910 is further configured to obtain service data in each data system in at least one data system that is accessed in advance;
the storage module is used for storing the service data in each data system into a data warehouse; the data warehouse comprises a plurality of data storage layers, and business data processed in different processing modes are correspondingly stored in each data storage layer.
In one embodiment, a memory module includes:
the screening unit is used for screening the service data in each data system to obtain service processing data;
the dividing unit is used for dividing the service processing data according to the service types to obtain service data sets corresponding to the service types;
the storage unit is used for storing the business data in each data system to a first data storage layer in the data warehouse; storing the business processing data to a second data storage layer in the data warehouse; and storing the service data sets corresponding to the service types into a third data storage layer in the data warehouse.
In one embodiment, the screening unit is specifically configured to:
and screening the service data in each data system through data cleaning and dimensional modeling to obtain service processing data.
In one embodiment, the data obtaining module 910 includes:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring initial service data in each data system in at least one data system accessed in advance;
and the preprocessing unit is used for carrying out data cleaning and data conversion on the initial service data in each data system to obtain the service data in each data system.
In one embodiment, the data visualization manner at least comprises the following steps: visual graphic report forms and a theme cockpit.
In one embodiment, the apparatus 900 further comprises:
and the sharing module is used for responding to the cross-platform sharing request, sending the data analysis result to the third-party platform and indicating the third-party platform to display on the third-party platform in a visual mode.
The various modules in the data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Based on the same inventive concept, the embodiment of the present application further provides a data processing system for implementing the above-mentioned data processing method.
In one embodiment, as shown in FIG. 10, a data processing system is provided, the system 1000 comprising: a business intelligence platform 1010 and at least one data system 1020; at least one data system 1020 is connected to the business intelligence platform 1010; the business intelligence platform 1010 includes a data repository 1011, data in the data repository 1011 being obtained by the business intelligence platform from at least one data system 1020;
the business intelligent platform 1010 is used for responding to the data processing request, acquiring target business data corresponding to the data processing request from the data warehouse 1011, and analyzing the target business data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset visual mode.
In one embodiment, as shown in FIG. 11, in the data processing system 1000, the business intelligence platform 1010 includes a cross-platform management module 1012 and a data exposure module 1013;
the cross-platform management module 1012 is configured to respond to the cross-platform sharing request, send a data analysis result to a third-party platform, and instruct the third-party platform to display the data analysis result on the third-party platform in a visual manner; the cross-platform sharing request is triggered by a user based on a visual data sharing link in a cross-platform management module;
and the data display module 1013 is configured to display the data analysis result in a visualization manner.
It should be understood that the technical solution executed by the business intelligence platform in the system is similar to the implementation solution described in the above method, and reference may be made to the above explanation and limitation of the data processing method, which is not described herein again.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 12. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 12 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and displaying the data analysis result in a preset data visualization mode.
The principle and technical effect of the data processing method in the present application are similar to those of the above method embodiments, and are not described herein again.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and displaying the data analysis result in a preset data visualization mode.
The foregoing embodiments provide a computer-readable storage medium, which implements the principles and technical effects of the data processing method in the present application similar to those of the foregoing embodiments, and therefore, the details are not repeated herein.
In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps of:
responding to the data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and displaying the data analysis result in a preset data visualization mode.
The principle and technical effect of the data processing method in the present application are similar to those of the above method embodiments, and are not described herein again.
It should be noted that the enterprise information (including but not limited to enterprise device information, personal information of employees in the enterprise, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) referred to in this application are information and data authorized by the enterprise or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data processing, the method comprising:
responding to a data processing request, and acquiring target business data corresponding to the data processing request from a data warehouse;
analyzing the target service data based on the data analysis requirement corresponding to the data processing request to obtain a data analysis result;
and displaying the data analysis result in a preset data visualization mode.
2. The method of claim 1, further comprising:
acquiring service data in each data system in at least one data system accessed in advance;
storing the business data in each data system into a data warehouse; the data warehouse comprises a plurality of data storage layers, and business data processed in different processing modes are correspondingly stored in each data storage layer.
3. The method of claim 2, wherein storing the business data in each of the data systems into a data repository comprises:
screening the service data in each data system to obtain service processing data;
dividing the service processing data according to service types to obtain service data sets corresponding to the service types;
storing the business data in each data system to a first data storage layer in the data warehouse; storing the business process data to a second data storage layer in the data repository; and storing the service data sets corresponding to the service types to a third data storage layer in the data warehouse.
4. The method according to claim 3, wherein the screening the service data in each of the data systems to obtain service processing data includes:
and screening the service data in each data system through data cleaning and dimensional modeling to obtain the service processing data.
5. The method according to claim 3 or 4, wherein the obtaining of the service data in each data system in at least one data system accessed in advance comprises:
in at least one data system accessed in advance, acquiring initial service data in each data system;
and carrying out data cleaning and data conversion on the initial service data in each data system to obtain the service data in each data system.
6. The method according to any of claims 1-4, wherein the data visualization means comprises at least: visual graphic report forms and a theme cockpit.
7. The method according to any one of claims 1-4, further comprising:
responding to a cross-platform sharing request, sending the data analysis result to a third-party platform, and indicating the third-party platform to display on the third-party platform in the visual mode.
8. A data processing system, the system comprising: a business intelligence platform and at least one data system; the at least one data system is connected with the business intelligence platform; the business intelligence platform comprises a data warehouse, wherein data in the data warehouse is acquired by the business intelligence platform from the at least one data system;
the business intelligent platform is used for responding to a data processing request, acquiring target business data corresponding to the data processing request from the data warehouse, and analyzing the target business data based on a data analysis requirement corresponding to the data processing request to obtain a data analysis result; and displaying the data analysis result in a preset visual mode.
9. The system of claim 8, wherein the business intelligence platform comprises a cross-platform management module and a data presentation module;
the cross-platform management module is used for responding to a cross-platform sharing request, sending the data analysis result to a third-party platform and indicating the third-party platform to display on the third-party platform in the visual mode; the cross-platform sharing request is triggered by a user based on a visual data sharing link in a cross-platform management module;
and the data display module is used for displaying the data analysis result according to the visualization mode.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111310394.7A CN114218309A (en) | 2021-11-04 | 2021-11-04 | Data processing method, system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111310394.7A CN114218309A (en) | 2021-11-04 | 2021-11-04 | Data processing method, system and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114218309A true CN114218309A (en) | 2022-03-22 |
Family
ID=80696581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111310394.7A Pending CN114218309A (en) | 2021-11-04 | 2021-11-04 | Data processing method, system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114218309A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817338A (en) * | 2022-06-28 | 2022-07-29 | 杭州湖畔网络技术有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN115730605A (en) * | 2022-11-21 | 2023-03-03 | 刘奕涵 | Data analysis method based on multi-dimensional information |
| CN115794804A (en) * | 2023-02-07 | 2023-03-14 | 北京至臻云智能科技有限公司 | Engineering internal control data visualization processing system and method based on big data technology |
| CN115794044A (en) * | 2023-01-31 | 2023-03-14 | 帆软软件有限公司帆软南京分公司 | Analysis theme system and analysis theme display method of BI tool |
| CN116383299A (en) * | 2023-03-31 | 2023-07-04 | 国任财产保险股份有限公司 | Data display system based on distributed database |
-
2021
- 2021-11-04 CN CN202111310394.7A patent/CN114218309A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817338A (en) * | 2022-06-28 | 2022-07-29 | 杭州湖畔网络技术有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN115730605A (en) * | 2022-11-21 | 2023-03-03 | 刘奕涵 | Data analysis method based on multi-dimensional information |
| CN115730605B (en) * | 2022-11-21 | 2024-02-02 | 暨南大学 | Data analysis method based on multidimensional information |
| CN115794044A (en) * | 2023-01-31 | 2023-03-14 | 帆软软件有限公司帆软南京分公司 | Analysis theme system and analysis theme display method of BI tool |
| CN115794804A (en) * | 2023-02-07 | 2023-03-14 | 北京至臻云智能科技有限公司 | Engineering internal control data visualization processing system and method based on big data technology |
| CN116383299A (en) * | 2023-03-31 | 2023-07-04 | 国任财产保险股份有限公司 | Data display system based on distributed database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109272155B (en) | Enterprise behavior analysis system based on big data | |
| CN114218309A (en) | Data processing method, system and computer equipment | |
| US9152947B2 (en) | Real-time social networking | |
| Zhang et al. | Visual analytics for the big data era—A comparative review of state-of-the-art commercial systems | |
| Elgendy et al. | Big data analytics: a literature review paper | |
| US9164990B2 (en) | Annotating contextual workspaces | |
| US8583678B2 (en) | Graphical exploration of a database | |
| McLachlan et al. | LiveRAC: interactive visual exploration of system management time-series data | |
| Pereira et al. | A platform for integrating heterogeneous data and developing smart city applications | |
| US20130139081A1 (en) | Viewing previous contextual workspaces | |
| CN103678468B (en) | The system and method for improved consumption model for profiling | |
| CN114090838B (en) | Method, system, electronic device and storage medium for visually displaying big data | |
| CN113157947A (en) | Knowledge graph construction method, tool, device and server | |
| US20210390496A1 (en) | Method for model-based project scoring classification and reporting | |
| CN107609040A (en) | Data analysing method, data processing equipment and computer-readable recording medium | |
| Parush et al. | Impact of visualization type and contextual factors on performance with enterprise resource planning systems | |
| CN112633621A (en) | Power grid enterprise management decision system and method based on PAAS platform | |
| CN112035555A (en) | Information display method, device and equipment | |
| Yurchuk | Features of business intelligence development in the conditions of digital transformations | |
| Menzel et al. | Context-sensitive process and data management on mobile devices | |
| KR20080112141A (en) | Construction project performance management method and system using bsc and the data warehouse | |
| US20140143278A1 (en) | Application programming interface layers for analytical applications | |
| Chaudhari et al. | Visualization of Uncertainties and Noise in Dark Data: Methods & Techniques | |
| US20220270024A1 (en) | Method and system to predict stakeholder project impact | |
| Choudhary et al. | Enhancing the website usage using process mining |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |