TW201947492A - System and method for operational data convergence - Google Patents

System and method for operational data convergence Download PDF

Info

Publication number
TW201947492A
TW201947492A TW107116271A TW107116271A TW201947492A TW 201947492 A TW201947492 A TW 201947492A TW 107116271 A TW107116271 A TW 107116271A TW 107116271 A TW107116271 A TW 107116271A TW 201947492 A TW201947492 A TW 201947492A
Authority
TW
Taiwan
Prior art keywords
data
information
area
converging
real
Prior art date
Application number
TW107116271A
Other languages
Chinese (zh)
Inventor
郭健男
沈仁傑
王聖文
Original Assignee
玉山商業銀行股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 玉山商業銀行股份有限公司 filed Critical 玉山商業銀行股份有限公司
Priority to TW107116271A priority Critical patent/TW201947492A/en
Publication of TW201947492A publication Critical patent/TW201947492A/en

Links

Abstract

The disclosure is related to a system for operational data convergence. The system adopts a three-tier data storage structure. A high-performance relational database is used to implement an original data zone for providing original data. A big data platform with high price-performance ratio storing and distributed processing capability is used to implement a data value-added zone for synchronizing data with the relational database in real time, and data collection. A low-cost file server is used to implement a data storage zone for data storage and auditing. Further, an information system is provided to be a data transmission tool between the relational database and the big data platform. A software method is introduced to monitor the flow change of the relational database and inform the change to the information system. The changed data stored in the information system is then written into the big data platform in sequence for data synchronization.

Description

運營資料匯流系統與方法    Operational data converging system and method   

說明書公開一種資訊系統,特別是一種藉由三層式的資料儲存架構實現的運營資料匯流系統,以及其中運行方法。 The specification discloses an information system, in particular an operational data convergence system implemented by a three-tiered data storage architecture, and an operating method therein.

金融機構如銀行、證券公司等,經常性地產生大量的資訊,資訊處理需要強而有力的資訊系統,然而,如果有需要即時處理的資訊,卻因為資訊系統效率低落而無法即時處理,即無法讓使用者取得即時而有用的資訊。 Financial institutions such as banks and securities companies often generate a large amount of information. Information processing requires a powerful information system. However, if there is information that needs to be processed in real time, it cannot be processed in real time because of the low efficiency of the information system. Give users instant and useful information.

在金融交易中需要跨系統交換資料,這類系統資料交換多透過批次檔案方式提供,若因為資訊時效性低將無法有效提供業務加值運用。舉例來說,若有消費者在甲地以信用卡消費達一額度,銀行將給與當地消費的優惠,卻因為系統的問題無法即時處理這筆交易與即時遞送優惠訊息,當消費者已經離開甲地,優惠訊息才到達消費者,即無法優惠任何事情。 In financial transactions, it is necessary to exchange data across systems. Such system data exchange is mostly provided through batch files. If the timeliness of information is low, it will not be able to effectively provide business value-added applications. For example, if a consumer spends up to a credit card in A, the bank will give a discount on local consumption. However, because of a system problem, the transaction cannot be processed immediately and the preferential information is delivered immediately. When the consumer has left A In order to reach the consumer, the discount message cannot reach anything.

更者,除了以上即時資訊的處理問題外,在大量歷史資料的運用、查詢與備存上,由於金融機構的各種資料來自不同的系統,資料也與日俱增,會使得其中資訊系統架構複雜,各系統資料內容記載資訊結構上不一致且無統一系統管理,不僅造成跨系統資料整合應用困難,且資料分析多仰賴過往經驗,內部溝通須耗費大量時間,資料正確性亦需花大量人力確認,調閱作業也相形複 雜與費時。如此將產生大量耗時耗人力的工作,增加營運成本。 In addition, in addition to the above-mentioned real-time information processing issues, in the use, query and storage of a large amount of historical data, as various data of financial institutions come from different systems, the data is also increasing day by day, which will make the information system architecture complicated and each system The information structure of the data content is inconsistent and there is no unified system management, which not only causes difficulties in the integration and application of cross-system data, but also depends on past experience for data analysis. Internal communication requires a lot of time, and the correctness of the data also requires a lot of manpower to confirm and review the operation It is also complicated and time consuming. This will generate a lot of time-consuming and labor-intensive work and increase operating costs.

揭露書提出一種運營資料匯流系統,以及其中運作方法,運營資料匯流系統主要採用一種三層式的資料儲存架構,主要有以伺服系統軟體程序與資料庫實現的原始資料區、資料加值區以及資料備存區。 The disclosure proposes an operating data convergence system and its operation method. The operating data convergence system mainly uses a three-tiered data storage architecture. It mainly includes the original data area, data value-added area implemented by software programs and databases of the server system, and Data storage area.

其中,根據一實施例,所述運營資料匯流系統分別以高效資料處理的關聯式資料庫實現原始資料區,以提供原始資料服務;由高性價比存儲及分散處理的大數據平台實現資料加值區,即時同步關聯式資料庫的資料,以提供加值彙整服務;以及由低成本庫存的檔案伺服器實現資料備存區,用以提供備存與稽核服務。 Wherein, according to an embodiment, the operational data converging system implements the original data area with an associated data database for efficient data processing to provide the original data service; the data value-added area is implemented by a big data platform with cost-effective storage and decentralized processing , Real-time synchronization of the data in the relational database to provide value-added integration services; and a data storage area implemented by a low-cost inventory file server to provide backup and audit services.

進一步地,運營資料匯流系統以一訊息系統作為關聯式資料庫與大數據平台間的資料傳送工具,且關聯式資料庫與訊息系統之間執行一訊息提供程式,能將關聯式資料庫中的表格(table)依照鍵值(key)視為字串流,並能即時偵測關聯式資料庫中數據流變化,並將變化傳送至訊息系統。 Further, the operating data convergence system uses a message system as a data transmission tool between the relational database and the big data platform, and a message provider is executed between the relational database and the message system, which can transfer the information in the relational database. The table is treated as a stream according to the key, and it can detect changes in the data stream in the relational database in real time and send the changes to the message system.

更者,所述訊息系統與大數據平台之間可執行一訊息寫入程式,用以將訊息系統中留存的異動資料依序寫入大數據平台,以達成資料同步。 Furthermore, a message writing program can be executed between the information system and the big data platform to sequentially write the transaction data retained in the information system to the big data platform to achieve data synchronization.

運營資料匯流系統取得數據的方式是收錄外部或內部原始資料,包括可以即時同步作業,或是以一整批資料收錄的作業收錄資料。其中可包括轉換與清理資料的程序,以提供高品質資料。 The way of operating the data converging system to obtain data is to collect external or internal raw data, including data that can be synchronized in real time or collected in a batch of data. This can include procedures for converting and cleaning data to provide high-quality data.

進一步地,運營資料匯流系統中的資料服務中,可以通過資料加值區的功能提供多樣性內容,並通過資料提供的功能提供資料查詢、訂閱與整批資料提供的服務。 Further, in the data service in the operation data converging system, it is possible to provide diversified content through the function of the data value-added area, and through the data provision function to provide the services of data inquiry, subscription, and the provision of a batch of data.

在運營資料匯流方法的實施例中,通過系統取得外部資訊或內部資訊,經一整批資料收錄功能對外部資訊或內部資訊執行資 料轉換或資料清理,以形成一資料加值區中的非結構資料,或直接以經資料整合運算後,成為商業智慧應用中自主分析來源,並以一視覺化呈現給使用者。其中,內部資訊經即時同步作業同步到原始資料區,作為原始資料查詢之用,也可用於資料加值區即時串流分析之用。 In the embodiment of the operation data converging method, external or internal information is obtained through the system, and data conversion or data cleaning is performed on the external or internal information through a batch of data collection functions to form a non-structure in a data value-added area. The data, or directly integrated with the data, becomes the source of autonomous analysis in business intelligence applications, and is presented to the user in a visualization. Among them, the internal information is synchronized to the original data area through the real-time synchronization operation, for the purpose of querying the original data, and also for the real-time streaming analysis of the data value-added area.

為了能更進一步瞭解本發明為達成既定目的所採取之技術、方法及功效,請參閱以下有關本發明之詳細說明、圖式,相信本發明之目的、特徵與特點,當可由此得以深入且具體之瞭解,然而所附圖式僅提供參考與說明用,並非用來對本發明加以限制者。 In order to further understand the technology, methods and effects adopted by the present invention to achieve the intended purpose, please refer to the following detailed description and drawings of the present invention. It is believed that the purpose, features and characteristics of the present invention can be deepened and specific It is understood, however, the drawings are provided for reference and description only, and are not intended to limit the present invention.

11‧‧‧原始資料區 11‧‧‧ raw data area

12‧‧‧資料加值區 12‧‧‧ Data Added Area

13‧‧‧資料備存區 13‧‧‧Data storage area

21‧‧‧高效資料處理區 21‧‧‧Efficient data processing area

22‧‧‧高性價比資料處理區 22‧‧‧ Cost-effective data processing area

23‧‧‧低成本資料處理區 23‧‧‧ Low-cost data processing area

31‧‧‧關聯式資料庫 31‧‧‧Related Database

32‧‧‧訊息提供程式 32‧‧‧ Feeder

33‧‧‧訊息系統 33‧‧‧Information System

34‧‧‧訊息寫入程式 34‧‧‧Message Writer

35‧‧‧大數據平台 35‧‧‧ Big Data Platform

41‧‧‧資料來源 41‧‧‧Source

411‧‧‧內部資訊 411‧‧‧Internal Information

412‧‧‧外部資訊 412‧‧‧External Information

42‧‧‧資料收錄單元 42‧‧‧Data Collection Unit

421‧‧‧即時同步作業 421‧‧‧Real-time synchronization

422‧‧‧整批資料收錄 422‧‧‧Batch data collection

43‧‧‧資料服務單元 43‧‧‧Data Service Unit

431‧‧‧資料描述區 431‧‧‧Data description area

432‧‧‧原始資料區 432‧‧‧ raw data area

433‧‧‧資料加值區 433‧‧‧ Data Added Area

434‧‧‧資料備存區 434‧‧‧Data storage area

44‧‧‧資料提供單元 44‧‧‧ Information Supply Unit

441‧‧‧資料查詢 441‧‧‧Inquiry

442‧‧‧資料訂閱服務 442‧‧‧Data Subscription Service

443‧‧‧整批資料提供 443‧‧‧ Full batch of information provided

45‧‧‧資料應用 45‧‧‧Data Application

451‧‧‧商業智慧 451‧‧‧Business Intelligence

452‧‧‧報表平台 452‧‧‧Reporting Platform

453‧‧‧簡訊平台 453‧‧‧Newsletter Platform

454‧‧‧應用系統 454‧‧‧Application System

501‧‧‧原始資料供應 501‧‧‧ Source of raw materials

502‧‧‧原始資料查詢 502‧‧‧Source data query

601‧‧‧歷史資料查詢 601‧‧‧Historical data query

602‧‧‧資料模型建立 602‧‧‧Data Model Establishment

603‧‧‧複雜高效運算 603‧‧‧ Complex and efficient operations

604‧‧‧非結構資料存放 604‧‧‧Unstructured data storage

605‧‧‧資料整合運算 605‧‧‧Data integration operation

606‧‧‧自主分析預處理 606‧‧‧Automatic analysis preprocessing

607‧‧‧即時串流分析 607‧‧‧Real-time streaming analysis

608‧‧‧資料模型運算 608‧‧‧Data Model Operation

701‧‧‧資料備存 701‧‧‧Data storage

702‧‧‧稽核查詢 702‧‧‧ Audit inquiry

81‧‧‧外部資訊 81‧‧‧External Information

82‧‧‧內部資訊 82‧‧‧Internal Information

83‧‧‧即時同步作業 83‧‧‧Real-time synchronization

84‧‧‧整批資料收錄 84‧‧‧Batch data collection

85‧‧‧原始資料區 85‧‧‧ raw data area

86‧‧‧資料加值區 86‧‧‧Data Value Added Area

87‧‧‧資料查詢 87‧‧‧Inquiry

88‧‧‧資料描述區 88‧‧‧Data description area

89‧‧‧商業智慧 89‧‧‧Business Intelligence

90‧‧‧行銷分析人員 90‧‧‧ Marketing Analyst

步驟801~821‧‧‧運營資料匯流流程範例 Steps 801 ~ 821‧‧‧‧ Example of operation data flow

圖1顯示運營資料匯流系統的三層式架構實施例示意圖;圖2顯示運營資料匯流系統中三層式資料儲存方式的成本與效能關係圖;圖3顯示運營資料匯流系統的系統架構實施例之一;圖4顯示運營資料匯流系統的系統架構實施例之二;圖5顯示運營資料匯流系統的原始資料區實施例;圖6顯示運營資料匯流系統的資料加值區實施例;圖7顯示運營資料匯流系統的資料庫存區實施例;圖8顯示運營資料匯流系統中運行的方法流程實施例。 Figure 1 shows a schematic diagram of a three-tiered architecture embodiment of the operational data convergence system; Figure 2 shows a cost and performance relationship diagram of the three-tiered data storage method in the operational data convergence system; and Figure 3 shows an example of a system architecture of the operational data convergence system. 1; FIG. 4 shows the second embodiment of the system architecture of the operational data convergence system; FIG. 5 shows the original data area embodiment of the operational data convergence system; FIG. 6 shows the data value-added area embodiment of the operational data convergence system; and FIG. 7 shows the operation An embodiment of the data storage area of the data converging system; FIG. 8 shows an embodiment of the method flow of operating the data converging system.

揭露書關於一種運營資料匯流方法與系統,所提出的運營資料匯流系統在概念上實現一種如圖1所示三層式資料架構實施例示意圖,其中主要包括一原始資料區11,此區在一實施例中可由高效資料處理的關聯式資料庫(Relational Database,RDB)實現,用以提供原始資料服務;一資料加值區12,此區為一種大數據平 台,在一實施例中可由高性價比存儲及分散處理的Hadoop開源計畫實現,這是一個Apache軟體基金會的開源計畫,具有強大的分散運算能力,可在系統中擔負處理各種網路使用記錄的分析,但不限於前述之使用範圍,在此則用以提供資料加值彙整服務;以及一資料備存區13,此區主要是由低成本庫存的檔案伺服器實現,用以提供資料備存與稽核。 The disclosure relates to a method and system for converging operational data. The proposed operational data converging system conceptually implements a three-tier data architecture embodiment as shown in FIG. 1, which mainly includes a raw data area 11, which is located in a In the embodiment, it can be implemented by a relational database (RDB) for efficient data processing to provide raw data services. A data value-added area 12, which is a big data platform, can be cost-effective in one embodiment. Implementation of storage and decentralized Hadoop open source project. This is an open source project of the Apache Software Foundation. It has powerful decentralized computing capabilities and can handle analysis of various network usage records in the system, but it is not limited to the aforementioned use. Range, which is used to provide data value-added integration services; and a data storage area 13, which is mainly implemented by a low-cost inventory file server to provide data storage and auditing.

運營資料匯流系統可採用多種技術結合資料流程,手段主要是通過不同成本與效能考量的資料處理方案整合金融機構(如金控、銀行)內部與外部資訊,分為如圖1所述原始資料區11、資料加值區12與資料備存區13等三層架構,依照其屬性,考量成本與儲存效能,如圖2所示運營資料匯流系統中三層式資料儲存方式的成本與效能關係圖。 The operating data converging system can use a variety of technologies to combine data processes. The main means is to integrate the internal and external information of financial institutions (such as financial control and banks) through data processing solutions with different cost and efficiency considerations. It is divided into the original data area as shown in Figure 1. 11. The three-tier structure of data value-added area 12 and data storage area 13, according to its attributes, consider costs and storage performance. As shown in Figure 2, the cost and performance relationship diagram of the three-tier data storage method in the operating data converging system. .

圖2所示為各種資料處理方案有關儲存成本(縱軸)與資料處理的回應速度(橫軸)的關係。 Figure 2 shows the relationship between storage costs (vertical axis) and response speed (horizontal axis) of data processing for various data processing schemes.

高效資料處理區21,位於儲存成本高,但是回應速度快的位置,為時常有大量數據湧入的儲存方案,適合圖1顯示運營資料匯流系統中原始資料區11。高性價比資料處理區22是位於儲存成本適中,但有不錯回應速度的儲存方案,適合運營資料匯流系統中資料加值12時常需要數據分析與提供客戶即時需要的需求。低成本資料處理區23則是一種低回應速度但是儲存成本低的儲存方案,適用運營資料匯流系統中資料備存區13的需求,此類資料並不經常更動,僅須應付調閱檔案的需求。 The high-efficiency data processing area 21 is located at a location with high storage cost but fast response speed. It is a storage solution that often has a large amount of data influx. It is suitable for the raw data area 11 in the operating data converging system shown in Figure 1. The cost-effective data processing area 22 is a storage solution with a moderate storage cost but a good response speed, which is suitable for the data value added 12 in the operating data converging system, which often requires data analysis and provides customers with immediate needs. The low-cost data processing area 23 is a storage solution with low response speed but low storage cost. It is suitable for the needs of the data storage area 13 in the operating data converging system. Such data is not often changed, and it only needs to meet the needs of accessing files. .

根據以上所述由高效資料處理、高性價比存儲及分散處理、低成本庫存的三層式資料架構,使得揭露書所提出的運營資料匯流系統建構涵蓋時效性、多樣性、整合性、安全性與高品質共五大面向的資訊服務,以單一平台架構,滿足金融機構高效穩定又符合成本考量的需求。 Based on the three-tier data architecture described above with efficient data processing, cost-effective storage and decentralized processing, and low-cost inventory, the construction of the operational data converging system proposed in the disclosure book covers timeliness, diversity, integration, security, and security. High-quality five-oriented information services, with a single platform structure, meet the needs of financial institutions with high efficiency and stability and cost considerations.

根據運營資料匯流系統實施例,可參考圖3所示系統架構實 施例,所述原始資料區(圖1,11)以關聯式資料庫(RDB)31實現,資料加值區(圖1,12)為一大數據平台35。 According to the embodiment of the operational data converging system, reference may be made to the embodiment of the system architecture shown in FIG. 3. The original data area (FIG. 1, 11) is implemented by a relational database (RDB) 31, and the data value added area (FIG. 1, 12 ) Is a large data platform 35.

系統運作時,關聯式資料庫(RDB)31即時同步資料至大數據平台35,同步方式例如採用異動資料擷取(Change Data Capture,CDC)技術,目的是處理運營資料匯流系統中大量產生的數據,其中採用的大數據平台35例如Apache軟體基金會的Hadoop開源計畫,具有強大的分散運算能力,在系統中擔負處理各種網路使用記錄的分析,但不限於前述之應用實例。在運營資料匯流系統中,由關聯式資料庫31透過異動資料擷取(CDC)程式將資料即時同步至如Hadoop實現的大數據平台35。 When the system is operating, the relational database (RDB) 31 synchronizes data to the big data platform 35 in real time. The synchronization method, for example, adopts Change Data Capture (CDC) technology. The purpose is to process a large amount of data generated in the operational data converging system. The big data platform 35 used in it, such as the Hadoop open source project of the Apache Software Foundation, has strong decentralized computing capabilities and is responsible for processing and analyzing various network usage records in the system, but it is not limited to the aforementioned application examples. In the operating data converging system, the data is synchronized to the big data platform 35 implemented by Hadoop in real time by the relational database 31 through a CDC program.

揭露書所提出的運營資料匯流系統係通過一訊息系統33作為關聯式資料庫31與大數據平台35間的資料傳送工具。在關聯式資料庫31與訊息系統33之間執行一訊息提供程式32,將關聯式資料庫31中的表格(TABLE)依照鍵值(KEY)視為字串流(Stream),訊息提供程式32能即時偵測關聯式資料庫31中數據流變化,並將變化傳送至訊息系統33。另於訊息系統33與大數據平台35之間執行一訊息寫入程式34,將訊息系統33中留存的異動資料,依序透過一種串流處理程序寫入大數據平台35,達成資料同步。 The operational data converging system proposed in the disclosure uses a message system 33 as a data transmission tool between the relational database 31 and the big data platform 35. A message providing program 32 is executed between the relational database 31 and the message system 33. The table (TABLE) in the relational database 31 is regarded as a stream according to the key value (KEY). The message providing program 32 Real-time detection of changes in data flow in the associated database 31 and transmission of the changes to the message system 33. In addition, a message writing program 34 is executed between the message system 33 and the big data platform 35, and the transaction data stored in the message system 33 is sequentially written into the big data platform 35 through a stream processing program to achieve data synchronization.

以上所述訊息系統33,例如Apache軟體基金會中一個處理即時資料的Kafka,此為設計用以處理一個系統中往來活動資料與營運數據處理的資料管理與訊息系統。所述系統以網站為例,網站的活動資料如網頁訪問次數、被瀏覽內容以及搜索記錄等,這些活動資料最終寫入某個檔案或資料庫中,用於後續分析與統計;而營運數據例如網站伺服器的性能數據,如中央處理器運行、輸出入介面使用、運行時間等。 The above-mentioned information system 33, such as Kafka, which handles real-time data in the Apache Software Foundation, is a data management and information system designed to process transaction data and operational data processing in a system. The system takes a website as an example. The website's activity data, such as the number of web page visits, viewed content, and search records, etc., are finally written into a file or database for subsequent analysis and statistics. Operational data such as Web server performance data, such as CPU operation, input / output interface usage, runtime, etc.

當所述訊息系統33用於揭露書所揭示的運營資料匯流系統時,則是處理訊息發布與訂閱的相關工作,讓訊息發布者寫入內 容,訊息接收者則是可以從訊息系統讀取資料,其中通過如Kafka實現的訊息系統33,可以保障運營資料匯流系統中訊息發布者將訊息發送到指定對象。 When the information system 33 is used to disclose the operational data converging system disclosed in the book, it is related to the work of publishing and subscribing to the message, allowing the publisher of the message to write the content, and the receiver of the message can read the data from the message system. Among them, the information system 33 implemented by, for example, Kafka, can guarantee that the message publisher in the operational data converging system sends the message to the designated object.

在訊息傳遞的過程中,資料中的元數據(metadata)成為數據管理的重要資訊,元數據可提供作為歷史調閱與分析運用,若應用於運營資料匯流系統,當資料更新,元數據也同步更新,元數據基礎資訊如資料表名稱、欄位名稱、欄位內容等,元數據進階資訊如資料業務說明、欄位業務描述、技術邏輯、資料更新頻率等。 In the process of message transmission, the metadata in the data becomes important information for data management. The metadata can be used as historical review and analysis. If it is applied to the operation data convergence system, when the data is updated, the metadata is also synchronized. Update, basic metadata information such as table name, field name, field content, etc. Advanced metadata information such as data business description, field business description, technical logic, data update frequency, etc.

依據以上實施例所描述的運營資料匯流系統的資料處理架構,系統架構主要如圖4顯示,應用在金融機構可以概略分為五的部分,分別可以伺服系統、資料庫,以及運行於伺服系統中的軟體程序等方式實現。 According to the data processing architecture of the operational data converging system described in the above embodiments, the system architecture is mainly shown in Figure 4. It can be roughly divided into five parts when applied to financial institutions, which can be servo systems, databases, and run in servo systems. Software programs.

第一部分是資料來源41,包括來自金融機構內部資訊411(存放款、信用卡、財富、財金…等)與外部資訊412(公開資訊、社群資訊…等),或是其中之一。 The first part is the data source 41, including internal information 411 (deposits, credit cards, wealth, finance, etc.) and external information 412 (public information, community information, etc.) from financial institutions, or one of them.

就銀行業務來說,例如消費者在各地使用金融卡與信用卡消費的資訊,都會即時匯入系統,這些數據將以系統的資料收錄單元42處理,且需要高效能資料處理能力的伺服系統,包括需要顧及時效性的即時同步作業421,這類資料例如刷卡消費查核、轉帳、即時回覆簡訊等。系統通過整批資料收錄422的作業可以產生高品質的資料,其中包括資料轉換,如ETL(萃取(extract)、轉置(transform)與載入(load))資料處理程序,以及資料清理程序等工作,可以確保產出高品質資料,對不同來源產生的資料進行統合與格式均一等處理。 As far as banking is concerned, for example, consumers' information on the use of financial cards and credit cards in various places will be imported into the system in real time. These data will be processed by the system's data collection unit 42 and require a high-performance data processing server system, including Real-time synchronization operations 421 that need to be considered for timeliness, such as checking credit card consumption, transferring funds, and responding to instant messages in real time. The system can generate high-quality data through a batch of data collection 422 operations, including data conversion, such as ETL (extract, transform, and load) data processing procedures, and data cleaning procedures, etc. The work can ensure the production of high-quality data, and integrate and uniformly process the data generated from different sources.

系統包括資料服務單元43,可以根據所收錄的資料提供多樣性內容,例如:其中設有資料描述區431,其中涵蓋企業資料字典、血緣與衝 擊分析等工具,可以將收錄的資料經(語意)分析後形成系統中的知識工具。 The system includes a data service unit 43, which can provide diversified content according to the collected data. For example, there is a data description area 431, which includes tools such as enterprise data dictionary, blood relationship and impact analysis. The collected data can be translated (semantic). After analysis, knowledge tools in the system are formed.

設有原始資料區432,此區用以存放經收錄、轉換與清理後的原始資料(raw data),相較於最後備存的資料,這部份較佳是較近期資料,可以形成關聯式資料庫(RDB),以供應查詢原始資料的服務。 There is a raw data area 432, which is used to store the collected, converted and cleaned up raw data. Compared with the last saved data, this part is more recent data and can be related. Database (RDB) to provide services for querying raw data.

資料服務單元43設有資料加值區433,產生整合性的數據,例如,其中可使用如前述Hadoop(Apache基金會下的計畫Hadoop®)實現的大數據伺服系統,在此系統中實現歷史資料查詢、資料模型建立、複雜高效運算、非結構資料存放、資料整合運算、自主分析預處理、即時串流分析,以及資料模型運算等功能。 The data service unit 43 is provided with a data value-added area 433 to generate integrated data. For example, the big data servo system implemented by the aforementioned Hadoop (the project Hadoop® under the Apache Foundation) can be used to implement history in this system. Data query, data model establishment, complex and efficient calculation, non-structured data storage, data integration calculation, autonomous analysis pre-processing, real-time streaming analysis, and data model calculation and other functions.

資料服務單元43設有資料備存區434,資料備存區434一般為較低成本的儲存方案,用以儲存歷史資料,提供資料備存、稽核查詢等功能。 The data service unit 43 is provided with a data storage area 434. The data storage area 434 is generally a low-cost storage solution for storing historical data and providing functions such as data storage, auditing and query.

運營資料匯流系統包括一資料提供單元44,通過運營資料匯流系統中的軟體程序提供多樣的服務。其中提供的服務包括資料查詢441,例如可通過應用程式介面(API)提供具有權限管理的安全性措施,資料查詢441的相關作業如資料庫虛擬化整合、查詢權限管理(安全性)、資料去識別化等。資料提供單元44提供資料訂閱服務442,可以根據使用者請求提供資料訂閱、推播等功能。資料提供單元44提供整批資料提供443,其中方式包括資料轉換、ETL資料處理與批次遞送等。 The operating data converging system includes a data providing unit 44 that provides various services through software programs in the operating data converging system. The services provided include data query 441. For example, security measures with permission management can be provided through the application programming interface (API). Related operations of data query 441 include database virtualization integration, query permission management (security), and data retrieval. Identification. The data providing unit 44 provides a data subscription service 442, which can provide functions such as data subscription and push broadcast according to user requests. The data providing unit 44 provides a whole batch of data providing 443, including methods of data conversion, ETL data processing, and batch delivery.

最後,運營資料匯流系統所收錄得資料經資料服務單元43與資料提供單元44產出後,形成各種資料應用45,根據實施例,在金融機構的應用上可包括商業智慧451,能提供視覺化與自主分析的作業;報表平台452提供報表查詢;簡訊平台453提供簡訊發送的服務;以及應用系統454,包含績效管理與風險管理的功能, 但應用系統454之實作不限於前述之使用範圍。 Finally, the data collected by the operating data converging system is output by the data service unit 43 and the data providing unit 44 to form various data applications 45. According to the embodiment, the application of financial institutions can include business intelligence 451, which can provide visualization And self-analysis; report platform 452 provides report query; SMS platform 453 provides newsletter sending service; and application system 454 includes performance management and risk management functions, but the implementation of application system 454 is not limited to the aforementioned scope of use.

運營資料匯流系統的核心在於其中資料服務單元(圖4,43)中的原始資料區(圖4,432)、資料加值區(圖4,433)以及資料備存區(圖4,434)。 The core of the operating data convergence system lies in the original data area (Figure 4, 432), data value-added area (Figure 4, 433), and data storage area (Figure 4, 434) in the data service unit (Figure 4, 43). .

圖5首先顯示運營資料匯流系統運行時其中資料服務單元中原始資料區(圖4,432)的實施例。圖中顯示原始資料區432主要包括以執行於電腦系統的軟體程序與配合資料庫實現的原始資料供應501與原始資料查詢502兩個功能。原始資料供應501為資料經同步資料源收錄、轉換與清理後,進入關聯式資料庫中儲存,除可以提供高品質的資料外,更通過原始資料查詢502的功能形成可查詢的結構型資料。 FIG. 5 first shows an embodiment of the original data area (FIG. 4, 432) in the data service unit when the operation data converging system is running. The figure shows that the raw data area 432 mainly includes two functions: a raw data supply 501 and a raw data query 502 implemented by software programs running on a computer system and cooperating with a database. The original data supply 501 is data that is collected, converted, and cleaned by a synchronous data source, and stored in a relational database. In addition to providing high-quality data, it also uses the function of the original data query 502 to form queryable structured data.

圖6顯示運營資料匯流系統的資料加值區實施例。圖中顯示資料加值區433,當各種資料經系統收錄後,通過系統中軟體程序與資料庫的運作,將資料加以彙整,產生整合性的大數據,使得系統可以提供歷史資料查詢601、資料模型建立602、複雜高效運算603、非結構資料存放604、資料整合運算605、自主分析預處理606、即時串流分析607,以及資料模型運算608等功能。 FIG. 6 shows an embodiment of a data value-added area of an operation data converging system. The figure shows the data value-added area 433. After various data are collected by the system, the data are integrated through the operation of software programs and databases in the system to generate integrated big data, which enables the system to provide historical data query 601, data Model creation 602, complex and efficient operations 603, unstructured data storage 604, data integration operations 605, autonomous analysis pre-processing 606, real-time streaming analysis 607, and data model operations 608.

歷史資料查詢601讓系統收錄的資料,成為可查詢的歷史資料。 Historical data query 601 enables the data collected by the system to become queryable historical data.

資料模型建立602功能提供系統根據收錄的資料建立資料模型(data model),並定義輸入與輸出,配合資料模型運算608,由輸入的資料即時運算後回覆結果。 The data model creation 602 function provides the system to create a data model based on the collected data, define inputs and outputs, cooperate with the data model calculation 608, and reply to the results after the input data is calculated in real time.

複雜高效運算603為系統因為資料虛擬化架構提供了對各種數據高效運算的能力。非結構資料存放604讓系統可以儲存影音資料、外部開放式資料(open data)等非結構資料。資料整合運算605提供系統可以快速整合跨系統資源,而不受限於單一資料來源,適合金融機構等需要處理複雜來源資料的用途。自主分析預處理606提供系統在自主分析之前,能夠預先處理內部資訊、 外部資訊,例如資料轉換、清理與同步等動作。 The complex and efficient operation 603 provides the system with the ability to efficiently operate various data because of the data virtualization architecture. The unstructured data storage 604 allows the system to store unstructured data such as audiovisual data, external open data, and the like. Data integration operation 605 provides a system that can quickly integrate cross-system resources without being limited to a single data source. It is suitable for financial institutions and other applications that need to process data from complex sources. The autonomous analysis pre-processing 606 provides the system with the ability to process internal information and external information in advance, such as data conversion, cleaning, and synchronization, before autonomous analysis.

即時串流分析607提供系統處理即時資料,能夠即時將資訊以簡訊、推播等技術播送出去,並能形成訂閱內容。 Real-time streaming analysis 607 provides the system to process real-time data, which can broadcast information in real-time using technologies such as newsletters and push broadcasts, and can form subscription content.

圖7顯示運營資料匯流系統的資料庫存區實施例,資料備存區434主要的功能是提供資料備存701,是以低成本的儲存方案將近期資料儲存後備查,並同時用於金融機構內稽核查詢702之用。 Figure 7 shows an example of the data storage area of the operating data converging system. The main function of the data storage area 434 is to provide data storage 701. It uses a low-cost storage solution to store recent data for future reference, and it is also used in financial institutions. Audit query 702.

如此可知,運營資料匯流系統通過資料收錄單元42、資料服務單元43與資料提供單元44提供資料整合的作業程序,使得系統所收錄的原始資料可以形成知識、可查詢資料、資料模型,除了通過低成本儲存方案建立的資料備存外,仍可提供資料查詢、訂閱與整批提供的服務,並產生最後多樣性的資料應用45。 It can be seen that the operation data integration system provides data integration operation procedures through the data collection unit 42, the data service unit 43, and the data providing unit 44, so that the original data collected by the system can form knowledge, queryable data, and data models. In addition to the data storage established by the cost storage scheme, it is still possible to provide data query, subscription and batch services, and generate the final diversity of data applications45.

上述實施例顯示,運營資料匯流系統通過伺服系統端的軟體程序與資料庫的應用,讓整個系統可兼顧時效性(即時、類即時、批次),主要是指資料查詢與應用的時效性,通過資料收錄單元42的功能,達成與資料來源彼此之間同步資料的功效,以利快速反應客戶需求。 The above examples show that the operating data converging system allows the entire system to take into account the timeliness (real-time, real-time, batch) through the application of software programs and databases on the server system side, mainly referring to the timeliness of data query and application. The function of the data collecting unit 42 achieves the effect of synchronizing the data with the data source, so as to facilitate quick response to customer needs.

系統提供資料多樣性,包括原始/彙整/歷史資料查詢、即時資料推播、批次資料提供等功能,其中提供三層式的資料儲存架構,將關聯式資料庫、以平行運算系統實現的大數據平台,以及低成本儲存方案整合後,強化資料儲存管理,實現資料提供服務,以增進使用效益。 The system provides data diversity, including original / aggregated / historical data query, real-time data push, batch data provision, and other functions. It provides a three-tiered data storage architecture that integrates a relational database with a large-scale parallel operation system. After the integration of data platforms and low-cost storage solutions, data storage management will be strengthened to provide data services to improve the use efficiency.

所述高品質,則是指系統提供資料清理(如ETL)、資料品質檢驗等功能,能促進資料整合。 The high quality means that the system provides functions such as data cleaning (such as ETL) and data quality inspection, which can promote data integration.

系統的整合性指資料整合運算與資料模型運算等功能,達成資料彙整,並形成業務模型。 System integration refers to functions such as data integration calculations and data model calculations to achieve data aggregation and form a business model.

安全性則是指權限管理,提供查詢權限管理機制,可以根據權限提供適當的資訊,包括資料去識別化的運用。 Security refers to permission management, providing query permission management mechanism, which can provide appropriate information according to permissions, including the use of data to identify.

圖8顯示運營資料匯流系統中運行的方法流程實施例,在此 流程實施例中,資料源為外部資訊81與內部資訊82,系統將得到的資訊形成運營資料匯流系統處理的目標,在此流程中,經即時同步作業83、整批資料收錄84、原始資料區85、資料加值區86,提供資料查詢87,以及形成資料描述區88的內容,更是作為商業智慧89與行銷分析人員90執行分析的應用。 FIG. 8 shows an embodiment of a method flow running in an operational data converging system. In this process embodiment, the data sources are external information 81 and internal information 82. The system uses the obtained information to form the processing data converging system processing target. In the process of real-time synchronization 83, batch data collection 84, original data area 85, data value-added area 86, data query 87 is provided, and the content description area 88 is formed. It is also used as business intelligence 89 and marketing analysts 90 Applications that perform analysis.

流程中,自外部資訊81取得外部公開圖資資訊(步驟801),圖資在整批資料收錄84中經過資料轉換(步驟805),形成資料加值區86中存放的非結構資料(步驟809),接著經資料整合運算(步驟810)後,成為商業智慧89應用的一環,就是提供自主分析(步驟818),目的是能夠呈現給使用者,如行銷分析人員90。 In the process, external public map information is obtained from the external information 81 (step 801), and the map data is converted in the batch of data collection 84 (step 805) to form unstructured data stored in the data value added area 86 (step 809). ), And after data integration calculation (step 810), it becomes a part of the business intelligence 89 application, which is to provide autonomous analysis (step 818), which can be presented to users, such as marketing analysts 90.

在另一方面,系統取得企業(如金融機構)內部資訊(步驟802),這些資料可經資料清理(步驟806)後,提供給資料加值區86,並接續提供自主分析(步驟818)。 On the other hand, the system obtains the internal information of the enterprise (such as a financial institution) (step 802). These data can be provided to the data value-added area 86 after the data is cleared (step 806), and then provide independent analysis (step 818).

在金融機構運用的流程中,內部資訊82例如一個提供消費者的特惠資料(步驟803),這些資料經即時同步作業83中同步資料(步驟804),為的是同步到原始資料區85中成為原始資料供應(步驟807)的一部分,能夠作為原始資料查詢(步驟808)之用,也用於資料加值區86即時串流分析,分析大數據資料串流(步驟811),同樣提供自主分析(步驟818)。 In the process used by financial institutions, the internal information 82, for example, provides consumers with preferential data (step 803). These data are synchronized in real-time synchronization operation 83 (step 804) in order to be synchronized to the original data area 85 to become Part of the original data supply (step 807) can be used as the original data query (step 808), and also used in the data value-added area 86 for real-time streaming analysis, analyzing big data data streaming (step 811), and also providing autonomous analysis (Step 818).

當原始資料供應(步驟807)成為原始資料查詢(步驟808)之用時,系統通過資料查詢87的功能判斷這些是否為系統所需要的資料?(步驟813),若是系統所需資料(是),則執行自主分析(步驟818),否則,資料可以成為資料描述區88中的資料字典(步驟817)。資料字典的內容是能夠讓行銷分析人員確認需要分析的資料,其中目的之一是能夠讓使用者篩選不需要分析的資料。 When the original data supply (step 807) becomes the original data query (step 808), the system judges whether these are required data by the function of the data query 87? (Step 813), if the system needs data (Yes), perform autonomous analysis (Step 818), otherwise, the data can become a data dictionary in the data description area 88 (Step 817). The content of the data dictionary allows marketing analysts to identify the data they need to analyze. One of the purposes is to allow users to filter the data that they don't need to analyze.

當系統取得經轉換的外部資訊、經清理的內部資訊、經即時串流分析的內部資訊,或是查詢原始資料得到的資料時,通過商業智慧89應用中自主分析步驟後(步驟818),資訊可經視覺化呈 現(步驟819),讓行銷分析人員90使用視覺化報表工具,結合即時資料串流分析與批次整理資料,依業務維度,進行行銷活動成效分析,直到結束(步驟820)整個流程。 When the system obtains the converted external information, the cleaned internal information, the internal information analyzed by real-time streaming, or the data obtained by querying the original data, after the independent analysis step in the Business Intelligence 89 application (step 818), the information It can be visualized (step 819), allowing marketing analysts 90 to use visual reporting tools, combining real-time data streaming analysis and batch organization of data, and analyze the effectiveness of marketing activities based on business dimensions until the end (step 820). Process.

另一方面,由行銷分析人員90提起資料分析的需求,系統進行分析(步驟821),可先至資料描述區88查詢資料字典(步驟817),能夠讓行銷分析人員90確認需要分析的資料,篩選不需要分析的資料。 On the other hand, the marketing analyst 90 raises the need for data analysis and the system performs the analysis (step 821). First, the data description area 88 can be queried for the data dictionary (step 817). The marketing analyst 90 can confirm the data to be analyzed. Screen for data that does not require analysis.

接著,運營資料匯流系統提供的資料查詢87功能中,先執行查詢權限管理(步驟816)的步驟,判斷登入系統查詢的使用者(即本範例的行銷分析人員90)是否有權限?(步驟814),若判斷具有權限(是),將執行資料去識別化(步驟812),將資料去識別化目的一方面是讓系統移除資料中可以識別來源或身份的資訊,作為系統可查詢的原始資料(步驟808),另一方面是讓使用者可以確認分析的資料。反之,若使用者並沒有查詢權限(否),則拒絕查詢(步驟815)。 Next, in the data query 87 function provided by the operation data confluence system, first perform the query permission management (step 816) step to determine whether the user who logs in to the system query (that is, the marketing analyst 90 in this example) has permission? (Step 814) If it is judged that it has authority (Yes), the data de-identification will be performed (Step 812). The purpose of the data de-identification is to allow the system to remove the information that can identify the source or identity in the data. The query of the original data (step 808), on the other hand, allows the user to confirm the analyzed data. Conversely, if the user does not have the query authority (No), the query is rejected (step 815).

以下列舉揭露書所提出運營資料匯流系統的優勢。 The following lists the advantages of the operational data converging system proposed in the disclosure.

由於運營資料匯流系統結合即時資料同步技術、大數據資料串流、ETL等技術,彙整全行資料,可大幅簡化過往複雜的資料流程,降低系統維運成本,並可滿足即時加值彙整、模型建置運算、歷史大量資料及非結構化資料處理的資料服務需求。系統透過資料虛擬化架構,快速整合跨系統資源,提供單一資料查詢管道與資料去識別化功能,實現集中權限管理與提升資料安全性。 The operational data converging system combines real-time data synchronization technology, big data data streaming, ETL and other technologies to consolidate the entire bank's data, which can greatly simplify the complex data flow in the past, reduce system maintenance costs, and meet the real-time value-added aggregation and model. Build data service requirements for operations, historical bulk data, and unstructured data processing. Through the data virtualization architecture, the system quickly integrates cross-system resources, provides a single data query channel and data de-identification function, realizes centralized permission management and improves data security.

運行在特定企業中,運營資料匯流系統提供企業資料定義,並透過合宜的權限管理,便利跨部處資訊共享,降低各業務系統間溝通與應用的門檻,不僅能提升系統分析效率,也可驅動業務自主分析能量,打破傳統數字管理,以數據帶動業務發展。同時也透過資料品質管理,隨時維持資料之正確性。 Running in specific enterprises, the operational data converging system provides corporate data definitions and facilitates information sharing across departments through appropriate authority management, reducing the barriers to communication and application between business systems, which can not only improve system analysis efficiency, but also drive The business analyzes energy autonomously, breaks through traditional digital management, and drives business development with data. At the same time, the accuracy of the data is maintained at all times through data quality management.

透過彙整全行資料源以及結合異動資料擷取(CDC)即時資 料同步技術,並透過大數據資料串流技術與ETL處理技術,除可降低各系統介接之複雜度外,也同時大幅簡化資料流程;儘管日後行內系統數量增加,整體系統複雜度亦不受影響,有效降低系統維運成本。 By integrating the entire bank's data sources, combined with real-time data synchronization (CDC) data synchronization technology, and through big data data streaming technology and ETL processing technology, in addition to reducing the complexity of each system interface, it also greatly simplifies the data at the same time. Although the number of systems in the industry will increase in the future, the overall system complexity will not be affected, which effectively reduces the system maintenance costs.

最後,系統整合行內線上與線下資料,同時結合公開資訊,了解顧客即時動態,掌握即時商機。 Finally, the system integrates online and offline data in the industry, and combines public information to understand real-time customer dynamics and grasp real-time business opportunities.

如此,根據以上運營資料匯流方法與系統的實施例,可知,當運營資料匯流系統核心建構一種三層式資料架構,將資料分別為原始資料區、資料加值區以及資料備存區,實現多種技術結合資料流程,整合金融機構內部各系統資訊,建構涵蓋時效性、多樣性、整合性、安全性與高品質共五大面向的資訊服務,藉此可以結合即時資料同步、大數據串流分析、機敏性資料遮罩等技術,透過資料虛擬化架構,利用應用程式介面(API)概念達成單一平台多樣性資料服務,如此可以打造商用資料庫與開放平行運算環境的即時串流技術,實現三層式資料架構間系統介接,強化資料儲存管理,輔以資料定義,增進資料使用效益。因此具有以下優勢: 一、達到金融機構高穩定性的要求;二、能以低成本的方式進行平台架構;三、提供即時/類即時/批次型等多樣化的資料服務;四、將傳統結構化及非結構化資料整合於單一平台;以及五、提供整合性及去識別化的查詢服務。 In this way, according to the embodiment of the operation data converging method and system above, it can be known that when the core of the operation data converging system constructs a three-tier data structure, the data is divided into the original data area, the data value-added area, and the data storage area, achieving a variety of Combining technology with data processes, integrating information from various systems within financial institutions, and constructing five major information services covering timeliness, diversity, integration, security and high quality, which can be combined with real-time data synchronization, big data streaming analysis, Techniques such as sensitive data masking, through data virtualization architecture, use the concept of application programming interface (API) to achieve a single platform of diverse data services, so that real-time streaming technology for commercial databases and open parallel computing environments can be created, achieving three layers The system interface between the two types of data structure, strengthen the data storage management, supplement the data definition, and improve the data use efficiency. Therefore, it has the following advantages: First, to meet the requirements of high stability of financial institutions; Second, the platform structure can be carried out at a low cost; Third, provide a variety of data services such as real-time / real-time / batch type; Fourth, the traditional Integrated structured and unstructured data on a single platform; and 5. Provide integrated and de-identified query services.

惟以上所述僅為本發明之較佳可行實施例,非因此即侷限本發明之專利範圍,故舉凡運用本發明說明書及圖示內容所為之等效結構變化,均同理包含於本發明之範圍內,合予陳明。 However, the above description is only a preferred and feasible embodiment of the present invention, and therefore does not limit the patent scope of the present invention. Therefore, any equivalent structural changes made by using the description and illustrated contents of the present invention are also included in the present invention. Within the scope, joint Chen Ming.

Claims (10)

一種運營資料匯流系統,包括:一原始資料區,由一關聯式資料庫實現,用以儲存收錄自一外部資訊與一內部資訊,或是其中之一的原始資料,並經資料轉換與資料清理的資料;一資料加值區,實現一大數據平台,即時同步該關聯式資料庫的資料,用以提供資料加值彙整服務;以及一資料備存區,由一檔案伺服器實現,用以提供資料備存與稽核。     An operation data converging system includes: a raw data area implemented by a relational database for storing raw data collected from an external information and an internal information, or one of them, and data conversion and data cleaning Data; a data value-added area, realizing a large data platform, real-time synchronization of the data of the related database, to provide data value-consolidation services; and a data storage area, implemented by a file server, for Provide data storage and audit.     如請求項1所述的運營資料匯流系統,其中,一訊息系統作為該關聯式資料庫與該大數據平台間的資料傳送工具。     The operating data converging system according to claim 1, wherein an information system is used as a data transmission tool between the relational database and the big data platform.     如請求項2所述的運營資料匯流系統,其中該關聯式資料庫與該訊息系統之間執行一訊息提供程式,將該關聯式資料庫中的表格依照鍵值視為字串流,且該訊息提供程式能即時偵測該關聯式資料庫中數據流變化,並將變化傳送至該訊息系統。     The operating data convergence system as described in claim 2, wherein an information provider is executed between the relational database and the information system, and the tables in the relational database are treated as a stream according to the key value, and the The message provider can detect changes in the data flow in the associated database in real time and send the changes to the message system.     如請求項2所述的運營資料匯流系統,其中該訊息系統與該大數據平台之間執行一訊息寫入程式,將該訊息系統中留存的異動資料,依序透過一種串流處理程序寫入該大數據平台,達成資料同步。     The operation data converging system according to claim 2, wherein a message writing program is executed between the information system and the big data platform, and the transaction data retained in the information system is sequentially written through a stream processing program This big data platform achieves data synchronization.     如請求項1所述的運營資料匯流系統,其中收錄該原始資料的方式包括一即時同步作業,以及一整批資料收錄的作業。     The operation data converging system according to claim 1, wherein the way of collecting the original data includes a real-time synchronization operation and a batch of data collection operations.     如請求項5所述的運營資料匯流系統,其中該整批資料收錄的作業包括萃取、轉置與載入資料處理程序,以及一資料清理程序。     The operating data converging system as described in claim 5, wherein the operation of collecting the entire batch of data includes extraction, transposition and loading of data processing procedures, and a data cleaning procedure.     如請求項1所述的運營資料匯流系統,其中該資料加值區包括的功能包括:歷史資料查詢、資料模型建立、複雜高效運算、非結構資料存放、資料整合運算、自主分析預處理、即時串流分析,以及資料模型運算。     The operational data converging system as described in claim 1, wherein the data value added area includes functions including: historical data query, data model establishment, complex and efficient calculation, non-structured data storage, data integration calculation, autonomous analysis pre-processing, real-time Stream analysis and data model operations.     如請求項1所述的運營資料匯流系統,其中該資料備存區此用較低成本的儲存方案,用以儲存歷史資料,提供的功能包括資料備存與稽核查詢。     The operational data converging system as described in claim 1, wherein the data storage area uses a lower cost storage solution to store historical data, and the functions provided include data storage and audit query.     如請求項1至8中任一項所述的運營資料匯流系統,更包括一資料提供單元,通過該運營資料匯流系統中的軟體程序提供的服務包括資料查詢、資料訂閱服務,以及整批資料提供。     The operation data converging system according to any one of claims 1 to 8, further comprising a data providing unit, and services provided through software programs in the operation data converging system include data inquiry, data subscription services, and batch data provide.     一種應用如請求項1所述的運營資料匯流系統的運營資料匯流方法,包括:取得一外部資訊或一內部資訊;經一整批資料收錄功能對該外部資訊或該內部資訊執行資料轉換或資料清理;形成一資料加值區中的非結構資料,或直接以經資料整合運算後,成為商業智慧應用中自主分析,並以一視覺化呈現給使用者;其中,該內部資訊經即時同步作業同步到該原始資料區,作為原始資料查詢之用,也用於該資料加值區即時串流分析之用;其中,該運營資料匯流系統提供的一資料查詢功能,包括執行查詢權限管理的步驟,判斷登入系統查詢的使用者的權限。     An operational data converging method applying the operational data converging system according to claim 1, comprising: obtaining an external information or an internal information; performing a data conversion or data on the external information or the internal information through a batch of data collection function Clean up; form an unstructured data in the data value-added area, or directly integrate the data and calculate it to become an independent analysis in business intelligence applications and present it to the user in a visualization; where the internal information is synchronized in real time Synchronize to the original data area for the original data query and also for the real-time streaming analysis of the data value-added area. Among them, a data query function provided by the operational data converging system includes the steps of performing query authority management To determine the permissions of the user who logs in to the system.    
TW107116271A 2018-05-14 2018-05-14 System and method for operational data convergence TW201947492A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
TW107116271A TW201947492A (en) 2018-05-14 2018-05-14 System and method for operational data convergence

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
TW107116271A TW201947492A (en) 2018-05-14 2018-05-14 System and method for operational data convergence

Publications (1)

Publication Number Publication Date
TW201947492A true TW201947492A (en) 2019-12-16

Family

ID=69582816

Family Applications (1)

Application Number Title Priority Date Filing Date
TW107116271A TW201947492A (en) 2018-05-14 2018-05-14 System and method for operational data convergence

Country Status (1)

Country Link
TW (1) TW201947492A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI802056B (en) * 2020-10-27 2023-05-11 大陸商中國銀聯股份有限公司 Data verification method, device, equipment, system and storage medium
US11687853B2 (en) 2020-09-14 2023-06-27 Data Systems Consulting Co., Ltd. Electronic device for detecting business system and detection method thereof

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11687853B2 (en) 2020-09-14 2023-06-27 Data Systems Consulting Co., Ltd. Electronic device for detecting business system and detection method thereof
TWI802056B (en) * 2020-10-27 2023-05-11 大陸商中國銀聯股份有限公司 Data verification method, device, equipment, system and storage medium

Similar Documents

Publication Publication Date Title
CN104298771A (en) Massive web log data query and analysis method
US20190050435A1 (en) Object data association index system and methods for the construction and applications thereof
Bedeley Big Data opportunities and challenges: the case of banking industry
US9123006B2 (en) Techniques for parallel business intelligence evaluation and management
Scannapieco et al. Placing big data in official statistics: a big challenge
Wongthongtham et al. Ontology and trust based data warehouse in new generation of business intelligence: State-of-the-art, challenges, and opportunities
Liang et al. Financial big data analysis and early warning platform: a case study
CN110968571A (en) Big data analysis and processing platform for financial information service
Srivastava et al. Fraud detection in the distributed graph database
Salma et al. Domain-driven design of big data systems based on a reference architecture
Kanchi et al. Challenges and Solutions in Big Data Management--An Overview
Jiang Research on big data audit based on financial sharing service model using fuzzy AHP
Benjelloun et al. Big Data Processing: Batch-based processing and stream-based processing
CN111639121A (en) Big data platform and method for constructing customer portrait
TW201947492A (en) System and method for operational data convergence
CN110544035A (en) internal control detection method, system and computer readable storage medium
Nagdive et al. Web server log analysis for unstructured data using apache flume and pig
Ibtisum A Comparative Study on Different Big Data Tools
Sirisha et al. IoT-based data quality and data preprocessing of multinational corporations
Pan et al. Research on the status of e-commerce development based on big data and Internet technology
Ahmed et al. Agent-based big data analytics in retailing: a case study
Pei et al. Bank customer loyalty under the background of internet finance and multimedia technology
Balakrishnan et al. Implementing data strategy: design considerations and reference architecture for data-enabled value creation
Mishra et al. Challenges in big data application: a review
Helfert et al. Big data quality-towards an explanation model in a smart city context