TWI830105B - Smart comparison system - Google Patents

Abstract

An intelligent comparison system includes a data identification module, a data conversion module and a data comparison module. The data identification module obtains an external data to be identified from an external source server system and an internal data to be identified from an internal source server system, and inputs the external data to be identified and the internal data to be identified respectively into corresponding Multiple machine learning models for different products are used to obtain respectively corresponding externally recognized data and internally recognized data. The data conversion module performs a plurality of conversion procedures on the externally identified data and the internally identified data to obtain corresponding externally converted data and internally converted data respectively. The data comparison module compares the external converted data and the other transaction records of the same product in the internal converted data to generate a comparison result data.

Description

Smart comparison system

The present invention relates to a comparison system, and in particular, to an intelligent comparison system for content comparison of files from various sources and different formats.

Backend operations in the financial market often require verification of data between different systems. In the past, manual downloading of reports, processing of letters, and use of EXCEL were used for data verification. Due to the diversity of sources, it is usually necessary to adopt a customized single automated process, resulting in low efficiency of the system. Take transaction verification as an example. The original physical documents such as emails or faxes provided by the counterparty are opened one by one by the handler to determine the transaction type, product type, transaction conditions and other information, and then checked with the transaction information in the bank's transaction system. , and submit relevant files for review for secondary confirmation. There are many counterparties and products, file formats and transaction content descriptions are different. The operation time is lengthy and omissions in document delivery are prone to occur. Therefore, in the face of diverse and changing source information, how to implement an automated comparison system has become a problem to be solved.

Therefore, the purpose of the present invention is to provide an intelligent comparison system for content comparison of files in various formats.

Therefore, the smart comparison system of the present invention is suitable for an internal source server system and an external source server system, and includes a data identification module, a data conversion module and a data comparison module.

The data recognition module includes a plurality of machine learning models corresponding to different products, and obtains an external data to be identified from the external source server system according to a predetermined schedule, and obtains it from the internal source server system. An internal data to be identified, and the external data to be identified and the internal data to be identified are respectively input into the machine learning models to obtain respectively corresponding externally identified data and internally identified data. The externally identified data and the internally identified data respectively include multiple transaction records, and these transaction records contain multiple field data with different formats.

The data conversion module is electrically connected to the data identification module to receive the externally identified data and the internally identified data, and performs multiple conversion procedures on the externally identified data and the internally identified data respectively to obtain respective Corresponding to one external converted data and one internal converted data, and the transaction records of the external converted data and the internal converted data respectively contain the field data in the same format.

The data comparison module is electrically connected to the data conversion module, and compares the external converted data and the other transaction records of the same product in the internal converted data to generate a comparison result data .

In some implementations, the smart comparison system is also applicable to a client device that is electrically connected to the data comparison module and receives the comparison result data to display the comparison result data.

In other implementations, the predetermined schedule is set by a Robotic Process Automation (RPA) system.

In other implementation forms, the conversion procedures include a format conversion procedure, a data merging procedure, a terminology conversion procedure, an expansion field procedure, and an input code procedure. This format converter is related to the format conversion of date representation. The data merging process is related to the aggregation of data for the same purpose. This terminology converter is concerned with the name conversion of the way a term is represented. The field-expanding procedure involves adding at least one of the field data to the data. The input code procedure is associated with executing predetermined code for data processing.

In some implementations, the data comparison module also stores a primary key setting value. When the data comparison module determines that the primary key setting value indicates a keyless comparison mode, the data comparison module The group compares the field data of the same product type and corresponding transaction records in the external converted data and the internal converted data to generate the comparison result data.

In some implementations, the data comparison module also stores a mandatory correspondence relationship among the field data that is set as a mandatory conformer. When the data comparison module determines that the primary key setting value indicates that it belongs to In a primary key value comparison mode, the data comparison module also compares the external converted data and the internal converted data that belong to the same product and other corresponding transaction records based on the mandatory correspondence. The data in these fields are compared to generate the comparison result data.

In some implementations, the data comparison module also stores multiple weight coefficients of some of the field data and another threshold corresponding to one of the products. The data comparison module also stores According to the weight coefficient and the corresponding threshold, the externally converted data and the internally converted data belonging to the same product category and corresponding to the field data are compared to generate the comparison result data.

In other implementations, the sum of the weighting coefficients in the same product category is equal to a preset maximum sum, and the data comparison module calculates the corresponding columns in the product category. The sum of at least one weight coefficient corresponding to the data judged to be consistent is to obtain a summed value. When the data comparison module determines that the summed value is equal to the maximum sum, the comparison generated The result data includes other field data corresponding to the product that are completely consistent. When the data comparison module determines that the total value is greater than or equal to the threshold and less than the maximum sum, the comparison is generated The result data includes other field data corresponding to the product that are suspected to be consistent. When the data comparison module determines that the total value is less than the threshold, the generated comparison result data includes other field data corresponding to the product. The data in other fields are completely inconsistent.

In other implementations, the data comparison module also stores a comparison schedule, and the data comparison module compares the external converted data and the external converted data according to the time points indicated by the comparison schedule. The corresponding field data in the internally converted data are compared to generate the comparison result data.

The effect of the present invention is that the smart comparison system of the present invention obtains the externally recognized data and the internally recognized data of the data content through the data recognition module and sends them to the data conversion module to obtain the same format. The transaction records of the external converted data and the internal converted data are sent to the data comparison module, and the data comparison module belongs to the same product category in the external converted data and the internal converted data. Comparing the transaction records to generate the comparison result data, compared with the conventional technology in terms of multiple sources, manual identification, and comparison of data, it can achieve the effect of saving time and effort.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are designated with the same numbering.

Referring to Figure 1, an embodiment of the smart comparison system 100 of the present invention is suitable for an internal source server system 1, an external source server system 2 and a client device 6, and includes a data identification module 3, a a data conversion module 4 and a data comparison module 5.

The external source server system 2 is, for example, a securities counter trading center server system, a securities dealer server system, or a central bank exchange rate server system, but is not limited to the above. The internal source server system 1 is, for example, a data market (Data Mart) server system, a transaction account server system (Murex), a derivatives server system (FTS), a bond server system (FITS), but not Subject to the above. The external source server system 2 and the internal source server system 1 provide an external data to be identified and an internal data to be identified respectively. The external data to be identified and the internal data to be identified include various transaction confirmations, such as stock transactions. Orders and foreign exchange transaction orders, but not limited to the above. For example, the external data to be identified and the internal data to be identified are, for example, multiple attachment files of multiple emails, and each attachment file is, for example, csv, txt, xlsx, xls, FIX, json, xml, pdf. and other file formats. Each transaction confirmation (i.e., the attachment file) includes at least one of a text content and a non-text content. The non-text content is, for example, a picture, a table, or an image. The smart comparison system 100 determines which product category the attachment file of the email (ie, the external data to be identified and the internal data to be identified) belongs to through an email rule. The email rule is, for example, based on the subject of the email. , keywords in the text of the letter, or keywords in the attachment content to determine the product category.

The data identification module 3 includes a plurality of machine learning models corresponding to different products, and obtains the external data to be identified from the external source server 2 according to a predetermined schedule, and obtains the external data to be identified from the internal source server System 1 obtains the internal data to be identified. The machine learning model is a supervised learning model and is built based on a training data set. This training data set contains various known transaction confirmations mentioned above. Other transaction confirmations belonging to the same product (such as stock transaction orders) will be trained in the corresponding machine learning model. For example, 5 stock trading orders of Company A and 3 stock trading orders of Company B will be trained in the corresponding machine learning model. It should be noted that the 5 stock trading orders of Company A and 3 stock trading orders of Company B will be trained in the same machine learning model. Because the three stock trading orders of the company belong to different companies, the machine learning model will divide the five stock trading orders of company A into group A, and the three stock trading orders of company B into group B. Use For field comparison. The scheduled schedule is set through a Robotic Process Automation (RPA) system. For example, at 3:30 pm on a working day, the process robot system obtains the external data to be identified from the external source server system 2 by recording the user's operations previously.

After receiving the external data to be recognized and the internal data to be recognized, the data recognition module 3 first uses optical character recognition (Optical Character Recognition, OCR) technology to identify the external data to be recognized and the internal data to be recognized. The non-text content in is extracted and converted into the corresponding text content. The data recognition module 3 then inputs the external data to be identified and the internal data to be identified that only contain the text content into the machine learning models respectively, that is, corresponding to the at least one attachment file of the same product. will be input into the machine learning model corresponding to the product to obtain corresponding externally recognized data and internally recognized data respectively. The externally identified data and the internally identified data respectively include multiple transaction records, and these transaction records contain multiple field data with different formats. For example, the file type of the externally identified data and the internally identified data is a csv file format, and each column in the csv file is equivalent to one of the transaction records.

The data conversion module 4 is electrically connected to the data identification module 3 to receive the externally identified data and the internally identified data, and performs a plurality of conversion procedures on the externally identified data and the internally identified data respectively, to Corresponding external converted data and internal converted data are obtained respectively, and the transaction records of the external converted data and the internal converted data respectively contain the field data in the same format.

The conversion procedures include a format conversion procedure, a data merging procedure, a terminology conversion procedure, an expansion field procedure, and an input code procedure. This format converter is related to the format conversion of date representation. For example, date representation methods include 110/10/7, 7.Oct.2021, 2021.10.7, etc., all of which are uniformly converted into a predetermined format, such as 2021/10/7. The format conversion process can also be related to converting text to numbers or text to date. For example, uppercase numbers are converted to 1. For example, November 19, 2021 is converted to 2021/11/19.

The data merging process is related to the aggregation of data for the same purpose. For example, if the three transaction confirmations contained in the external to-be-identified data corresponding to the externally identified data respectively correspond to the stock buying and selling transaction information provided by three securities firms, then the data merging process will follow the preset merging According to the rules, the stock buying and selling information corresponding to these three securities firms will be merged into one data.

This terminology converter is concerned with the name conversion of the way a term is represented. For example, in the section corresponding to stock buying and selling transaction information, the number "1" in the buy or sell field (i.e. one of the data in this field) in the externally identified data indicates "buy", and the number "2" indicates "buy". For "Sell", "1" is converted into "B (Buy)" and "2" is converted into "S (Sell)" through this term conversion process. The field-expanding procedure involves adding at least one of the field data to the data. Similarly, for example, in the part corresponding to the stock trading information, the externally identified data includes the handling fee field, the transaction tax field, the net payment amount field, and the stock code field, but does not include these field data. For the data in the broker name field, the field data of "broker" can be added to the data through the expanded field program to make the data more complete. The input code procedure is associated with executing predetermined code for data processing. Similarly, for example, in the part corresponding to the stock trading information, the transaction amount field of broker A (that is, one of the data in this field) uses thousands (k) yuan as the unit, and the transaction amount fields of other brokers use one yuan as the unit, then the unit The predetermined program code is used to multiply each value of the transaction amount field corresponding to broker A by 1000 to convert it into yuan.

The data comparison module 5 stores a primary key setting value and is electrically connected to the data conversion module 4. The primary key setting value is, for example, 1 or 0. When the data comparison module determines that the primary key setting value (for example, equal to 0) indicates a keyless value comparison form, the data comparison module 5 determines that the external converted data and the internal converted data belong to the same Compare the field data of other corresponding transaction records of the product to generate a comparison result data. That is to say, in this keyless comparison form, each comparison is independent and has no connection with the previous comparison.

The data comparison module 5 also stores a mandatory correspondence relationship among the field data that are set as mandatory conformers. That is to say, the user can set each of the field data in the smart comparison system 100 Whether there is a corresponding mandatory correspondence. When the data comparison module 5 determines that the primary key setting value (for example, equal to 1) indicates a primary key value comparison form, the data comparison module also compares the external converted data and the internal data based on the forced correspondence. The field data in the converted data belonging to the same product and the corresponding transaction records are compared to generate the comparison result data. For example, in the corresponding part of the exchange form information, the exchange amount field (i.e. one of the field data) in the external converted data and the internal converted data is 10.1 yuan and 10.0 yuan respectively, and 10.1 yuan should be judged as compulsorily matching 10.0 yuan, and the user has saved the corresponding compulsory correspondence (that is, 10.1 yuan in the exchange amount field is equal to 10.0 yuan), then the data comparison module 5 is set to the primary key value ratio In the pair form, when it is judged again that the exchange amount fields are 10.1 yuan and 10.0 yuan respectively, it will be forced to judge that the two are consistent (that is, equal).

In addition, the data comparison module 5 also stores a plurality of weight coefficients for some of the field data and another threshold corresponding to one of the products. The data comparison module 5 also stores a plurality of weight coefficients for some of the field data. Corresponding to the threshold value, the external converted data and the internal converted data belong to the same product category and corresponding field data are compared to generate the comparison result data. To be more specific, the sum of the weighting coefficients in the same product category is equal to a preset maximum sum, and the data comparison module 5 calculates that the corresponding field data in the product category is judged. The sum of the at least one weight coefficient corresponding to the matching person is used to obtain an added value. When the data comparison module 5 determines that the total value is equal to the maximum sum value, the generated comparison result data including the other field data corresponding to the product is completely consistent. When the data comparison module 5 determines that the total value is greater than or equal to the threshold and less than the maximum sum, the generated comparison result data includes other field data corresponding to the product and is suspected to be consistent. . When the data comparison module 5 determines that the total value is less than the threshold, the generated comparison result data including other field data corresponding to the product is completely inconsistent.

It should be noted that the setting of the weight coefficient of each field data is implemented before the external converted data and the internal converted data are imported into the data comparison module 5. The range of each weight coefficient is, for example, 0 to 100, and the maximum total value is, for example, 100. The weight coefficient of the more important field data is set to a higher value, and the weight coefficient of the completely unimportant field data is set to a higher value. The weight coefficient is set to 0, for example. For example, when the product of the transaction confirmation is a stock transaction order, the most important field data among the field data is the stock code, and the other field data in descending order of importance are The order is the balance and delivery date. When the product of the transaction confirmation is a foreign exchange transaction order, the most important field information among these fields is the name of the broker (Broker), and the other fields in descending order of importance are The data are amount, currency, and exchange rate in order. When the product in the transaction confirmation is a bond transaction order, the most important field data among the field data is the bond code (ISIN CODE), and the corresponding weight coefficient is set to 80, for example. The important field data is the inventory balance, and the corresponding weight coefficient is set to 20. The weight coefficients of the other field data are set to 0, and the threshold is set to 80. When the bond code of the field data of the external converted data and the internal converted data is the same, and the inventory balance of the field data is different, the comparison result data generated by the data comparison module 5 includes the corresponding The data in these fields are suspected to match. When the bond code of the field data and the inventory balance of the field data of the external converted data and the internal converted data are the same, the comparison result data generated by the data comparison module 5 includes the corresponding The data in these fields are completely consistent.

In addition, it is important to add that: for example, there are 10 rows (i.e. 10) of transaction records in the csv file of the externally identified data, and there are also 10 rows of transaction records in the csv file of the internally identified data. This data is more than Module 5 compares one of the 10 columns of transaction records in the csv file of the externally identified data with the 10 columns of transaction records in the csv file of the internally identified data one by one, and generates corresponding The coefficient of the total value of 10 transactions is determined, and the highest score among the total values of the 10 transactions is used as the benchmark for comparison. Or, if the 10 columns of transaction records in the CSV file of the externally identified data have a corresponding relationship with the 10 columns of transaction records in the CSV file of the internally identified data, then the data comparison module 5 selects one of the two CSV files. Compare transaction records in the same column.

The data comparison module 5 also stores a comparison schedule. The data comparison module 5 compares the corresponding external converted data and the internal converted data according to the time point indicated by the comparison schedule. The data in these fields are compared to generate the comparison result data. For example, at 3:30 pm every working day, the internal source server system 1 will generate the internal transaction data, and the external source server system 2 will generate the corresponding external transaction data. Therefore, the data is smaller than The comparison module 5 can set the data comparison execution time, frequency and number of repeated executions according to the time point indicated by the comparison schedule. Execution frequency can be daily, weekly, or monthly. For example, at 4:30 pm every working day, data comparison is performed.

The client device 6 is electrically connected to the data comparison module 5 and receives the comparison result data to display the comparison result data.

To sum up, the above embodiments have the following advantages: Advantage 1: The smart comparison system 100 of the present invention obtains the corresponding externally recognized data and the internally recognized data through the data recognition module 3 and transmits them to the data. The conversion module 4 is used to obtain the transaction records of the external converted data and the internal converted data in the same format and send them to the data comparison module 5, which compares the external converted data The data is compared with other transaction records of the same product in the internally converted data to generate the comparison result data. Compared with manual identification and comparison of data, it can save time and effort.

The second advantage is that the smart comparison system 100 of the present invention adopts low-level programming design, such as imported data format adjustment, conversion, filtering, comparison logic condition setting, comparison power weight adjustment, automatic scheduling, and automatic notification, which greatly reduces the cost. The user needs to communicate with the information management personnel during operation, and it also reduces the time for the information management personnel to maintain the smart comparison system 100 of the present invention. Therefore, the purpose of the present invention can indeed be achieved.

However, the above are only examples of the present invention. They cannot be used to limit the scope of the present invention. All simple equivalent changes and modifications made based on the patent scope of the present invention and the contents of the patent specification are still within the scope of the present invention. within the scope covered by the patent of this invention.

1: Internal source servo system 2: External source servo system 3: Data recognition module 4: Data conversion module 5: Data comparison module 6: Client device 100:Smart comparison system

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: FIG. 1 is a block diagram illustrating an embodiment of the smart comparison system of the present invention.

1: Internal source servo system

2: External source servo system

3: Data recognition module

4: Data conversion module

5: Data comparison module

6: Client device

100:Smart comparison system

Claims (7)

An intelligent comparison system is applicable to an internal source server system and an external source server system, and includes: a data recognition module, including multiple machine learning models corresponding to different products, and according to a The scheduled schedule obtains an external data to be identified from the external source server system and an internal data to be identified from the internal source server system, and inputs the external data to be identified and the internal data to be identified into the machines respectively Learning model to obtain corresponding externally identified data and internally identified data respectively. The externally identified data and the internally identified data respectively include multiple transaction records. These transaction records include multiple columns with different formats. bit data; a data conversion module electrically connected to the data identification module to receive the externally identified data and the internally identified data, and perform a plurality of conversion processes on the externally identified data and the internally identified data respectively. , to obtain corresponding externally converted data and internally converted data respectively, and the transaction records of the externally converted data and the internally converted data respectively contain the field data in the same format; and a data comparison To the module, electrically connect the data conversion module, and compare the external converted data and the other transaction records of the same product in the internal converted data to generate a comparison result data; wherein , the data comparison module also stores a primary key setting value. When the data comparison module determines that the primary key setting value indicates a keyless comparison mode, the data comparison module compares the external converted data and The transaction records that belong to the same product category and correspond to the internally converted data The recorded field data is compared to generate the comparison result data; the data comparison module also stores a mandatory correspondence relationship among the field data that is set as a mandatory match. When the data is compared When the module determines that the primary key setting value indication belongs to a primary key value comparison form, the data comparison module also compares the external converted data and the internal converted data that belong to the same product category based on the mandatory correspondence. The field data of the corresponding transaction records are compared to generate the comparison result data. The smart comparison system as described in claim 1 is also applicable to a client device that is electrically connected to the data comparison module and receives the comparison result data to display the comparison result data. The intelligent comparison system as described in claim 1, wherein the predetermined schedule is set by a Robotic Process Automation (RPA) system. The intelligent comparison system as described in claim 1, wherein the conversion procedures include a format conversion procedure, a data merging procedure, a term conversion procedure, an expansion field procedure, and an input code procedure, and the format conversion procedure The procedure is related to the format conversion of the date representation, the data merging procedure is related to the integration of data for the same purpose, the term conversion procedure is related to the name conversion of the term representation, and the expand field procedure is related to the addition of these columns in the data. At least one of the bits of data, the input code procedure is related to executing predetermined code for data processing. The smart comparison system as described in claim 4, wherein the data comparison module also stores multiple weighting coefficients of some of the field data, and compares According to one of the other thresholds of the product, the data comparison module also uses the weight coefficient and the corresponding threshold to compare the external converted data and the internal converted data that belong to the same product category and correspond to each other. The data in these fields are compared to generate the comparison result data. The smart comparison system as described in claim 5, wherein the sum of the weight coefficients in the same product category is equal to a preset maximum sum, and the data comparison module calculates the corresponding value in the product category. The sum of the at least one weight coefficient corresponding to the field data that is judged to be consistent is used to obtain a summed value. When the data comparison module determines that the summed value is equal to the maximum sum, it is generated The comparison result data includes other field data corresponding to the product and is completely consistent. When the data comparison module determines that the summed value is greater than or equal to the threshold and less than the maximum sum, the generated The comparison result data includes other field data corresponding to the product that are suspected to be consistent. When the data comparison module determines that the summed value is less than the threshold, the generated comparison result data includes the data corresponding to the product. The data in these fields for other products are completely inconsistent. The smart comparison system as described in claim 4, wherein the data comparison module also stores a comparison schedule, and the data comparison module performs the external conversion according to the time point indicated by the comparison schedule. The final data is compared with the corresponding field data in the internally converted data to generate the comparison result data.

Images