TWI712950B - Data processing method and apparatus - Google Patents

Abstract

The present invention provides a data processing method used to direct streaming sequential data having raw data to a Not Only SQL (NoSQL) data base. The data processing method includes the following steps: generating a first sequential data point that includes a tuple data, a first test time and a first test result according to the raw data; determining whether the NoSQL data base includes a second sequential data point that includes the tuple data; if affirmative, generating a third sequential data point that includes the tuple data, the third test time and the third test result according to the first and second sequential data points, replacing the second sequential data points with the third sequential data points and to store into the NoSQL data base; and if not, storing the first sequential data point into the NoSQL data base.

Description

Data processing method and device

The present invention relates to a data processing method and device, in particular to a method and device that can directly and efficiently process non-sequential time series data.

In the product manufacturing process, it will be imported into different functional stations (each assembly station, test station, maintenance station), each product will flow through various stations along the production line, and each product will flow through each station At this time, because repeated tests may be required, a set of timing data will be generated.

In the existing practice, if there are too many objects in the timing data and the timing is too long, a large amount of storage space will be consumed. And when the time series data is obtained non-sequentially, it is necessary to re-sort the time series data, which greatly reduces the calculation speed. Therefore, how to efficiently and quickly process non-sequential data and save storage space through the improvement of data calculation methods to overcome the above-mentioned shortcomings has become one of the important issues to be solved by this business.

The technical problem to be solved by the present invention is to provide a data processing method in view of the shortcomings of the prior art, which can process sequential and non-sequential time series data at the same time, does not affect the operation efficiency due to too long time series, and can effectively use the data storage space .

In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a data processing method for importing at least one stream time series data into a non-relational database. The stream time series data includes original data, and data processing The method includes: generating a first time series data point based on the original data by a processing unit, the first time series data point including at least two-tuple data, a first test time, and a first test result; Determine whether a second time series data point including binary data is already stored in the non-relational database; if the processing unit determines that there is a second time series data point in the non-relational database, the logic unit further performs the following Steps: compare the first test time of the first sequential data point and the second test time of the second sequential data point; use the later of the first test time and the second test time as the third test time; A test result and the second test result of the second sequential data point perform logical operations to generate a third test result; a third sequential data point is generated according to the third test time and the third test result, where the third sequential data point includes The binary data, the third test time, and the third test result; and the third time series data point is substituted for the second time series data point and stored in the non-relational database.

In order to solve the above technical problem, another technical solution adopted by the present invention is to provide a data processing device for importing at least one stream of time series data into a non-relational database, and the at least one stream of time series data includes original data. The data processing device includes: a processing unit and a logic unit. The processing unit generates a first time series data point based on the original data. The first time series data point includes at least two-tuple data, a first test time, and a first test result, and determines whether the non-relational database already contains The second sequential data point of the binary data. The logic unit is coupled to the processing unit. If the logic unit determines that there is a second time series data point in the non-relational database, the logic unit compares the first test time of the first time series data point and the second time series data point The second test time, the later of the first test time and the second test time is used as the third test time, and the logic operation is performed on the first test result and the second test result of the second sequential data point to generate the first test result Three test results, generating a third time series data point based on the third test time and the third test result, and replacing the second time series data point with the third time series data point and storing it in the non-relational database; where the third time series data point includes Two-tuple data, third test time, and third test result.

One of the beneficial effects of the present invention is that the data processing method provided by the present invention can establish a multi-level structure of data flow by "using the tuple of each product number and each station" and "only comparing The technical solution of "update the product status in time for two pieces of data with the same two-tuple" will greatly increase the computing efficiency and reduce the storage space of the system.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings about the present invention. However, the provided drawings are only for reference and description, and are not used to limit the present invention.

Referring to FIG. 1, FIG. 1 is a block flowchart of a data processing method according to an embodiment of the present invention. The data processing method of the embodiment of the present invention is used for real-time management of the test time and test results of products at different test stations. The data processing method of the present invention is mainly achieved by a processing unit and a logic unit to execute a program stored in a non-transitory computer-readable medium. It is worth noting that the processing unit does not need to perform any logical operations, as long as the original data can be formatted. The logic unit can be any firmware or software with a logic operation function, such as a processor, arithmetic unit, instructions in the memory, a programming language, etc., and the present invention does not impose any limitation on this. The product in this embodiment is a mobile phone, but the types of products applicable to the present invention are not limited to this.

The data processing method of Figure 1 may include the following steps:

Step S1: Import the product number of the product in the non-relational database. Step S2: Import the first test time and the first test result of the product at the test station in the non-relational database. Generally speaking, after a product has been tested at multiple test stations, the test time and test results of each product are recorded. At this time, in order to track and control the yield of the product, the product number (such as production serial number, model, barcode, etc.) and test station (number or name of the test station) of the product itself, together with the above-mentioned test time and test results, will be A set of streaming time series data is formed and imported into a non-relational database for storage, as shown in steps S1 to S2 in Figure 1. It is worth noting that in practical applications, the data processing method of the present invention is not only limited to test stations, but can also be applied to maintenance stations or assembly lines. Of course, it can also be used as disclosed in this embodiment. Product status tracking for Test and Repair.

Step S3: Utilize the processing unit to consolidate the product number and the test station into a tuple data. Then, the processing unit composes the first time sequence data point of the product composed of the two-tuple data composed of the product number and the test station, the first test time, and the first test result and stores it in the non-relational database. More precisely, referring to FIG. 2, the data format of the time series data point of this embodiment is composed of three parts in sequence: a key field, a test time field, and a value field. . For example, the key value field of the first time series data point 210 is used to store the binary data A20170912X01_ST1, which is the product number "A20170912X01" and the test station type "ST1"; the time value of the first time series data point 210 The field is used to store the first test time "20170921081820"; and the value field of the first time series data point 210 is used to store the first test result and/or the first maintenance record "Pass". Similarly, the key value field of the second time series data point 220 is used to store the binary data A20170912X01_ST1; the time value field of the second time series data point 220 is used to store the second test time "20170921093854"; and the second time series data The value field of point 220 is used to store the first test result and/or the first maintenance record "Pass". From the above, it can be seen that the key value fields of the first time series data point 210 and the second time series data point 220 have the same two-tuple data, which are represented by the same test station for testing. However, the time value field and the value field of the first time series data point 210 and the second time series data point 220 respectively have different values, which means that the same product is tested at the same test station at different test time points. Get different test results. It is worth noting that, different from the traditional time series data processing method, the data processing method of the present invention first aggregates the product's binary data (product number and test station type) and places it in the key value field of the time series data point. The time series data points are stored in the non-relational database, which is beneficial for subsequent comparisons.

Step S4: The processing unit is used to determine whether a second time series data point including the two-tuple data is already stored in the non-relational database. Step S5: If the processing unit determines that the second time series data point already exists in the non-relational database, the logic unit further performs logic operations. Step S6: Conversely, if the processing unit determines that the second time series data point is not stored in the non-relational database, the processing unit stores the first time series data point in the non-relational database. Simply put, in step S4, the data processing method of the present invention determines whether a second time series data point including the two-tuple data already exists in the non-relational database, that is, judging the non-relational database Whether there are other test results of the same product at the same test station (ie: the same two-tuple data) at different times. This means that if the non-relational database already contains the second sequence data point including the data of the two-tuple (product-test station), it means that the product has been tested at the test station and the test time and Test Results. In other words, the second time series data point has the two-tuple (the same two-tuple as the first time series data point), the second test time (different from the first test time of the first time series data point), and the second 2. Test results. If the result is that the non-relational database already has a second time series data point, the data processing method of the present invention proceeds to step S5: the logic unit performs a logic operation. If the result is that there is no second time series data point, it means that this product has only been tested this time at this test station. Therefore, the data processing method of the present invention directly proceeds to step S6: the processing unit stores the first time series data point in Non-relational database. It must be noted that the binary data of the present invention is a combination of product-testing station. Therefore, the above-mentioned "including binary data" must be the same product number and the same test station. In view of the fact that the same product may be tested at different test stations, in this situation, the binary data will be different. Furthermore, thanks to the data processing method of the present invention, the binary data of the product (product number and test station) are first collected and placed in the key value field of the time series data point. Therefore, the data comparison step here , The processing unit only needs to compare the key value field of the time series data point existing in the non-relational database with the key value field of the first time series data point, confirm that the key value is exactly the same, and then perform the following field Logical operation effectively saves the time of data sorting and comparison, and can calculate and update the current status of the product in real time.

The logic operation includes the following steps. Step S51: Compare the first test time of the first time series data point and the second test time of the second time series data point. Step S52: Use the later one of the first test time and the second test time as the third test time. Step S53: Generate a third time series data point, where the third time series data point includes binary data, a third test time, and a third test result. If the processing unit determines that the second time series data point already exists, the logic unit compares the time value field and the value field of the first time series data point and the second time series data point. First, the logic unit compares the first test time and the second test time in the time value field of the first sequential data point and the second sequential data point, and generates a third test according to the first test time and the second test time Time (ie: the updated test time). The third test time is the relatively later test time of the first test time and the second test time. In other words, if the first test time is earlier than the second test time, the third test time is equal to the second test time. If the first test time is later than the second test time, the third test time is equal to the first test time. In other words, the same product is tested at different time points at the same test station, and only the relatively late time point will be recorded.

Further refer to Figure 3, which is the truth table of the test results. When the logic unit performs a logical operation on the first test result and the second test result in the numerical fields of the first time series data point and the second time series data point, the following various logical operation results will be generated, namely: the third test result . The third test result may include: a pass, a fail, and a retest pass. Referring to the table in Figure 3, if the test result A with an earlier test time is the test passed and the test result B with a later test time is the test passed, the logical operation result (ie, the third test result X) is the test passed. If the test result A with the earlier test time is the test passed, and the test result B with the later test time is the test failed, the third test result X is the test failed. If the test result A with the earlier test time is the test failure, and the test result B with the later test time is the test passed, the third test result X is the retest passed. If the test result A with an earlier test time is a test failure, and the test result B with a later test time is a test failure, the third test result X is a test failure. Then, the logic operation is continued. If the test result A with the earlier test time is the test passed, and the test result B with the later test time is the retest passed, the logic operation result X is the retest passed. If the test result A with the earlier test time is passed the re-test, and the test result B with the later test time is the test passed, the third test result X is the re-test passed. If the test result A with the earlier test time is passed the retest, and the test result B with the later test time is the test failed, the third test result X is the test failed. If the test result A with an earlier test time is a test failure, and the test result B with a later test time is a retest passed, the third test result X is a retest passed. It is worth noting that although a retest pass means that the product passed the final test, it is different from the pass state in terms of product yield.

Refer to Figures 4A and 4B, Figures 4A and 4B are the third test results obtained at different time points R1, R2, and R3 for Figure 3, and the test result A with the earlier test time is regarded as the test failure (fail) and the test time is longer. The late test result B is an example of a pass. If the time series data points of the product are retrieved from the non-relational database at time R1, because the product has not been tested, the third test result X is none; if it is from the non-relational database at time R2 The time sequence data point data of the product is retrieved in the, because the product is only tested for the first time and the result is a test failure, the third test result X is the test failure; if the product is retrieved from the non-relational database at time R3 Because the product has been tested twice at the test station and has different test results, the third test result X obtained after the logic operation of this embodiment is passed the retest. It can be known that although the last test result of the product at the test station was passed, the previous test has failed.

After the third test time of the third time series data point and the third test result are generated, the logic unit proceeds to step S54: replace the second time series data point with the third time series data point and store it in the non-associated database. Simply put, it is to update the test data of the product at the test station. Of course, as mentioned above, the part of the value column can also be included in the maintenance record of the product at the repair station. The logical operation method and data processing method are the same as the above, so I will not repeat it here.

One of the beneficial effects of the present invention is that the data processing method provided by the present invention can establish a multi-level structure of data flow by "using the tuple of each product number and each station" and "only comparing The technical solution of "two pieces of data with the same two-tuple and timely update of the product test status", thereby greatly improving computing efficiency. On the other hand, in the non-relational database, only the latest test time and updated test results of each product are stored. Compared with the conventional technology, which stores different test times and different test results of each product, the present invention The data processing method can reduce storage space.

Furthermore, compared to traditional data processing methods, all data needs to be loaded into memory or stored in the database, filtered out all test data of the product at the same station, and the order of all test data is compared one by one. , The final test result can be judged by obtaining the correct order of all test data. The data processing method of the present invention only needs to compare two sets of test data and update the current test status of the product in real time. It can not only process sequential and non-sequential data at the same time Time series data does not affect computing efficiency due to too long time series, and can effectively use data storage space. Furthermore, the data processing method of the present invention does not need to wait until all the test data of the product at the site is fully available to know the test status of the product, but can update the real-time test results of the data at different points in time, which is practical in operation. More effectively grasp the exact dynamics of the product.

The content disclosed above is only a preferred and feasible embodiment of the present invention, and does not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the description and schematic content of the present invention are included in the application of the present invention. Within the scope of the patent.

no

Figure 1 is a block flow diagram of the data processing method of the present invention.

FIG. 2 is a schematic diagram of the format of the time series data point of the present invention.

Figure 3 is a schematic diagram of the truth table of the test results of time series data points.

4A and 4B are schematic diagrams of the test results obtained at different time points for FIG. 3.

Claims (18)

A data processing method is used to import at least one stream time series data into a non-relational database, the at least one stream time series data includes a raw data, and the data processing method includes: by a processing The unit generates a first time series data point based on the original data, the first time series data point at least including a tuple data, a first test time, and a first test result; and The processing unit determines whether a second time series data point including the binary data is already stored in the non-relational database, and if the processing unit determines that the non-relational database already exists If there is the second sequential data point, a logic unit is used to further perform the following steps: compare the first test time of the first sequential data point and a second test of the second sequential data point Time; the later of the first test time and the second test time is used as a third test time; a second test result of the first test result and the second sequential data point Perform logic operations to generate a third test result; generate a third time series data point according to the third test time and the third test result, wherein the third time series data point includes the binary data , The third test time and the third test result; and the third time series data point is substituted for the second time series data point and stored in the non-associated database. For example, the data processing method described in item 1 of the scope of patent application further includes: if the processing unit determines that the second sequential data point is not stored in the non-relational database, the processing unit will The first sequential data point is stored in the non- Relational database. Such as the data processing method described in item 1 of the scope of patent application, wherein the two-tuple data includes a product number and a station category. The data processing method described in item 1 of the scope of patent application, wherein the data format of each of the first time series data point, the second time series data point, and the third time series data point is composed of a A key value field, a time value field, and a value field are composed, wherein the key value field of the first time series data point includes the two-tuple data, and the first time series data point The time value field includes the first test time, and the value field of the first time series data point includes the first test result. For example, the data processing method described in item 4 of the scope of patent application, wherein the step of determining whether the second time series data point already exists in the non-relational database includes only checking whether the non-relational database contains The value of the key value field of the first time series data point. The data processing method described in item 1 of the scope of patent application, wherein each of the first test result, the second test result, and the third test result includes one of the following: test passed, test Failed and passed the retest. For example, the data processing method described in item 6 of the scope of patent application, wherein: if the first test result is a test passed and the second test result is a test passed, then the third test result is a test passed; If the first test result is a test passed, the second test result is a test failure, and the first test time is earlier than the second test time, then the The third test result is a test failure; if the first test result is a test passed, the second test result is a test failure, and the second test time is earlier than the first test time, then the third The test result is a retest passed; and if the first test result is a test failure and the second test result is a test failure, the third test result is a test failure. For example, the data processing method described in item 6 of the scope of patent application, wherein: if the first test result is passed the re-test and the second test result is the test passed, the third test result is the re-test passed; The first test result is a retest passed, the second test result is a test failure, and the first test time is earlier than the second test time, then the third test result is a test failure; A test result is a retest passed, the second test result is a test failure, and the second test time is earlier than the first test time, the third test result is a retest passed; and if the first test time The test result is the test passed, and the second test result is the retest passed, and the third test result is the retest passed. According to the data processing method described in item 1 of the scope of patent application, the non-relational database includes one of the following: HBase, Dynamo, BigTable, Cassandra, Hypertable, and Mongodb. A data processing device is used to import at least one stream time series data into a non-relational database, the at least one stream time data includes a raw data, and the data processing device includes: a processing unit, A first time series data point is generated according to the original data, so The first time series data point includes at least a tuple data, a first test time, and a first test result, and it is determined whether the non-relational database already contains the tuple data A second time series data point of data; and a logic unit, coupled to the processing unit, if the logic unit determines that the second time series data point already exists in the non-relational database, the The logic unit compares the first test time of the first time series data point and a second test time of the second time series data point, and the first test time and the second test time are later One is used as a third test time, and a logic operation is performed on the first test result and a second test result of the second time series data point to generate a third test result and a third time series data point , And replace the second time series data point with the third time series data point and store it in the non-relational database; wherein, the third time series data point includes the two-tuple data, the third Test time and a third test result. For example, the data processing device according to item 10 of the scope of patent application, wherein, if the processing unit determines that the second time series data point is not stored in the non-relational database, the processing unit additionally The first time sequence data point is stored in the non-relational database. For example, the data processing device described in item 10 of the scope of patent application, wherein the two-tuple data includes a product number and a station category. The data processing device according to claim 10, wherein the data format of each of the first time series data point, the second time series data point, and the third time series data point is composed of a Key value field, a time value field and a value The key value field of the first time series data point includes the two-tuple data, and the time value field of the first time series data point includes the first test Time, and the value field of the first time series data point includes the first test result. The data processing device according to item 13 of the scope of patent application, wherein the step of the processing unit determining whether the second time series data point already exists in the non-relational database includes the processing unit only checking all Whether the non-relational database includes the value of the key value field of the first time series data point. The data processing device according to claim 10, wherein each of the first test result, the second test result, and the third test result includes one of the following: test passed, test Failed and passed the retest. For example, the data processing device described in item 15 of the scope of patent application, wherein: if the first test result is a test passed and the second test result is a test passed, then the third test result is a test passed; If the first test result is a test passed, the second test result is a test failure, and the first test time is earlier than the second test time, the third test result is a test failure; A test result is a test passed, the second test result is a test failure, and the second test time is earlier than the first test time, the third test result is a retest passed; and if the first test time The test result is a test failure, and the second test result is a test failure, and the third test result is a test failure. Such as the data processing device described in item 16 of the scope of patent application, in which: If the first test result is passed the retest and the second test result is passed the test, then the third test result is passed the retest; if the first test result is passed the retest, the second test result is The test fails, and the first test time is earlier than the second test time, the third test result is a test failure; if the first test result is a retest passed, the second test result is a test failure , And the second test time is earlier than the first test time, the third test result is passed the retest; and if the first test result is the test passed, the second test result is the retest passed, Then the third test result is passed the retest. According to the data processing device described in item 10 or 11 of the scope of patent application, the non-relational database includes one of the following: HBase, Dynamo, BigTable, Cassandra, Hypertable, and Mongodb.

Images