KR102620080B1 - Method and apparatus for processing order data - Google Patents

Abstract

The present invention relates to a data verification method for an electronic device, comprising: generating statistical data about an order based on order data; generating baseline data to verify the statistical data based on the order data; Comparing incremental values of each of the baseline data and the statistical data; and verifying the statistical data based on comparison of the incremental values.

Description

Method and device for processing order data {METHOD AND APPARATUS FOR PROCESSING ORDER DATA}

Embodiments of this specification relate to methods and devices for processing order data.

As the use of the Internet becomes more widespread, the e-commerce market is expanding. In particular, as infectious diseases spread, the proportion of products purchased by visiting offline stores is decreasing, while the proportion of products purchased through e-commerce using computers or smartphones is rapidly increasing.

E-commerce companies need to collect and generate statistical data such as order volume, page views, item clicks, etc. for analysis and decision-making related to e-commerce services for entities such as users. In order to generate statistical data based on order data, it is important to analyze all of a user's orders in real time, but this is quite difficult. A user's order history occupies a very large amount of data, and individual orders among these have changing characteristics that can be created and canceled at any time, making it difficult to analyze order data in real time. Therefore, there is a need for a method that can generate statistical data by analyzing large amounts of changing data in real time.

Additionally, validating statistical data generated in real time through a data pipeline is a very important task. Statistical data is updated in real time and users will use the updated statistical data each time the statistical data is updated. However, if a problem occurs in any part of the entire data pipeline, statistical data generated through the data pipeline may have data quality problems, for example, newly distributed code to implement the data pipeline. There may be a bug. If these problems are discovered late, they may cause other problems in the operation of e-commerce services, so to prevent this, it is necessary to periodically perform data quality checks on statistical data. However, data dumping and processing of aggregated data such as metrics (e.g., number of SDP views, number of clicks, total order amount, etc.) takes a lot of time, making it difficult to easily check the latest data, and data such as user order amount can contain 8TB of data, so a method to efficiently perform data quality inspection of large amounts of data is required.

A prior document regarding such a data processing method is Korean Patent Publication No. 10-2009-0121754.

The embodiment of the present disclosure is proposed to solve the above-mentioned problem, and generates statistical data based on real-time order data, and compares the incremental values of each statistical data with the baseline data also generated based on the order data. By verifying statistical data, the purpose is to efficiently perform statistical data analysis and verification of large amounts of data that change in real time.

In order to achieve the above-described problem, a data verification method for an electronic device according to an embodiment of the present disclosure includes generating statistical data about an order based on order data; generating baseline data to verify the statistical data based on the order data; Comparing incremental values of each of the baseline data and the statistical data; and verifying the statistical data based on comparison of the incremental values.

According to one embodiment, the order data includes an order identifier and order quantity for one or more orders, and the statistical data includes the total of the order quantities.

According to one embodiment, generating the statistical data includes receiving an order cancellation event; Confirming the quantity of orders canceled from the order cancellation event; and adding the canceled order quantity multiplied by -1 to the statistical data.

According to one embodiment, the statistical data is characterized in that it is generated through a data pipeline that guarantees exactly once processing of the order data.

According to one embodiment, the data pipeline includes a first pipeline that guarantees at least once processing and a second pipeline that guarantees exactly once processing, and the output of the first pipeline is Characterized in that it is input to the second pipeline.

According to one embodiment, the step of generating the statistical data is performed when R(n+1) indicating the n+1th change of the order data is input to the first pipeline: retrieving R(n) representing the change; Multiplying the value of R(n) by -1; transmitting -R(n) and R(n+1) to the second pipeline; and caching the R(n+1).

According to one embodiment, generating the statistical data further includes generating the statistical data by aggregating the -R(n) and R(n+1) in the second pipeline. It is characterized by

According to one embodiment, R(n) is characterized in that it includes an order ID and an order version.

According to one embodiment, the step of generating the baseline data may further include verifying the baseline data based on the order data.

According to one embodiment, the incremental value is calculated at a set cycle based on the statistical data.

An electronic device for verifying data according to an embodiment of the present disclosure includes: a memory storing at least one command; and generating statistical data for an order based on order data by executing the at least one command, generating baseline data for verifying the statistical data based on the order data, and generating the baseline data and the It is characterized in that it includes a processor that compares the increment values of each piece of statistical data and verifies the statistical data based on the comparison of the increment values.

A non-transitory computer-readable storage medium according to an embodiment of the present disclosure includes a medium configured to store computer-readable instructions, wherein the computer-readable instructions, when executed by a processor, cause the processor to: generating statistical data about orders based on; generating baseline data to verify the statistical data based on the order data; comparing incremental values of each of the baseline data and the statistical data; and verifying the statistical data based on comparison of the incremental values.

According to an embodiment of the present disclosure, the electronic device stores only data that should be reflected in the statistical data of the order in a first database, such as a counteraction record, and stores the amount of the canceled order in the first database multiplied by -1. By updating statistical data using , statistical data such as metrics can be calculated and updated in real time without retrieving all data from existing order history.

In addition, according to an embodiment of the present disclosure, the electronic device is not affected by the problem and changes the order even when there is an order error in the order data streaming to the message queue or a conflict occurs during data processing in the data pipeline. The integrity of real-time data processing can be effectively maintained by performing exception handling that allows the state to be normally reflected in the final statistical data.

In addition, according to an embodiment of the present disclosure, if the electronic device performs a data quality check at time T, the accuracy of the data before time T has been proven, and in order to perform a data quality check at time T+1, time T to T By performing a data quality check only on incremental values between +1, the accuracy of the data up to time T+1 can be proven, and thus the electronic device can verify the integrity of the data at all time intervals without unnecessary increase in the amount of computation. .

1 is an exemplary diagram schematically illustrating each configuration of an electronic device according to an embodiment of the present disclosure.
Figure 2 is an exemplary diagram for explaining a method of calculating statistical data on orders according to an embodiment of the present disclosure.
3A to 3C are exemplary diagrams illustrating a real-time data pipeline design for generating statistical data of an electronic device according to an embodiment of the present disclosure.
4 shows an example architecture representing data flows associated with a baseline generator and a data quality verifier according to an embodiment of the present disclosure.
5A and 5B are exemplary diagrams showing the operation of a data quality verifier according to an embodiment of the present disclosure.
Figure 6 is a flowchart showing the flow of a data verification method for an electronic device according to an embodiment of the present disclosure.
7A to 7C are exemplary diagrams for explaining a design for generating statistical data according to a plurality of embodiments of the present disclosure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

In describing the embodiments, description of technical content that is well known in the technical field to which the present invention belongs and that is not directly related to the present invention will be omitted. This is to convey the gist of the present invention more clearly without obscuring it by omitting unnecessary explanation.

For the same reason, some components are exaggerated, omitted, or schematically shown in the accompanying drawings. Additionally, the size of each component does not entirely reflect its actual size. In each drawing, identical or corresponding components are assigned the same reference numbers.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

At this time, it will be understood that each block of the processing flow diagram diagrams and combinations of the flow diagram diagrams can be performed by computer program instructions. These computer program instructions can be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions performed through the processor of the computer or other programmable data processing equipment are described in the flow chart block(s). It creates the means to perform functions. These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce manufactured items containing instruction means that perform the functions described in the flowchart block(s). Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in the flow diagram block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative execution examples it is possible for the functions mentioned in the blocks to occur out of order. For example, it is possible for two blocks shown in succession to be performed substantially at the same time, or it is possible for the blocks to be performed in reverse order depending on the corresponding function.

At this time, the term '~unit' used in this embodiment refers to software or hardware components such as FPGA or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

1 is an exemplary diagram schematically illustrating each configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1 , the electronic device 100 may include a processor 110 and a memory 120 and may perform a method of processing order information. In the electronic device 100 shown in FIG. 1, only components related to the present embodiments are shown. Accordingly, it is obvious to those skilled in the art that the electronic device 100 may further include other general-purpose components in addition to the components shown in FIG. 1 .

The processor 110 serves to control overall functions for processing order information in the electronic device 100. For example, the processor 110 generally controls the electronic device 100 by executing programs stored in the memory 120 within the electronic device 100. The processor 110 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the electronic device 100, but is not limited thereto.

The memory 120 is hardware that stores various types of data processed within the electronic device 100. The memory 120 may store data processed and data to be processed in the electronic device 100. Additionally, the memory 120 may store applications, drivers, etc. to be run by the electronic device 100. The memory 120 includes random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD- It may include ROM, Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or flash memory.

In one embodiment, the electronic device 100 may generate statistical data based on order data. In one embodiment, statistical data may include a profile or metric representing characteristics related to an e-commerce service of an entity (e.g., user, category, promotion), for example , may include attributes of entities that can be used directly (e.g., membership sign-up date), aggregated data for specific event types, predictive metrics, etc. Aggregated data includes behavioral statistics such as logins, number of clicks, access times, single detail page (SDP) views, and transactions such as number of orders, gross merchandise volume (GMV), order value, and total order volume. Statistics may include. Predictive metrics may include, for example, a predictive score related to a user's interest in “iPhone.” An e-commerce business may use these for analysis and decision-making related to the entity's e-commerce services. There is a need to collect and generate statistical data for the same.

In order to generate statistical data based on order data, it is important to analyze all of a user's orders in real time, but this is quite difficult. A user's order history occupies a very large amount of data, and individual orders among these have changing characteristics that can be created and canceled at any time, making it difficult to analyze order data in real time.

Specifically, when a user orders one or more items through an e-commerce service, the service server may generate order data representing the user's order information. In one embodiment, order data may include an order identifier and order quantity for one or more orders. After order data has been created, the User may cancel all or part of one or more items included in this order, and such cancellation of an order shall be reflected in the User's order profile along with the order creation date. Below, an example related to creating and canceling a user's order will be described with reference to FIG. 2.

Figure 2 is an exemplary diagram for explaining a method of calculating statistical data on orders according to an embodiment of the present disclosure. Referring to Figure 2, an example of order creation and cancellation is shown through several tables for order details 210 and order quantity metrics 220.

For example, on October 1, 2022, User 1 placed two orders (1001 and 1002), each with order quantities of 100 and 200, then a metric such as "User's Order Quantity" would be 300. Afterwards, User 1 canceled some items with order quantity 50 in order 1002 on October 2, 2022, and the metric of “User’s order quantity” should be changed to 250. However, because the real-time order message is the most recent snapshot, we can know that the order quantity of order 100 is currently 150, but we cannot know that his order quantity was 200 before, which makes it difficult to calculate the change in real-time order quantity. In order to calculate the metric of "user's order quantity", it is generally necessary to access all of the user's past order history data and add up the order quantity. Order history data has a very large amount of data, and metrics are calculated in real time based on this. It's not easy. For example, if the size of the current order data set is 8TB, it is difficult to calculate user metrics in real time, such as the number of orders or quantity of orders.

To address this difficulty, in one embodiment, the electronic device 100 stores some records using a first database to process changes in the status of an order and incrementally calculates metrics related to the order. It can be performed incrementally. For example, in calculating the “user's order quantity” metric, if the user canceled a previous order and the canceled order quantity is 200, the electronic device 100 may determine its counteraction value to be -200. . The electronic device 100 only needs to add this reaction value -200 to the user's order quantity, and there is no need to add all of the user's order quantity, effectively reducing the amount of data required for calculation. The electronic device 100 stores only the data that should be reflected in the statistical data of the order among the streaming real-time data in a first database, such as a counteraction record, and multiplies the amount of the canceled order in the first database by -1. By utilizing statistical data to update, metrics can be calculated in real time without retrieving all data from existing order history.

Similarly, in the example shown in order quantity metric 220 of FIG. 2, the “User’s Order Quantity” metric is created as 300 on October 1, 2022, and due to the user’s order quantity cancellation by 50, the metric is created on October 2, 2022. is treated as -50. The electronic device 100 may generate statistical data by receiving an order cancellation event, checking the canceled order quantity from the order cancellation event, and adding the canceled order quantity multiplied by -1 to the statistical data.

3A to 3C are exemplary diagrams illustrating a real-time data pipeline design for generating statistical data of an electronic device according to an embodiment of the present disclosure.

3A, an end-to-end real-time data pipeline architecture 300 is shown for the electronic device 100 to generate statistical data, such as metrics 340, from order data 310. . The electronic device 100 can build a data pipeline like the architecture 300 to automate the process of extracting, changing, combining, verifying, and loading real-time streaming data.

In one embodiment, statistical data may be generated through a data pipeline that ensures exactly once processing of order data 310. In one embodiment, the data pipeline includes a first pipeline 320 (e.g., a streaming ingestion pipeline) that guarantees at least once processing and a first pipeline 320 that guarantees exactly once processing. It may include a second pipeline 330 (eg, a real time aggregation pipeline), and the output of the first pipeline may be input to the second pipeline. In real-time data processing, it is important to ensure the accuracy of calculation results, and the data pipeline for accurate calculation may need to have exactly once processing semantics. Exactly once processing can mean that all data should be reflected in the final result exactly once. For example, even if the user cancels the order once, the cancellation data log streamed in real time may be entered repeatedly 10 times, and the electronic device 100 needs to remove such duplicate data and reflect only one canceled order. There is. Meanwhile, the first pipeline 320 only guarantees processing at least once and does not guarantee processing exactly once, so duplicate data can be accepted, and duplicate data is deduplicated in the second pipeline 330. It can be.

In one embodiment, the second pipeline 330 may guarantee at least once processing, deduplication, and idempotent write semantics to ensure exactly once processing. Process at least once semantics ensure that all data is processed, deduplication ensures that duplicate data in the input is removed, idempotent writing ensures that reprocessing data after multiple times does not affect the final output result, In other words, it is possible to guarantee that the result will not change even if the operation is applied multiple times.

In one embodiment, electronic device 100 may generate a unique identifier that can uniquely represent a micro batch of entity data sets to ensure idempotent writing in second pipeline 330. . Because the data in the second pipeline 330 is aggregated using different calculation logic, the original input data identifier is no longer used. The unique identifier can be set as a combination of entity ID, micro-batch ID, and metric ID. The micro-batch ID can be generated using a digest of the starting offset of every micro-batch, and the metric ID can contain a unique identifier of different calculation logic. As such, after a restart of the system or recovery from a crash, the electronic device 100 reprocesses from the start offset of the last failed microbatch, the microbatch ID will be the same as the last processing and the data is overwritten with the same identifier ( Since it will be overwritten, idempotent writing can be guaranteed.

In one embodiment, message queue 350 may include Apache Kafka, which can publish, subscribe, store, and process order data 310 streaming in real time. Apache Kafka is a large-capacity real-time log processing system and a message queuing system that processes messages using the publish/subscribe paradigm. Apache Kafka is a highly scalable, high-throughput distributed messaging system that guarantees stable and continuous message delivery performance. Kafka applies the producer-consumer problem and uses three system components, producer/consumer/broker, to set topics for large amounts of data and partition them based on the topics. Configure and save in order. In Kafka, this stored data is sequentially delivered to the consumer for efficient processing. Kafka is a solution specialized in large-capacity, real-time log processing, and can process data with a fault-tolerant, stable architecture and rapid performance in a messaging system whose main purpose is to safely deliver data without loss.

In one embodiment, the electronic device 100 generates statistical data based on order data when R(n+1) representing the n+1 change in order data is input to the first pipeline 320. When, retrieve R(n) representing the nth change of order data from the cache, multiply the value of R(n) by -1, and replace -R(n) and R(n+1) with the second It may include transmitting to the pipeline 330 and caching R(n+1). The electronic device 100 may generate statistical data by counting -R(n) and R(n+1) transmitted from the first pipeline 320 to the second pipeline 330. That is, the statistical data in the second pipeline 330 is R(0) + [R(1) - R(0)] + ... + [R(n) - R(n-1)] + [R (n+1) - R(n)] = R(n+1).

In one embodiment, R(n) may include information about the order ID and order version. The electronic device 100 verifies information about the order ID and order version included in R(n), and determines the order status (created or canceled) corresponding to the order ID or order version to be reflected in the metric, thereby generating the metric. can be created.

In the following, i) when there is a disorder in the order of order data streaming to the message queue, and ii) when a crash occurs during data processing in the data pipeline, the order is changed without being affected by these problems. This explains the exception handling method that allows the status to be normally reflected in the final statistical data.

In one embodiment, when an order error occurs in the order source message queue, the electronic device 100 may perform exception processing so that only the most recent version of the order data is reflected. For example, if R(n+k+m) is received before previous orders R(n+k) to R(n+k+m-1), in the second pipeline 330, R(n When +k+m) are aggregated, the statistical data is R(0) + [R(1) - R(0)] + ... + [R(n+k-1) - R(n+k- 2)] + [R(n+k+m) - R(n+k-1)] = R(n+k+m). That is, the electronic device 100 reflects only the most recent order data R(n+k+m) in the statistical data through the aggregation method of the second pipeline 330, and the previous order data R(n+ By discarding messages from k) to R(n+k+m-1), the changed order status can be finally reflected in the statistical data without being affected by the occurrence of problems.

In one embodiment, when a conflict occurs during data processing in the data pipeline, the electronic device 100 may perform exception processing so that the changed order status can be reflected in statistical data without being affected by the conflict. In the first pipeline 320, the data processing steps in the general case may be performed as follows N1 to N5:

N1: Check the source message queue and extract useful fields.

N2: Filter disqualified messages.

N3: Convert data into events.

N4: Transmit the converted event to the downstream event message queue.

N5: Commit the source message queue offset.

Here, committing the source message queue offset indicates that the next message is ready to be processed at the source.

Meanwhile, the electronic device 100 may further perform the following additional steps C1 to C4 on the counteraction record (360):

C1: Receives previously stored events from a database management system (DBMS).

C2: Calculate the counteraction value from the reaction record.

C3: Sends the calculated reaction value along with the general event.

C4: In DBMS, the previous event is overridden by a new event.

Here, further steps C1 and C2 can be combined with N3. Even if a collision occurs at stage N3 or before N3, there may be no impact of the collision because the messages in the source message queue are read and processed again. Additional step C3 may be performed together with N4.

In one embodiment, an additional step C4 may be performed before N4. That is, the electronic device 100 may override a previous event with a new event in the DBMS (C4) and then transmit the converted event to the downstream event message queue (N4). At this time, if N4 fails due to a conflict, a conflict may occur in the first pipeline 320 after R(n) is replaced with R(n+1) in the first case. After recovery, electronic device 100 will transmit R(n+1) again downstream, which causes calculations to be performed based on R(n) + R(n+1) in second pipeline 330. can do. Meanwhile, if N4 fails due to a collision, in the second case, only one of R(n+1) or -R(n) is transmitted downstream and a collision may occur in the first pipeline 320. After recovery, if -R(n) is transmitted, it is okay, but if only R(n+1) is transmitted, calculation is performed based on R(n) + R(n+1) in the second pipeline 330. It can be done. Ultimately, in both the first and second cases, a problem may occur because the electronic device 100 performs calculations based on R(n) + R(n+1) in the second pipeline 330.

In one embodiment, an additional step C4 may be performed after N4. That is, the electronic device 100 can transmit the converted event to the downstream event message queue (N4) and then override the previous event with a new event in the DBMS (C4). At this time, if N4 partially fails (first case), only one of R(n+1) or -R(n) is transmitted downstream and a collision may occur in the first pipeline 320. After recovery, R(n+1) and -R(n) will be transmitted, and the second pipeline 330 may deduplicate previously received records. Additionally, if N4 is performed but C4 fails (second case), only redundant records R(n+1) and -R(n) will be retransmitted back downstream. On the other hand, if N4 and C4 are performed but N5 fails (case 3), only redundant records R(n+1) will be retransmitted downstream, and the second pipe guarantees exactly once processing even if redundant data is transmitted. Since deduplication will be done in line 330, there may not be a problem. Therefore, preferably, the electronic device 100 can perform the additional step C4 after N4.

In one embodiment, the DBMS may include Apache Cassandra, an open source distributed No-SQL DBMS. Cassandra can manage large amounts of data across numerous servers while delivering high performance without a single point of failure (SPOF), supports clusters across multiple data centers, and allows low-latency operation for all clients. You can.

In one embodiment, the DBMS may include Redis, an open source non-relational DBMS with a key-value structure.

In this way, the electronic device 100 is not affected by the problem even when there is an order error in the order data streaming to the message queue or a conflict occurs during data processing in the data pipeline, and the changed order status is normally reported as final statistics. By performing exception handling that can be reflected in the data, the integrity of real-time data processing can be effectively maintained.

3B, a real-time data pipeline architecture 300 and data source metadata 371 and metric metadata 372 that can be input to the architecture 300 are shown. In one embodiment, the electronic device 100 may register a data source through data source metadata 371 and specify which business fields can be extracted from this data source. In one embodiment, the electronic device 100 specifies the required metric through metric metadata 372, and provides information about the operation and conditions for the metric (e.g., type of service for which statistics are collected, statistics Collection start and end dates, etc.) can be set.

Referring to FIG. 3C, exemplary information representing a real-time data pipeline architecture 300 and processes 381 to 385 through which metrics are generated is shown. At 381, when a source order event is new in message queue 350, first pipeline 320 may read the event and transmit it according to the configuration of data source metadata 371. At 382, the required order event is processed, and the required business fields may be extracted and written to the associated business message queue topic according to the configuration of the metrics metadata 372. At 383 and 384, the second pipeline 330 may aggregate the required metrics at a fine-grained level according to the metric metadata 372. At 385, the publisher may obtain the final metrics 340 and push them to the order profile message queue topic.

Meanwhile, validating statistical data generated in real time through a data pipeline is a very important task. Statistical data is updated in real time and users will use the updated statistical data each time the statistical data is updated. However, if a problem occurs in any part of the entire data pipeline, statistical data generated through the data pipeline may have data quality problems, for example, newly distributed code to implement the data pipeline. There may be a bug. If these problems are discovered late, they may cause other problems in the operation of e-commerce services, so to prevent this, it is necessary to periodically perform data quality checks on statistical data. However, data dumping and processing of aggregated data such as metrics (e.g., number of SDP views, number of clicks, total order amount, etc.) takes a lot of time, making it difficult to easily check the latest data, and data such as user order amount can contain 8TB of data, so a method to efficiently perform data quality inspection of large amounts of data is required. Embodiments related to statistical data verification are described below with reference to FIGS. 4 and 5.

4 shows an example architecture representing data flows associated with a baseline generator and a data quality verifier according to an embodiment of the present disclosure. Referring to FIG. 4, an architecture 410 including a data pipeline for generating statistical data described in FIG. 3A and an architecture 420 for verifying data quality of statistical data are shown.

In one embodiment, the electronic device 100 may generate baseline data for verifying statistical data based on order data. The electronic device 100 may generate baseline data from the original data (golden truth data) of the order data, and the original data may be stored in a DBMS dump used by a batch pipeline and a real-time pipeline. It can be extracted from a message queue dump used by In one embodiment, the original data may be sampled based on the user's identifier to reduce computational cost.

In one embodiment, generating baseline data may further include verifying the baseline data based on order data. For example, data may be duplicated in the message queue dump used by the real-time pipeline, or a conflict may occur during the process of extracting the original data from the message queue dump, so the original data cannot be guaranteed to be 100% accurate. The electronic device 100 may also perform data quality verification on baseline data generated based on order data.

In one embodiment, the electronic device 100 may compare the incremental values of each of the baseline data and the statistical data and verify the statistical data based on the comparison. For example, the baseline data and statistical data may include order quantities, and the incremental values of the baseline data and statistical data range from "value(till today)," which is the order quantity until today, to "value(till," which is the order quantity until yesterday. yesterday)", that is, it can be calculated as "value(till today) - value(till yesterday)". In this way, the electronic device 100 tracks the incremental value of data in time units between T and T+1, based on the data value confirmed at time T, and verifies the accuracy of the tracked increment values to determine the time point T. It is possible to determine whether the data value confirmed at T+1 is accurate, and through this, large amounts of data can be efficiently verified without the need to go through verification procedures for all data from time point 0 to time T+1.

Specifically, V(n) represents the calculated metric value at time n, B(n) represents the baseline metric value at time n, and P(n) represents the correctness of V(n). It represents and can have a value of either true or false. That is, if P(n) is true, V(n)=B(n). The time interval of N may be hours. At this time, the electronic device 100 can prove that P(n) is true at all times n in the following order.

1) First, P(0)=True, 0 indicates the time when data verification begins, and this indicates that all history of the calculated metric will be verified from the beginning.

2) Prove that for all k>=n+1, if P(k) is true, then P(k+1) is also true. For a particular k, assuming P(k) is true when n=k, compare B(j) and V(j) for k<j<=k+1, and all values in this interval are In the same case, P(k+1) is also true.

3) If 1) and 2) above are proven to be true, P(n) becomes true at all times n by mathematical induction.

Through the above mathematical induction method, if the electronic device performs a data quality check at time T, the accuracy of the data up to time T has been proven, and in order to perform a data quality check at time T+1, the data quality check between time T and T+1 has been proven. By performing a data quality check only on the incremental values, the accuracy of the data up to time T+1 can be proven, and thus the electronic device 100 can verify the integrity of the data in all time periods without an unnecessary increase in the amount of calculation.

In one embodiment, the incremental value may be calculated at a set period based on statistical data. For example, the cycle for calculating increment values (e.g., 1 hour, 1 day, week) can be adaptively determined based on statistical data (e.g., type of item being ordered, frequency of user ordering, etc.) there is. In the case of statistical data on the user's order quantity, depending on the characteristics of the item, the cycle for calculating the increment value is set short (for example, 1 day) for the order quantity of the item that the user frequently orders, and for the item that the user does not frequently order By setting a long cycle for calculating the incremental value (for example, one week), the amount of calculations required for data verification of statistical data can be efficiently reduced.

5A and 5B are exemplary diagrams showing the operation of a data quality verifier according to an embodiment of the present disclosure.

Referring to Figure 5A, an architecture 500 including a data quality validator 510 is shown. The data quality verifier 510 may include a data quality verifier (DQ Validator) included in the architecture 420 of FIG. 4 . The data quality verifier 510 can measure the accuracy of the data by comparing baseline data generated from the baseline generator and statistical data obtained from the metric dump. In one embodiment, the data quality verifier 510 may be driven by a Spark Job.

Referring to FIG. 5B, a code 520 that can check whether constraints set by the user are satisfied is shown. In the example shown in Figure 5b, the data quality verifier 510 sets the fresh order amount for one day, which is statistical data to be inspected, as a key ("order-fresh-day-amount") through code 520, and sets it as a constraint. Whether the value is within a set range (e.g., 0 to 100000), whether the size differs significantly compared to the previous day's order quantity (e.g., within a range of 0.9 to 1.1 times the previous day's order quantity), and how consistent it is (e.g. , 99% agreement) can be tested. The data quality verifier 510 may transmit a notification message to the administrator terminal if at least one of the constraints is not satisfied as a result of comparing statistical data and baseline data (or their incremental values).

Figure 6 is a flowchart showing the flow of a data verification method for an electronic device according to an embodiment of the present disclosure. The steps shown in FIG. 6 may be performed by the electronic device 100 shown in FIG. 1, and descriptions that overlap with the above may be omitted below.

In step S610, the electronic device may generate statistical data about the order based on the order data. In one embodiment, statistical data may include a profile or metric representing characteristics related to an e-commerce service of an entity (e.g., user, category, promotion) and may be used directly, for example. It may include attributes of an entity (e.g., membership sign-up date), aggregated data for specific event types, predictive metrics, etc. Aggregated data includes behavioral statistics such as logins, number of clicks, access times, single detail page (SDP) views, and transactions such as number of orders, gross merchandise volume (GMV), order value, and total order volume. Statistics may include. Predictive metrics may include, for example, a predictive score related to a user's interest in “iPhone.” An e-commerce business may use these for analysis and decision-making related to the entity's e-commerce services. There is a need to collect and generate statistical data for the same.

In step S620, the electronic device may generate baseline data for verifying statistical data based on the order data. The electronic device may generate baseline data from original data of the order data, and the original data may be extracted from a DBMS dump used by the batch pipeline and a message queue dump used by the real-time pipeline. In one embodiment, the original data may be sampled based on the user's identifier to reduce computational cost.

In step S630, the electronic device may compare the incremental values of each of the baseline data and statistical data. In step S640, the electronic device may verify statistical data based on comparison of incremental values. For example, the baseline data and statistical data may include order quantities, and the incremental values of the baseline data and statistical data range from "value(till today)," which is the order quantity until today, to "value(till," which is the order quantity until yesterday. yesterday)", that is, it can be calculated as "value(till today) - value(till yesterday)". In this way, the electronic device 100 tracks the incremental value of data in time units between T and T+1, based on the data value confirmed at time T, and verifies the accuracy of the tracked increment values to determine the time point T. It is possible to determine whether the data value confirmed at T+1 is accurate, and through this, large amounts of data can be efficiently verified without the need to go through verification procedures for all data from time point 0 to time T+1.

7A to 7C are exemplary diagrams for explaining a design for generating statistical data according to a plurality of embodiments of the present disclosure.

Referring to FIG. 7A, a data pipeline architecture 710 is shown for the electronic device 100 to generate a metric from a data source according to an embodiment of the present disclosure. Metrics can be split into today's metrics and metrics from before today, today's metrics can be calculated directly through a streaming processing engine (e.g., a Spark Streaming application), and metrics before today can be calculated directly from the batch. It can be calculated in batches and stored in a key-value store. The final metric can be generated by combining real-time metrics and batch metrics. For source data whose state may change, such as order data, electronic device 100 may use reactions to process changes in the state of the record. As shown in FIG. 7A, a database 711 may be added to record the most recent status record, and in one embodiment, the database is a Cassandra database management system capable of supporting key-value reading and writing. It may include databases managed by . Taking orders as an example, the data pipeline can store all newly processed source order events in the database 711 and replace old events, if any. For every source order event received, if the order was not newly created, the pipeline can obtain the previous record of that order and calculate the reaction value. For example, if an order is cancelled, the pipeline could send downstream the previous order quantity multiplied by -1.

Referring to FIG. 7B, a data pipeline architecture 720 is shown for the electronic device 100 to generate a metric from a data source according to an embodiment of the present disclosure. As shown in FIG. 7B, the metric calculation module 721 may first store business event details and then generate necessary metrics based on the stored detailed events. The architecture 720 is characterized by aggregating detailed events during the recalculation period in the metric calculation module, which has the advantage of improving metric calculation speed and reducing storage costs, but increases implementation complexity and exceeds the predefined aggregation range. The downside is that it is difficult to add new metrics.

Referring to FIG. 7C, a data pipeline architecture 730 is shown for the electronic device 100 to generate a metric from a data source according to an embodiment of the present disclosure. As shown in Figure 7C, the metric calculation module 731 may aggregate business events into a fine grained level pre-aggregated format and first store the data. The metric calculation module 731 may then send a notification to the real-time publisher 732 and cause the real-time publisher 732 to aggregate the pre-aggregated data into final metrics. For source data whose state may change, such as order data, the electronic device 100 may use a reaction to process the state change of the source event. The architecture 730 stores detailed business events in the database 733 of the data transformation module rather than the metric calculation module, because the generated reaction event can be processed as a general business event.

Meanwhile, the specification and drawings disclose preferred embodiments of the present invention, and although specific terms are used, they are used in a general sense to easily explain the technical content of the present invention and to aid understanding of the present invention. It is not intended to limit the scope of the invention. In addition to the embodiments disclosed herein, it is obvious to those skilled in the art that other modifications based on the technical idea of the present invention can be implemented.

100: electronic device
110: processor
120: memory

Claims (12)

A method for verifying data in an electronic device, comprising:
generating statistical data about orders based on order data;
generating baseline data to verify the statistical data based on the order data;
Comparing incremental values of each of the baseline data and the statistical data; and
Verifying the statistical data based on comparison of the incremental values,
The statistical data is generated through a data pipeline that ensures exactly once processing of the order data,
The data pipeline includes a first pipeline that guarantees at least once processing and a second pipeline that guarantees exactly once processing,
The output of the first pipeline includes duplicate data associated with the order data,
A data verification method for an electronic device, wherein deduplication of the duplicate data is performed in the second pipeline.
According to paragraph 1,
The order data includes an order identifier and order quantity for one or more orders,
The data verification method of an electronic device, wherein the statistical data includes the total amount of the order.
According to paragraph 2,
The step of generating the statistical data is
Receiving an order cancellation event;
Confirming the quantity of orders canceled from the order cancellation event; and
A data verification method for an electronic device, further comprising adding the canceled order quantity multiplied by -1 to the statistical data.
delete delete According to paragraph 1,
The step of generating the statistical data is
When R(n+1) representing the n+1 change of the order data is input to the first pipeline:
retrieving R(n) representing the nth change of the order data from the cache;
Multiplying the value of R(n) by -1;
transmitting -R(n) and R(n+1) to the second pipeline; and
Caching the R(n+1)
Including, a data verification method of an electronic device.
According to clause 6,
The step of generating the statistical data is
A data verification method for an electronic device, further comprising generating the statistical data by aggregating the -R(n) and R(n+1) in the second pipeline.
According to clause 6,
Wherein R(n) includes an order ID and an order version.
According to paragraph 1,
The step of generating the baseline data is
Data verification method of an electronic device, further comprising verifying the baseline data based on the order data.
According to paragraph 1,
A data verification method for an electronic device, wherein the increment value is calculated at a set cycle based on the statistical data.
An electronic device for verifying data, comprising:
a memory storing at least one instruction; and
By executing the at least one command, statistical data for an order is generated based on order data, baseline data for verifying the statistical data is generated based on the order data, and the baseline data and the statistics are generated. A processor that compares the increment values of each data and verifies the statistical data based on the comparison of the increment values,
The statistical data is generated through a data pipeline that ensures exactly once processing of the order data,
The data pipeline includes a first pipeline that guarantees processing at least once and a second pipeline that guarantees processing exactly once,
The output of the first pipeline includes duplicate data associated with the order data,
An electronic device for verifying data, wherein deduplication of the duplicate data is performed in the second pipeline.
A non-transitory computer-readable storage medium, comprising:
A medium configured to store computer readable instructions,
The computer-readable instructions, when executed by a processor, cause the processor to:
generating statistical data about orders based on order data;
generating baseline data to verify the statistical data based on the order data;
comparing incremental values of each of the baseline data and the statistical data; and
Performing a data verification method for an electronic device, comprising verifying the statistical data based on comparison of the incremental values,
The statistical data is generated through a data pipeline that ensures exactly once processing of the order data,
The data pipeline includes a first pipeline that guarantees processing at least once and a second pipeline that guarantees processing exactly once,
The output of the first pipeline includes duplicate data associated with the order data,
A non-transitory computer-readable storage medium, wherein deduplication of the redundant data is performed in the second pipeline.

Images