WO2024035825A1 - Systèmes et procédés de réconciliation pour flux non délimités - Google Patents

Systèmes et procédés de réconciliation pour flux non délimités Download PDF

Info

Publication number
WO2024035825A1
WO2024035825A1 PCT/US2023/029911 US2023029911W WO2024035825A1 WO 2024035825 A1 WO2024035825 A1 WO 2024035825A1 US 2023029911 W US2023029911 W US 2023029911W WO 2024035825 A1 WO2024035825 A1 WO 2024035825A1
Authority
WO
WIPO (PCT)
Prior art keywords
shard
window
reconciliation
events
computing system
Prior art date
Application number
PCT/US2023/029911
Other languages
English (en)
Inventor
Duncan Stewart
Alok Dwivedi
Original Assignee
Discover Financial Services
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/884,857 external-priority patent/US20240053999A1/en
Priority claimed from US17/884,836 external-priority patent/US12045262B2/en
Application filed by Discover Financial Services filed Critical Discover Financial Services
Publication of WO2024035825A1 publication Critical patent/WO2024035825A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G06F9/4887—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues involving deadlines, e.g. rate based, periodic

Definitions

  • a producer computing system may convert an unbounded stream of events into a series of bounded, self-reconcilable streams of events that can be consumed and reconciled by a consumer computing system.
  • the disclosed technology may take the form of a method to be carried out by a computing system that involves (i) determining a start of a window in an unbounded stream of events produced by the computing system, (ii) based on determining the start of the window, injecting, into the unbounded stream of events, a window open sentinel that indicates one or more reconciliation rules for the window, (iii) determining a start of a shard in the unbounded stream of events, wherein the shard is within the window, (iv) based on determining the start of the shard, injecting, into the unbounded stream of events, a shard open sentinel that indicates one or more reconciliation rules for the shard, (v) transmitting a set of events produced within the shard, (vi) executing the one or more reconciliation rules for the shard for a set of events produced within the shard to thereby generate one or more reconciliation attributes for the shard, (vii) determining a
  • the computing system is a producer computing system and the method may also involve receiving, from a consumer computing system, an indication that the shard was unsuccessfully reconciled and based on receiving the indication, retransmitting the set of events produced within the shard. Additionally, or alternatively, the method may also involve receiving, from a consumer computing system, an indication that the window was unsuccessfully reconciled, and based on receiving the indication, retransmitting the set of events produced within the window.
  • the one or more reconciliation rules for the window may include instructions for determining (i) a given number of events produced within the window, (ii) a given number of shards produced within the window, and (iii) a signature for each event produced within the window.
  • the one or more reconciliation rules for the window may include instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the window, (ii) a number of a given type of event produced within the window, or (iii) a signature for each of a given type of event produced within the window.
  • the one or more reconciliation rules for the shard may include instructions for determining (i) a given number of events produced within the shard and (ii) a signature for each event produced within the shard.
  • the one or more reconciliation rules for the shard may include instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the shard, (ii) a number of a given type of event produced within the shard, or (iii) a signature for each of a given type of event produced within the shard.
  • executing the one or more reconciliation rules for the shard for a set of events produced within the shard may involve executing the one or more reconciliation rules for the shard after all events in the set of events are produced within the shard.
  • executing the one or more reconciliation rules for the shard for a set of events produced within the shard may involve executing the one or more reconciliation rules for the shard after each event in the set of events is produced within the shard.
  • executing the one or more reconciliation rules for the window for a set of events produced within the window may involve executing the one or more reconciliation rules for the window after all events in the set of events are produced within the window.
  • executing the one or more reconciliation rules for the shard for a set of events produced within the window may involve executing the one or more reconciliation rules for the window after each event in the set of events is produced within the window.
  • the method may also involve determining the one or more reconciliation rules for the shard and/or the window.
  • one or more reconciliation rules for the window are different from the one or more reconciliation rules for the shard.
  • the disclosed technology may take the form of a method to be carried out by a computing system that involves (i) detecting a window open sentinel for a window within an unbounded stream of events, wherein the window open sentinel indicates one or more reconciliation rules for the window, (ii) detecting a shard open sentinel for a shard within the window, wherein the shard open sentinel indicates one or more reconciliation rules for the shard, (iii) receiving a set of events produced within the shard, (iv) detecting a shard close sentinel for the shard, wherein the shard close sentinel includes an indication of one or more producergenerated reconciliation attributes for the shard, (v) executing the one or more reconciliation rules for the shard for a set of events produced within the shard to thereby generate one or more consumer-generated reconciliation attributes for the shard, (vi) based on detecting the shard close sentinel, reconciling the determined one or
  • the computing system is a consumer computing system
  • the method may also involve determining that the one or more consumer-generated reconciliation attributes for the shard match the one or more producer-generated reconciliation attributes for the shard, and transmitting, to a producer computing system, an indication that the shard was successfully reconciled. Additionally, or alternatively, the method may also involve determining that the one or more consumer-generated reconciliation attributes for the window match the one or more producer-generated reconciliation attributes for the window, and transmitting, to a producer computing system, an indication that the window was successfully reconciled.
  • the computing system is a consumer computing system
  • the method may also involve determining that the one or more consumer-generated reconciliation attributes for the shard does not match the one or more producer-generated reconciliation attributes for the shard, discarding the set of events produced within the shard, and transmitting, to a producer computing system, a request to retransmit the set of events produced within the shard.
  • the method may also involve creating a data object corresponding to the shard, and after each event in the set of events produced within the shard is received, adding the event to the data object for the shard.
  • receiving the set of events produced within the shard may involve receiving at least one event in the set of events produced within the shard before detecting the shard open sentinel, and creating the data object corresponding to the shard may involve creating the data object corresponding to the shard based on receiving the at least one event in the set of events produced within the shard before detecting the shard open sentinel.
  • executing the one or more reconciliation rules for the shard may involve executing the one or more reconciliation rules for the shard after all events in the set of events produced within the shard are received.
  • executing the one or more reconciliation rules for the shard may involve executing the one or more reconciliation rules for the shard after each event in the set of events produced within the shard is received.
  • the window open sentinel includes an indication of a window identifier
  • each event in the set of events received within the window includes an indication of the window identifier
  • the one or more reconciliation rules for the window may include instructions for determining (i) a given number of events produced within the window, (ii) a given number of shards produced within the window, and (iii) a signature for each event produced within the window.
  • the one or more reconciliation rules for the window may include instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the window, (ii) a number of a given type of event produced within the window, or (iii) a signature for each of a given type of event produced within the window
  • the one or more reconciliation rules for the shard may include instructions for determining (i) a given number of events produced within the shard and (ii) a signature for each event produced within the shard.
  • the one or more reconciliation rules for the shard may include instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the shard, (ii) a number of a given type of event produced within the shard, or (iii) a signature for each of a given type of event produced within the shard.
  • the one or more reconciliation rules for the window are different from the one or more reconciliation rules for the shard.
  • a computing system that includes a network interface for communicating over at least one data network, at least one processor, at least one non-transitory computer-readable medium, and program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor to cause the computing system to carry out the functions disclosed herein, including but not limited to the functions of any of the foregoing methods.
  • a non-transitory computer-readable medium provisioned with program instructions that, when executed by at least one processor, cause a computing system to carry out the functions disclosed herein, including but not limited to the functions of any of the foregoing methods.
  • FIG. 1 depicts an example computing platform that employs an event-driven architecture.
  • FIG. 2 depicts a schematic example of an unbounded stream of events within an event- driven architecture.
  • FIG. 3 depicts an example unit of reconciliation within an unbounded stream of events in accordance with the disclosed technology.
  • FIG. 4 depicts a flow diagram of an example process carried out by a producer computing system in accordance with the disclosed technology.
  • FIG. 5 depicts a flow diagram of an example process carried out by a consumer computing system in accordance with the disclosed technology.
  • FIG. 6 is a simplified block diagram that illustrates some structural components that may be included in an example computing system.
  • Disclosed herein is a new technology for converting an unbounded stream of events that may be generated in the context of an event-driven system architecture into a bounded, self- reconcilable stream of events.
  • the disclosed technology may be carried out by any organization that engages in producing events, facilitating the delivery of events, and/or consuming events, examples of which include business organizations that facilitate e-commerce transactions, social media websites, and the management of Internet of Things (loT) devices, among many other possibilities.
  • LoT Internet of Things
  • a typical event-driven architecture is made up of a collection of discrete software components, sometimes referred to as “services,” each of which may carry out one or more functions within the event-driven architecture.
  • These software components may be implemented using various different techniques.
  • the software components may be implemented in part using a microservices architecture where one or more of the software components are deployed in containers and communicate using respective APIs.
  • one or more of the software components may utilize a serverless architecture where the given software component is implemented as a set of cloud-based functions that may only be run when another software component accesses its API.
  • a computing platform implementing an event-driven architecture may integrate aspects of any of these various techniques, among others, in any combination.
  • the physical instantiation of the one or more software components discussed above may also take various forms.
  • the physical hardware e.g., servers, processors, communication interfaces, etc.
  • the functions within the event-driven- architecture might not be organized along the same logical lines as the associated software components.
  • a given software component may be collectively implemented by two or more physically distinct computing systems within an overall computing platform.
  • two or more logically separate software components e.g., within separate containers or virtual machines
  • the software components may take other forms and may be implemented in other ways as well.
  • the events that are produced and consumed in an event-driven architecture may take various forms, one of which may include a change in state in the computing platform that implements the event-driven architecture.
  • an event may include a purchase order, a product availability inquiry, or an item return, among other possibilities.
  • an event may include a notification that a user has posted an update (e.g., a message, a photo, a reaction to another user’s post, etc.).
  • an event may include a temperature sensor reading from a network- connected thermostat or a proximity alert from a network-connect motion detector, among other possibilities.
  • the discrete software components within an event-driven architecture may carry out various different functions. For instance, one or more of the discrete software components may generate events within the event-driven architecture.
  • the one or more software components (and the associated physical computing system(s)) that generate events within the event-driven architecture may generally be referred to as one or more “producers.”
  • a given producer within the event-driven architecture may take various forms. As one example in the context of an e-commerce platform, the given producer may take the form of a point-of-sale device, where the point-of-sale device generates an event for each sale or return.
  • the given producer may take the form of a mobile application, where the mobile application generates an event in response to receiving an inquiry about the availability of an item for sale.
  • the given producer may take the form of a retail website, where the retail website generates an event in response to a user placing a new order for an item.
  • the given producer may take many other forms as well, depending on the application of the event-driven architecture.
  • one or more of the discrete software components discussed above may consume the generated events within the event-driven architecture.
  • consuming an event may generally refer to receiving the event, and perhaps performing some processing associated with the event as well.
  • the one or more discrete software components (and associated physical computing system(s)) that consume events within the event- driven architecture may be referred to as one or more “consumers.”
  • a given consumer within the event-driven architecture may take various forms. As one example in the context of an e- commerce platform, the given consumer may take the form of a warehouse management database, where the warehouse management database may consume a produced sale event and in response, update its inventory and item availability records.
  • the given consumer may take the form of a finance system, where the finance system may consume a produced sale event (e.g., the same sale event as the previous example) and in response, update financial records to reflect the sale.
  • the given consumer may take the form of a customer relations system, where the customer relations system may consume an event corresponding to a question about availability of a particular item for sale and in response, generate a notification for a customer relations expert to respond to the inquiry.
  • the given consumer may take many other forms as well, depending on the application of the event-driven architecture.
  • one or more of the discrete software components discussed above may facilitate (i) the delivery of events from the one or more producers to the one or more consumers and (ii) the delivery of one or more responses from the one or more consumers to the one or more producers.
  • These software components may collectively form a messaging system through which events and responses can be sent.
  • a messaging system may take various forms.
  • the messaging system may take the form of an event broker that receives, stores, and delivers events from a producer to one or more consumers.
  • the messaging system may take the form of an event router that receives events from a producer and routes them to one or more consumers.
  • the messaging system may take the form of a messaging bus that receives produced events from a producer and delivers the produced events to one or more consumers that subscribe to the producer.
  • the messaging system may take other forms as well.
  • the messaging system may employ a variety of technologies to receive the events from a producer and deliver the events to a consumer.
  • the messaging system may employ one or more queues, where the messaging system receives a produced event and places the received event in a queue that can be accessed by a consumer.
  • the messaging system may employ one or more topics, where the messaging system applies additional metadata to the received event that facilitates the delivery of the event to consumers that have subscribed to the events corresponding to the topic.
  • the messaging system may employ one or more logs, where the messaging system stores received events in a log that can be later accessed by a consumer.
  • the messaging system may employ streams, where the messaging system receives a produced event and makes the event available to the consumer in real time.
  • the messaging system may employ other technologies as well.
  • the event-driven architecture includes one or more event producers, illustrated by way of example as producer 110a, producer 110b, and producer 110c.
  • the event-driven architecture also includes a messaging system 120 and one or more event consumers, illustrated by way of example as consumer 130a, consumer 130b, and consumer 130c.
  • the event producers may be configured to produce events associated with the computing platform 100 and may take various forms.
  • event producer 110a may be a point-of-sale system that is configured to produce payment transaction events each time an item is purchased.
  • each event may be a data object that includes information about the payment transaction.
  • each event may be organized according to a schema that defines certain data fields (e.g., a globally unique event identifier, a merchant identity, a payment amount, a transaction date, a transaction time, etc.) and corresponding values for the included data fields.
  • event producer 110b may be a customer service system that is configured to produce an event each time a dispute related to a payment transaction event is initiated (e.g., each time a business or customer disputes a transaction).
  • each event may be a data object that includes information about the disputed transaction.
  • a disputed transaction event may include a schema that may define different data fields (e.g., the disputed payment event’s globally unique event identifier, a merchant identity, a disputed payment amount, etc.) and corresponding values for those data fields.
  • the event producers, and the associated events they produce, may take many other forms as well.
  • Each of the producers shown in FIG. 1 may be configured to produce events and send them to the messaging system 120.
  • messaging system 120 may function to receive, store, and deliver the produced events.
  • the messaging system 120 may receive the produced events via a respective communication link that connects the messaging system 120 with each producer.
  • the communication links may include any one or more of point-to-point links, Personal Area Networks (PANs), Local-Area Networks (LANs), Wide-Area Networks (WANs) such as the Internet or cellular networks, cloud networks, and/or operational technology (OT) networks, among other possibilities.
  • PANs Personal Area Networks
  • LANs Local-Area Networks
  • WANs Wide-Area Networks
  • the communication networks and/or links that make up each respective communication link may be wireless, wired, or some combination thereof, and may carry data according to any of various different communication protocols.
  • Messaging system 120 may receive the produced events from each of the producers in other ways as well.
  • Messaging system 120 may store the produced events in various ways. As one example, messaging system 120 may employ a data store, where messaging system 120 stores each event in the data store as it is received. In some implementations, when messaging system 120 delivers a produced event to one or more consumers that are designated to receive the event, messaging system 120 may purge the event from the data store. In some other implementations, messaging system 120 may employ an event log, where messaging system 120 logs each event in the event log as it is received. In such an implementation, when messaging system 120 delivers a produced event to a consumer 130, messaging system 120 retains the event in the event log. This may be beneficial in some situations, as it may allow the messaging system 120 to resend events if issues arise during delivery of the produced events. Messaging system 120 may store the produced events in other ways as well.
  • Messaging system 120 may deliver the produced events in various ways.
  • messaging system 120 may be configured to push the produced events to appropriate consumers. For instance, one or more consumers may subscribe to a given type of produced event (e.g., payment transaction events). Thereafter, when messaging system 120 receives an event of the given type, messaging system 120 pushes the given event to consumer 130a and consumer 130b.
  • messaging system may publish received events in an event stream, which may be accessed by one or more consumers to retrieve the events. Messaging system 120 may deliver the produced events in other ways as well.
  • consumer 130a may be a pricing system that is configured to consume payment transaction events (e.g., events related to purchases made using credit cards) that are produced by a point-of-sale system producer. Consumer 130a may consume the payment transaction events and calculate a price to levy on each respective transaction (e.g., to be charged to the issuer of the credit card).
  • consumer 130b may be a dispute system that is configured to consume dispute events (e.g., events produced each time a business or customer disputes a transaction). Consumer 130b may consume the events and calculate a fee to be levied on the dispute event (e.g., fees to be refunded).
  • the event consumers may take other forms as well.
  • An unbounded steam of events is a continuous stream of events with no defined beginning or end.
  • an unbounded stream of events means that producers can continuously produce events without temporal boundaries (e.g., the producers produce events as they occur, and the consumers consume events as they are received).
  • an unbounded stream of events can be useful for delivering data within a computing platform in real time, or close to real time.
  • FIG. 2 shows one illustrative example of an unbounded stream of events within an event- driven architecture.
  • a first software component associated with a first computing system e.g., a producer 210
  • event 220, event 230, event 240, and event 250 are depicted, however the unbounded stream of events 200 may additionally include events both preceding event 220 and following event 250.
  • each event produced by producer 210 is immediately made available to messaging system 260, which in turn may make the events immediately available to one or more downstream consumers.
  • producer 210 may generate multiple events before making them available to the messaging system 260.
  • producer 210 may produce event 220, event 230, event 240, and event 250 at any time.
  • the four events shown in FIG. 2 might be produced approximately every 15 seconds over the course of one minute, or they might be produced at random times over the course of one hour or an entire day. Other examples are also possible.
  • an unbounded stream of events in an event-driven architecture may be associated with various drawbacks.
  • One potential drawback associated with unbounded streams of events is the difficulty in ensuring data reliability. For instance, because the events produced as part of an unbounded stream have no defined beginning or end, and can be produced at any time, it is difficult for a consumer to detect whether it is receiving every event that is produced.
  • an event that is produced by a producer may fail to reach the messaging system for any number of reasons (e.g., temporary network failure, third-party tampering). Similarly, an event may fail to be delivered from the messaging system to the consumer. In turn, this may expose the organization responsible for producing or consuming the event to operational or reputational risk.
  • Another potential drawback associated with unbounded streams of events in an event- driven architecture is the difficulty in ensuring data integrity — e.g., ensuring that the data within a received event has not been altered from the time the event was published by the producer.
  • an event may be exposed to tampering by a third-party after a producer generates the event. For instance, if a producer generates an event that is related to a financial transaction, a third-party may alter the values of the financial transaction at some point during intermediate processing or transit. As above, the consumer may have no way to detect such alterations, which may lead to the consumption and further processing of altered events.
  • the drawbacks described above may be particularly applicable when the computing platform that implements an unbounded stream to deliver events is operated by a financial services institution, as problems with data reliability, continuity, and/or integrity may directly affect the processing of payment transactions, the collection of fees, and/or customer satisfaction. For instance, altered financial transactions may constitute fraudulent activity that leads to economic losses for one or more entities involved (e.g., consumers, merchants, credit card issuers, etc.). Further, if an event is generated that pertains to a disputed transaction and that event is never delivered to the appropriate consumer, the disputed transaction may not be resolved promptly, or at all, which in turn may result in poor customer satisfaction and further economic losses.
  • entities involved e.g., consumers, merchants, credit card issuers, etc.
  • reconciliation may refer to a process by which a consumer attempts to confirm both the number and identity of events it receives as well as certain attributes associated with the events.
  • a new technology is discussed herein for implementing a reconciliation framework in which a producer (e.g., a producer software component operating on a producer computing system) may convert an unbounded stream of events into a series of bounded, self-reconcilable streams of events that can be consumed and reconciled by a consumer (e.g., a consumer software component operating on a consumer computing system).
  • a producer e.g., a producer software component operating on a producer computing system
  • a consumer e.g., a consumer software component operating on a consumer computing system
  • a given producer may function to inject boundary markers into an unbounded stream of events.
  • boundary markers are also referred to herein as “sentinels” herein and serve to divide the unbounded stream into one or more units that may be reconciled (e.g., a unit of reconciliation, or UoR) by a consumer.
  • an unbounded stream of events may be divided into one or more windows that contain a given number of events. Each window may be further sub-divided into one or more shards that contain a corresponding number of events from within the window, and so on.
  • each sentinel that is injected into the unbounded stream of events by the producer may take the form of a data object that includes information signaling either the beginning (e.g., an “open” sentinel) or the end (e.g., a “close” sentinel) of a UoR.
  • the producer may include within each sentinel information that enables a consumer to perform reconciliation on the events within each UoR.
  • the information in a given sentinel that enables reconciliation by a consumer may take various forms.
  • the producer may include an identification of one or more reconciliation rules for the UoR that provide the basis for performing reconciliation on the events within the UoR.
  • the reconciliation rules of a given UoR may involve instructions for performing one or more operations associated with the events within the UoR, such as counting the number of events in the UoR or calculating a sum of the values within a given data field across all events in the UoR.
  • Various other types of reconciliation rules are possible and will be discussed in further detail below.
  • the producer may execute the one or more reconciliation rules indicated in the open sentinel and thereby generate one or more producergenerated reconciliation attributes.
  • the producer may include these one or more producergenerated reconciliation attributes in the corresponding close sentinel (e.g., a window close sentinel or a shard close sentinel) for the UoR, which may be injected into the unbounded stream after all events in the UoR are produced.
  • the open sentinel provides an indication of the reconciliation rules that are to be used and the close sentinel includes an indication — in the form of the producer-generated reconciliation attributes — of the result that the consumer should obtain when executing the reconciliation rules.
  • the consumer may execute the reconciliation rules in relation to the events it consumed within the UoR and thereby generate one or more consumer-generated reconciliation attributes.
  • the consumer may then compare the consumer-generated reconciliation attributes with the producer-generated reconciliation attributes. If the attributes are the same, the consumer may determine that reconciliation of the UoR was successful. On the other hand, if one or more of the attributes differs, the consumer may determine that reconciliation was unsuccessful. Based on this determination, the consumer may transmit a request for the producer to re-send the events in the UoR (e.g., a replay request).
  • a request for the producer to re-send the events in the UoR (e.g., a replay request).
  • Other examples are also possible and will discussed in further detail below.
  • the reconciliation framework discussed herein results in an unbounded stream of events that is self-reconcilable based on information that is embedded in the stream itself.
  • the producer may send the unbounded stream of events, including the injected sentinels, to a messaging system, which may deliver the stream to one or more consumers, without requiring any additional communications from the producer to the consumer(s) regarding the reconciliation of the stream (although the consumer may notify the producer of successful and/or unsuccessful reconciliation events, as discussed below).
  • any consumer of the stream may, after successful reconciliation of each UoR, become a producer of the same stream to further downstream consumers.
  • the events and data within the reproduced stream may be afforded the same level of trust as if the stream had come from the original producer.
  • the self-reconcilable nature of the unbounded stream allows for reconciliation to be performed at any and all points through which the stream passes within an event-driven architecture.
  • the messaging system of the event-driven architecture may perform reconciliation of the unbounded stream, even though the messaging system may not undertake any further processing associated with the events. Thereafter, if reconciliation of a UoR within the stream fails at the consumer, the consumer may send a replay request to the messaging system, rather than the producer.
  • Other implementations are also possible.
  • FIG. 3 an illustrative example is depicted in which a portion of an unbounded stream of events may be converted into a series of bounded, self-reconcilable streams of events via the reconciliation framework described herein.
  • a first software component associated with a first computing system e.g., a producer 302
  • produces and sends events that are eventually received by a second software component associated with a second computing system e.g., a consumer 304, as part of an unbounded stream of events 300.
  • a first software component associated with a first computing system e.g., a producer 302
  • a second software component associated with a second computing system e.g., a consumer 304
  • FIG. 3 may be handled by one or more additional intermediate software components between the producer 302 and the consumer 304, such as a messaging system that facilitates delivery of the events, as shown in FIG. 1 and FIG. 2.
  • a messaging system that facilitates delivery of the events, as shown in FIG. 1 and FIG. 2.
  • the messaging system is not depicted in FIG. 3.
  • the example in FIG. 3 shows four events — event 314, event 316, event 324 and event 326 — that form a part of an unbounded stream of events 300, into which the producer 302 has injected several sentinels to divide the stream into self-reconcilable units.
  • the producer has injected a window open sentinel 308, marking the beginning of a window 306, and a window close sentinel 330, marking the close of the window 306.
  • the producer has injected respective open and close sentinels for two shards — shard 310 and shard 320 — that are sub-units of the window 306.
  • each shard contains a sub-portion of the events within the window, e.g., two events each.
  • FIG. 3 represents just one simplified example and that in some other implementations, a given window might contain more or fewer shards that in turn might contain more or fewer events, and, among other possibilities.
  • the producer 302 may initially determine the start of the window 306 within the unbounded stream of events 300. Generally speaking, the producer 302 may determine the start of a new window following the end of a previous window in the unbounded stream. In turn, the producer 302 may determine the end of a given window in an unbounded stream based on various criteria, which will be discussed further below in relation to the window close sentinel 330.
  • the producer 302 may generate a window open sentinel 308 that includes information to facilitate the eventual reconciliation of the window 306 by one or more downstream consumers, such as the consumer 304.
  • the information within the window open sentinel 308 to facilitate reconciliation may take various forms.
  • the window open sentinel 308 may include metadata indicating a unique window identifier that the producer 302 may associate with each event that is produced within the window (e.g., by tagging the event with the window identifier). This may allow the consumer 304 to associate each received event with the correct window at the time of reconciliation.
  • the information within the window open sentinel 308 may additionally include an identification of one or more reconciliation rules for performing reconciliation of the events within the window 306.
  • the reconciliation rules may take the form of metadata the encodes instructions for performing one or more operations on the set of events that are included within the window 306.
  • the reconciliation rules may take various forms and may be defined by the producer 302 based on one or more organization needs.
  • one type of reconciliation rule may include an instruction to count the number of events that are included within the window 306.
  • the producer 302 may execute this rule after all the events within the window 306 are published and may include the resultant value (e.g., as part of one or more producer-generated reconciliation attributes) in the window close sentinel 330, as discussed further below.
  • the consumer 304 may then use this value as the expected result when it executes the same rule for the events that it received in the window 306. If the value the consumer 304 obtains after executing the rule is the same as the expected result, the consumer 304 may determine that reconciliation according to the rule was successful.
  • Another, similar type of reconciliation rule may include an instruction to count the number of shards that are included within the window 306.
  • the reconciliation rules discussed above may enable a consumer 304 to validate that it consumed the correct number of events within a given window, they might not necessarily enable the consumer 304 to validate that the identity of the events it consumed were all the same as the events that were produced by the producer 302. For example, it may be possible for an event within window 306 to be ejected from the unbounded stream of events 300 and replaced by a substitute event (e.g., by a third party). In these situations, reconciliation rules that only include counting the number of events or shards within the window may not identify the replaced event.
  • the reconciliation rules may additionally or alternatively include an instruction to verify an event signature across all events in the window.
  • the events produced by the producer 302 may have a schema that includes a data field for a unique event identifier that may be used to generate an event signature.
  • the reconciliation rule may include instructions for combining the unique event identifiers for each event using one or more cipher techniques. Accordingly, the value that results is based on each individual event identifier.
  • the reconciliation rule may include instructions for generating a cryptographic hash of the resultant value using a hashing algorithm that is included as part of the reconciliation rule in the window open sentinel 308.
  • the cryptographic hash of the resultant value may then be included in the window close sentinel 330 (e.g., as a producer-generated reconciliation attribute).
  • the consumer 304 can recalculate the hashed value using the hashing algorithm in the window open sentinel 308, and then compare the resultant value to the value stored in the window close sentinel by the producer 302.
  • the producer 302 may define still further reconciliation rules that enable reconciliation of the events within the window 306 on a more granular level, such that the consumer 304 can validate the integrity of the information contained within the events. This may be useful in situations where it is desirable to prevent alterations to information within the events, where the alterations do not change the number or identity of the events within the window 306, and thus may not be detected by the reconciliation rules noted above. For instance, a given event might be altered by a third party to change the value of a payment transaction amount (e.g., from $50 to $500).
  • a payment transaction amount e.g., from $50 to $500.
  • One example of this type of reconciliation rule may include instructions for aggregating (e.g., adding) the values from a given data field across all events within the window 306.
  • the producer 302 may produce payment transaction events, and the reconciliation rule may include instructions for aggregating the value from a “Payment Amount” field for each respective event within the window 306.
  • the resultant value may be a total value (e.g., $30,000) that is included in the window close sentinel 330 (e.g., as a producer-generated reconciliation attribute).
  • the consumer 304 can execute the same rule at the time of reconciliation and validate that the aggregated value it obtains is the same as the producer’s, from which it may be inferred that none of the “Payment Amount” values from the events within the window were altered.
  • An aggregation rule may also take other forms (e.g., a mean aggregation rule) and may be used on other data fields as well.
  • Another example of a more granular reconciliation rule may be a grouping rule that includes instructions for counting the occurrences of each unique value within a given data field across all events within the window 306.
  • the reconciliation rule may include instructions for grouping the values within a “Merchant” field within the payment transaction events of the prior example.
  • the producer 302 might represent a computing system operated by an acquirer that handles payment transactions from multiple merchants). Accordingly, the producer 302 executes the reconciliation rule and generate resultant attributes indicating 30 transactions corresponding to the first merchant, 30 transactions corresponding to the second merchant, and 40 transactions corresponding to the third merchant.
  • these attributes may be included in the window close sentinel 330, enabling the consumer 304 to validate during reconciliation that the distribution of events among the consumers is correct.
  • a grouping rule of this type may be applied to other data fields as well, such as a “Dispute Type” field in the context of a dispute resolution platform, among other possibilities.
  • Yet another example of a more granular reconciliation rule may involve instructions for generating a signature that incorporates the values from one or more data fields across all events within the window 306, similar to the event signature rule discussed above.
  • the reconciliation rule may include instruction for (i) concatenating the values of the respective “Payment Amount” and “Merchant” fields into a single value for each event (e.g., in the form “Payment Amount Merchanf ’ (ii) generating a cryptographic hash of the combined value for each event, using a hashing algorithm included in the window open sentinel 308, and (iii) combining the hashed values into a single hashed value using one or more cipher techniques.
  • the resultant value that the producer 302 generates when executing the reconciliation rule is dependent on every value, in both data fields, across all events in the window 306.
  • the consumer’s successful reconciliation of the window 306 may provide a strong indication that each of the values was unchanged.
  • the producer 302 might define a signature rule that designates one or more data fields within the unbounded stream of events whose integrity is most important to validate during reconciliation.
  • Other examples are also possible, including signature rules that involve more or fewer data fields.
  • the one or more reconciliation rules may take other forms as well.
  • the producer 302 may execute the one or more reconciliation rules included in the window open sentinel 308 after all of the events in the window 306 have been produced.
  • the producer might execute the reconciliation rules incrementally, updating a set of producer-generated reconciliation attributes after each and every event is produced.
  • the producer 302 might execute the reconciliation rules for the entire window 306 after the last event (e.g., event 326) is produced.
  • the producer 302 may execute the reconciliation rules at other times as well, including combinations of the above.
  • the producer 302 may include the resulting producer-generated reconciliation attributes in the window close sentinel 330.
  • the producer-generated reconciliation attributes may include a set of values, with one or more of the values corresponding to each of the reconciliation rules that were executed by the producer 302. Each of these values, in turn, may represent an expected value that the consumer 304 may use as a reference to compare its own resultant values when executing the same reconciliation rules on the events that it consumed within the window.
  • the producer 302 might not include the producer-generated reconciliation attributes directly in the window close sentinel 330. Rather, the producer 302 may post the producer-generated reconciliation attributes to a secure registry or the like, and provide a link (e.g., a URI) or similar pointer in the window close sentinel that enables the consumer 304 to access the producer-generated reconciliation attributes. Other implementations are also possible for providing the consumer 304 with access to the producer-generated reconciliation attributes without including them directly in the window close sentinel 330.
  • a link e.g., a URI
  • the producer 302 and the consumer 304 may be configured to execute one or more of the above rules automatically, without reference to the window open sentinel 308.
  • both the producer 302 and the consumer 304 may be configured to count the number of events within the window 306, and the window open sentinel 308 may lack any instruction for doing so.
  • the producer 302 may still include the result of its execution of this automatic reconciliation rule in the window close sentinel 330, as a reference for the consumer 304 to use during reconciliation.
  • the producer 302 and the consumer 304 may be preconfigured to execute any of the reconciliation rules discussed above in this way.
  • the producer 302 and the consumer 304 might exclusively execute reconciliation rules that they are already configured to execute, such that the window open sentinel 308 does not provide an indication of any additional reconciliation rules.
  • the open and close sentinels may nonetheless still provide for the division of the unbounded stream into reconcilable units, defining the boundaries for when the producer 302 and the consumer 304 should execute the pre-encoded reconciliation rules.
  • the window close sentinel 330 may still provide a vehicle for communicating the producer-generated reconciliation attributes to be used by the consumer 304.
  • the window close sentinel 330 may include other information that may also facilitate reconciliation of the window 306.
  • the window close sentinel 330 may include an indication of the unique identifier for the window 306 that the producer 302 included in the window open sentinel 308.
  • the producer 302 may determine when to generate the window close sentinel 330 in various ways.
  • the producer 302 may be configured to inject a window close sentinel after a prescribed number of events has been produced, such as 100 events, after which the producer 302 will inject a new window open sentinel and start the count again.
  • the producer 302 might be configured to produce windows based on a prescribed time period, such as every 30 seconds. Accordingly, the producer 302 may generate and inject a window close sentinel 30 seconds after a window open sentinel was injected, following by another window open sentinel.
  • a producer 302 might produce events relatively slowly such that waiting for an event count to be met before sending a window close sentinel might undesirably delay reconciliation of some produced events.
  • the producer 302 might be configured to inject no window open sentinels into the stream until an event is produced, and then to inject a corresponding window close sentinel no later than 30 seconds (or some other time period) after the window open sentinel is injected.
  • Various other possibilities also exist.
  • the consumer 304 may consume published events within the window 306 as they are received and may generally perform reconciliation based on the information received in the window open sentinel 308 and the window close sentinel 330 as discussed above. More specifically, at the start of each new window, the consumer 304 may generate a state object that corresponds to the window 306. For example, the state object may be associated with the unique identifier for the window 306. As the consumer 304 receives additional events corresponding to the window 306 (e.g., events tagged with the window identifier), the consumer 304 may add the events to the state object.
  • additional events corresponding to the window 306 e.g., events tagged with the window identifier
  • the consumer 304 might execute the reconciliation rules indicated in the window open sentinel 308 as each new event in the window 306 is received, incrementally generating the consumer-generated reconciliation attributes that will be compared to the corresponding producer-generated information in the window close sentinel 330. In this way, the consumer 304 may be able to more quickly perform reconciliation when the window close sentinel 330 is received. Alternatively, the consumer 304 may wait to execute the reconciliation rules until after the window close sentinel 330 is received, and then do so all at once. Various combinations of the above are also possible.
  • the consumer 304 might receive an event that is tagged with a window identifier for which the consumer 304 has not yet received a window open sentinel. In this situation, the consumer 304 might determine that a new window has started and create a new state object with the associated window identifier, in anticipation of receiving the window open sentinel. In this regard, the consumer 304 may be unable to begin executing the reconciliation rules until they are received in the window open sentinel.
  • the consumer 304 might also receive a window close sentinel before all events in the window have been received.
  • the window close sentinel may include a producer-generated reconciliation attribute that indicates 100 events within the window, but the consumer 304 have only received 98 events.
  • the consumer 304 may be configured to include a tolerance, or expiry period, following the receipt of a window close sentinel to account for late arriving events. For instance, if a window is incomplete when the window close sentinel is received, the consumer 304 might wait an additional 10 seconds to allow the missing events to arrive before the consumer 304 determines that reconciliation of the window was unsuccessful.
  • the consumer 304 may notify the producer 302 whether reconciliation of each window (and shard) was successful of unsuccessful. For instance, if the consumer 304 determines that the consumer-generated reconciliation attributes for the window 306 match the producer-generated reconciliation attributes for the window 306, the consumer 304 may notify the producer 302 that the reconciliation of the window 306 was successful. Conversely, if the consumer 304 determines that the consumer-generated and the producer-generated reconciliation attributes for the window 306 do not match, the consumer 304 may notify the producer 302 that reconciliation of the window 306 was unsuccessful, and that the events of the window should be replayed. This type of notification may be referred to herein as a replay event.
  • the producer 302 may generally follow the same framework discussed above with respect to window 306 for generating and injecting shard open and shard close sentinels into the unbounded stream of events.
  • shard 310 includes a shard open sentinel 312, which may include an identifier associated with shard 310.
  • each shard within the window 306 may be assigned a shard identifier.
  • the shard identifier may take the form of an integer value that increases for each new shard within the window, although various other possibilities exist.
  • Each event produced within the shard may be associated with the shard identifier (e.g., by tagging the event with the shard identifier). Accordingly, each event produced within the shard may include an indication of both a window identifier and a shard identifier.
  • the consumer 304 may use the shard identifier to sub-divide events that the consumer 304 stores within the window state object as they are received within the window 306.
  • the shard open sentinel 312 may include an indication of one or more reconciliation rules for the shard 310.
  • the reconciliation rules for the shard 310 may generally take the same form as the reconciliation rules discussed above with respect to the window 306.
  • the producer 302 may execute the shard reconciliation rules and thereby generate one or more producer-generated reconciliation attributes for the shard that will be included in the shard close sentinel 318.
  • the producer 302 may define rules for reconciling, and the consumer 304 may reconcile, the unbounded stream of events at both the window level and the shard level, which may provide various benefits.
  • dividing the window 306 into smaller shards may allow for reconciliation to occur more frequently, which may allow for reconciliation errors to be recognized more quickly. This in turn may result in fewer events that need to be replayed.
  • the producer 302 may define the granularity with which reconciliation is to be performed at each different level.
  • the producer 302 may define shard reconciliation rules that validate the integrity of specific data fields within the unbounded stream of events, whereas the window reconciliation rules may be less granular, directly only to validating the number and identity of shards and events within the window.
  • the producer 302 may define shard reconciliation rules that are less granular, and thus less computationally expensive, whereas the less frequently executed window reconciliation rules may be more granular.
  • Other examples are also possible, and may be defined based on the business, security, and/or computational needs of the producer 302.
  • the shard close sentinel 318 may include the shard identifier associated with shard 310, as well as the one or more producer-generated reconciliation attributes for shard 310. Further, the producer 302 may determine when to inject the shard close sentinel 318 into the unbounded stream of events based on similar criteria as the window close sentinel. For instance, if the producer 302 is configured to produce a new window for every 100 produced events, the producer 302 may additionally be configured to sub-divide each window into shards that include 10 events each. Further, the producer 302 may define a similar relationship between windows and shards in a time-based rather than event-based reconciliation framework. Other examples are also possible.
  • the consumer 304 may perform reconciliation of the shard 310 substantially as discussed above with respect to window 306, by executing the reconciliation rules for the shard 310 and comparing the resulting consumer-generated reconciliation attributes for shard 310 with the producer-generated reconciliation attributes for shard 310 contained in the shard close sentinel 318. If reconciliation is unsuccessful, the consumer 304 may transmit a replay event for the shard 310 to the producer 302. [0102] As mentioned above, it should be noted that although a messaging system is not shown in FIG. 3, there may be some implementations where an intermediate messaging system might perform reconciliation of the stream of events produced by the producer 302 before the events reach the consumer 304. In this regard, the messaging system might perform some or all of the operations discussed above with respect to the consumer 304.
  • FIG. 4 illustrates an example process 400 that may be carried out by a computing system, such as producer computing system the implements the producer 302 described above with respect to FIG. 3.
  • Example process 400 may begin at block 402 with the producer 302 determining a start of a window (e.g., the window 306 described above with respect to FIG. 3) in an unbounded stream of events.
  • the producer 302 may determine the start of the window 306 in various ways. As one possibility, the producer 302 may determine that no window is currently open (e.g., due to a prior window closing) and thus a new window should be opened. As another possibility, the producer 302 may determine that now window is open, but also that no events are being produced. Accordingly, the producer 302 may determine the start of a new window with it determines that a new event will be produced. Other examples are also possible.
  • Block 404 may involve, based on determining the start of the window, injecting a window open sentinel (e.g., the window open sentinel 308) into the unbounded stream of events that indicates one or more reconciliation rules for the window.
  • the window open sentinel 308 may take various forms and may include various information. As one possibility, the window open sentinel 308 may include an identifier that serves to identify the window 306. As another possibility, the window open sentinel 308 may include an indication of one or more reconciliation rules for the window 306, which may take any of the forms discussed above.
  • Block 406 may involve determining a start of a shard (e.g., shard 310) in the unbounded stream of events, where the shard is within the window.
  • the producer may determine the start of the shard 310 in various ways. As one possibility, the producer may determine that no shards are currently open within the window (e.g., either due to a window open sentinel being generated, or due to a prior shard closing).
  • Block 408 may involve, based on determining the start of the shard 310, injecting, into the unbounded stream of events, a shard open sentinel (e.g., shard open sentinel 312) that indicates one or more reconciliation rules for the shard 310.
  • the shard open sentinel 312 may take various forms. As one possibility, the shard open sentinel 312 may include an identifier that serves to identify the shard 310. As another possibility, the shard open sentinel 312 may indicate one or more reconciliation rules for the shard 310, which may take any of the forms discussed above.
  • Block 410 may involve transmitting a set of events produced within the shard 310.
  • the producer 302 may transmit the set of events produced within the shard 310 in various ways. As one possibility, the producer 302 may transmit the set of events produced within the shard to a messaging system that is configured to facilitate delivery of the set of events produced within the shard 310 to a consumer (e.g., consumer 304). The producer 302 may transmit the set of events produced within the shard 310 in other ways as well.
  • Block 412 may involve executing the one or more reconciliation rules for the shard 310 for a set of events produced within the shard (e.g., event 314 and event 316) to thereby generate one or more producer-generated reconciliation attributes for the shard 310.
  • the one or more producer-generated reconciliation attributes for the shard 310 may correspond to the one or more reconciliation rules for the shard 310. For instance, if the one or more reconciliation rules for the shard includes a rule to count the number of events produced within the shard 310, the resulting producer-generated reconciliation attribute may identify the number of the events in the shard 310 (e.g., 10 events). Further, the one or more producer-generated reconciliation attributes for the shard 310 may represent the expected result that the consumer 304 should obtain when executing the one or more reconciliation rules for the shard 310 when attempting reconciliation of the shard.
  • the producer 302 may execute the one or more reconciliation rules for the shard at various times. As one possibility, the producer 302 may execute the one or more reconciliation rules for the shard 310 after the entire set of events are produced within the shard 310. As another possibility, the producer 302 may execute the one or more reconciliation rules for the shard 310 after each event in the set of events is produced within the shard 310. The producer 302 may execute the one or more reconciliation rules for the shard 310 at other times as well.
  • Block 414 may involve determining a close of the shard 310 in the unbounded stream of events.
  • the producer 302 may determine the close of the shard in various ways. As one possibility, the producer 302 may determine that one or more conditions have been met, examples of which may include (i) determining that a target number of events have been produced within the shard 310 (e.g., 10 events) or (ii) determining that a given time interval has expired (e.g., thirty seconds, one minute, one hour, etc.). The producer 302 may determine the close of the shard 310 in other ways as well.
  • Block 416 may involve, based on determining the close of the shard 310, injecting, into the unbounded stream of events, a shard close sentinel (e.g., shard close sentinel 318) that indicates the one or more producer-generated reconciliation attributes for the shard 310 that were generated by the producer 302 executing the one or more reconciliation rules for the shard 310.
  • the shard close sentinel 318 may take various forms and may include data such as (i) the shard identifier, and (ii) the one or more producer-generated reconciliation attributes for the shard.
  • the shard close sentinel 318 may take other forms as well.
  • Block 418 may involve executing the one or more reconciliation rules for the window 306 for a set of events produced within the window (e.g., the events within shard 310 and shard 320) to thereby generate one or more producer-generated reconciliation attributes for the window 306.
  • the one or more producer-generated reconciliation attributes for the window 306 may correspond to the one or more reconciliation rules for the window.
  • the one or more producer-generated reconciliation attributes for the window may represent the expected result that consumer 304 should obtain when executing the one or more reconciliation rules for the window 306 when attempting reconciliation of the window 306.
  • the producer 302 may execute the one or more reconciliation rules for the window 306 at various times. As one possibility, the producer 302 may execute the one or more reconciliation rules for the window 306 after the entire set of events are produced within the window 306. As another possibility, the producer 302 may execute the one or more reconciliation rules for the window 306 after each event in the set of events is produced within the window 306. The producer 302 may execute the one or more reconciliation rules for the window 306 at other times as well.
  • Block 420 may involve determining a close of the window 306 in the unbounded stream of events.
  • the producer 302 may determine the close of the window 306 in various ways. As one possibility, the producer 302 may determine that one or more conditions have been met, examples of which may include (i) determining that a target number of events have been published within the window 306, (ii) determining that a target number of shards have been opened and closed within the window, or (iii) determining that a given time interval has expired. The producer 302 may determine the close of the window 306 in other ways as well.
  • Block 422 may involve, based on determining the close of the window 306, injecting, into the unbounded stream of events, a window close sentinel (e.g., window close sentinel 330) that indicates the one or more producer-generated reconciliation attributes for the window 306.
  • a window close sentinel e.g., window close sentinel 330
  • the window close sentinel 330 may take various forms and may include data such as (i) the window identifier, and (ii) the one or more producer-generated reconciliation attributes for the window.
  • the window close sentinel 330 may take other forms as well.
  • one or more consumers 304 may be performing reconciliation on the window(s) or shard(s). In turn, if reconciliation of a given window or shard fails, the producer 302 may receive a request to resend the set of events produced within a given window or shard. In response to receiving such a request, the producer may “replay” the window or shard identified in the request by repeating one or more of the functions described above.
  • a producer 302 may inject a shard open sentinel and a shard close sentinel in accordance with the description above.
  • the producer 302 may receive an indication that reconciliation of the shard was unsuccessful.
  • the producer 302 may replay the shard by reproducing the set of events for the shard (e.g., by retransmitting the events to a messaging system).
  • one or more downstream consumer can reattempt reconciliation of the shard.
  • the producer 302 may, in addition to replaying the shard, replay the entire window of which the replayed shard is a part. In this respect, the producer 302 may replay the window by reproducing the set of events for the entire window. In turn, a downstream consumer can reattempt reconciliation of the entire window of which the shard is a part.
  • FIG. 5 illustrates an example process 500 that may be carried out by a computing system, such as consumer computing system the implements the consumer 304 described above with respect to FIG. 3.
  • Example process 500 may begin at block 502 with the consumer 304 detecting a window open sentinel (e.g., window open sentinel 308) for a window (e.g., window 306 described above with respect to FIG. 3) within an unbounded stream of events, where the window open sentinel 308 indicates one or more reconciliation rules for the window 306.
  • the consumer 304 may detect the window open sentinel 308 for the window 306 as a result of monitoring the unbounded stream of events for a window open sentinel, among other possibilities.
  • the window open sentinel 308 may indicate a window identifier as well as one or more reconciliation rules for the window, which may take the form of any of the reconciliation rules discussed above.
  • Block 504 may involve detecting a shard open sentinel (e.g., shard open sentinel 312) for a shard (e.g., shard 310) within the window 306, where the shard open sentinel 312 indicates one or more reconciliation rules for the shard 310.
  • the reconciliation rules for the shard 310 may take the form of any of the reconciliation rules discussed above.
  • Block 506 may involve receiving a set of events produced within the shard 310.
  • the consumer 304 may receive the set of events produced within the shard in various ways. As one possibility, the consumer may receive the set of events from a messaging system configured to facilitate the delivery of the set events produced within the shard 310 from the producer 302 to the consumer 304. In addition, the consumer 304 may store each event that is received within the shard in a state object that the consumer 304 created for the window 306.
  • Block 508 may involve detecting a shard close sentinel (e.g., shard close sentinel 318) for the shard 310, where the shard close sentinel 318 includes an indication of one or more producergenerated reconciliation attributes for the shard 310.
  • a shard close sentinel e.g., shard close sentinel 318
  • the consumer 304 may be configured to monitor the unbounded stream of events for the shard close sentinel 318.
  • the consumer 304 may detect the shard close sentinel 318 in other ways as well.
  • Block 510 may involve executing the one or more reconciliation rules for the shard 310 for a set of events produced within the shard 310 to thereby generate one or more consumergenerated reconciliation attributes for the shard 310.
  • the one or more consumergenerated reconciliation attributes for the shard 310 may correspond to the one or more reconciliation rules for the shard 310. For instance, if the one or more reconciliation rules for the shard includes a rule to count the number of events produced within the shard 310, the corresponding consumer-generated reconciliation attribute may identify the number of the events in the shard 310.
  • the consumer 304 may store each of the one or more consumer-generated reconciliation attributes for each event produced within the shard 310 in the data object for the shard 310.
  • Block 512 may involve, based on detecting the shard close sentinel 318, reconciling the determined one or more consumer-generated reconciliation attributes for the shard 310 against the one or more producer-generated reconciliation attributes for the shard 310.
  • the consumer 304 may compare the determined one or more consumer-generated reconciliation attributes for the shard 310 with the one or more producer-generated reconciliation attributes for the shard 310. In this respect, if the determined one or more consumer-generated reconciliation attributes for the shard 310 are different than the one or more producer-generated reconciliation attributes for the shard 310, the consumer 304 may determine that reconciliation of the shard was unsuccessful.
  • the consumer 304 may determine that the reconciliation of the shard 310 was successful.
  • Block 514 may involve detecting a window close sentinel (e.g., the window close sentinel 330), wherein the window close sentinel 330 includes an indication of one or more producergenerated reconciliation attributes for the window 306.
  • the consumer 304 may be configured to monitor the unbounded stream of events for the window close sentinel 330.
  • the consumer 304 may detect the window close sentinel 330 in other ways as well.
  • Block 516 may involve executing the one or more reconciliation rules for the window 306 for a set of events produced within the window 306 to thereby generate one or more consumergenerated reconciliation attributes for the window 306.
  • the one or more consumergenerated reconciliation attributes for the window 306 may correspond to the one or more reconciliation rules for the window 306. For instance, if the one or more reconciliation rules for the window includes a rule to count the number of shards within the window 306, there may be a corresponding consumer-generated reconciliation attribute that identifies the number of the shards in the window 306.
  • the consumer 304 may store the one or more consumer-generated reconciliation attributes for each event produced within the window in the state object for the window 306.
  • Block 518 may involve, based on detecting the window close sentinel 330, reconciling the determined one or more consumer-generated reconciliation attributes for the window 306 against the one or more producer-generated reconciliation attributes for the window 306.
  • the consumer 304 may compare the determined one or more consumer-generated reconciliation attributes for the window 306 with the one or more producer-generated reconciliation attributes for the window 306 to determine whether reconciliation of the window 306 is successful, as discussed above.
  • each block may represent a module, a segment, or a portion of program code, which includes one or more instructions executable by a processor for implementing specific logical functions or steps in the process.
  • the program code may be stored on any type of computer readable medium, for example, such as a storage device including a disk or hard drive.
  • the computer readable medium may include non- transitory computer readable medium, for example, such as computer-readable media that stores data for short periods of time like register memory, processor cache and Random Access Memory (RAM).
  • the computer readable medium may also include non-transitory media, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example.
  • the computer readable media may also be any other volatile or non-volatile storage systems.
  • the computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.
  • each block shown in FIG. 4 and FIG. 5 may represent circuitry that is wired to perform the specific logical functions in the process.
  • FIG. 6 a simplified block diagram is provided to illustrate some structural components that may be included in an example computing system 600.
  • computing system 600 may implement one or more of the software components shown in FIG. 2 and FIG. 3 (e.g., producer 302 or consumer 304) and may be configured to carry out any of the various functions disclosed herein — including but not limited to the functions included in the example data flows described with reference to FIGs. 4-5.
  • computing system 600 may generally comprise any one or more computer systems (e.g., one or more servers) that collectively include at least a processor 602, data storage 604, a communication interface 606, and a user interface 608, all of which may be communicatively linked by a communication link 610 that may take the form of a system bus, a communication network such as a public, private, or hybrid cloud, or some other connection mechanism. Each of these components may take various forms.
  • Processor 602 may comprise one or more processing components, such as general-purpose processors (e.g., a single- or a multi-core CPU), special-purpose processors (e.g., a GPU, application-specific integrated circuit, or digital-signal processor), programmable logic devices (e.g., a field programmable gate array), controllers (e.g., microcontrollers), and/or any other processor components now known or later developed.
  • processor 602 could comprise processing components that are distributed across a plurality of physical computing devices connected via a network, such as a computing cluster of a public, private, or hybrid cloud.
  • data storage 604 may comprise one or more non-transitory computer-readable storage mediums that are collectively configured to store (i) program instructions that are executable by processor 602 such that the computing system 600 is configured to perform certain functions, and (ii) data that may be received, derived, or otherwise stored, for example, in one or more databases, file systems, repositories, or the like, by the computing system 600.
  • the one or more non-transitory computer-readable storage mediums of data storage 604 may take various forms, examples of which may include volatile storage mediums such as random-access memory, registers, cache, etc.
  • data storage 604 may comprise computer-readable storage mediums that are distributed across a plurality of physical computing devices connected via a network, such as a storage cluster of a public, private, or hybrid cloud that operates according to technologies such as AWS for Elastic Compute Cloud, Simple Storage Service, etc.
  • Communication interface 606 may take the form of any one or more interfaces that facilitate wireless and/or wired communication between computing system 600 and other systems or devices.
  • each such interface may take any of various forms, examples of which may include an Ethernet interface, a serial bus interface (e.g., Firewire, USB 3.0, etc.), a chipset and antenna adapted to facilitate wireless communication, and/or any other interface that provides for any of various types of wireless communication (e.g., Wi-Fi communication, cellular communication, short-range wireless protocols, etc.) and/or wired communication.
  • Ethernet interface e.g., Ethernet interface, a serial bus interface (e.g., Firewire, USB 3.0, etc.), a chipset and antenna adapted to facilitate wireless communication, and/or any other interface that provides for any of various types of wireless communication (e.g., Wi-Fi communication, cellular communication, short-range wireless protocols, etc.) and/or wired communication.
  • a serial bus interface e.g., Firewire, USB 3.0, etc.
  • the computing system 600 may additionally include or have an interface for connecting to one or more user-interface components that facilitate user interaction with the computing system 600, such as a keyboard, a mouse, a trackpad, a display screen, a touch-sensitive interface, a stylus, a virtual-reality headset, and/or one or more speaker components, among other possibilities.
  • user-interface components such as a keyboard, a mouse, a trackpad, a display screen, a touch-sensitive interface, a stylus, a virtual-reality headset, and/or one or more speaker components, among other possibilities.
  • the computing system 600 is one example of a computing system that may be used with the embodiments described herein. Numerous other arrangements are possible and contemplated herein. For instance, in other embodiments, the computing system 600 may include additional components not pictured and/or more or fewer of the pictured components.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

L'invention concerne un environnement de réconciliation pour un flux non délimité d'événements dans une architecture commandée par événement, comprenant un système informatique producteur qui convertit le flux non délimité en une série d'unités d'événements auto-concilables délimitées pour une réconciliation par un système informatique de consommateur. Le producteur injecte, dans le flux non délimité, des marqueurs de limite « sentinelle » qui servent à diviser le flux non délimité en unités discrètes d'événements. Chaque sentinelle (i) signale un début ou une fin d'une unité et (ii) identifie une règle de réconciliation qui permet au consommateur d'effectuer une réconciliation sur les événements à l'intérieur de l'unité. Le consommateur exécute la ou les règles de réconciliation pour une unité donnée et compare un ou plusieurs attributs de réconciliation générés par le consommateur à un ou plusieurs attributs de réconciliation générés par un producteur pour les événements dans l'unité donnée afin de déterminer si une réconciliation pour les événements dans l'unité donnée a réussi ou non. Si une réconciliation n'a pas réussi, le consommateur peut demander au producteur d'envoyer à nouveau les événements dans l'unité donnée.
PCT/US2023/029911 2022-08-10 2023-08-09 Systèmes et procédés de réconciliation pour flux non délimités WO2024035825A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US17/884,836 2022-08-10
US17/884,857 2022-08-10
US17/884,857 US20240053999A1 (en) 2022-08-10 2022-08-10 Reconciliation systems and methods for unbounded streams
US17/884,836 US12045262B2 (en) 2022-08-10 2022-08-10 Reconciliation systems and methods for unbounded streams

Publications (1)

Publication Number Publication Date
WO2024035825A1 true WO2024035825A1 (fr) 2024-02-15

Family

ID=89852363

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2023/029911 WO2024035825A1 (fr) 2022-08-10 2023-08-09 Systèmes et procédés de réconciliation pour flux non délimités

Country Status (1)

Country Link
WO (1) WO2024035825A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9619545B2 (en) * 2013-06-28 2017-04-11 Oracle International Corporation Naïve, client-side sharding with online addition of shards
US10423387B2 (en) * 2016-08-23 2019-09-24 Interana, Inc. Methods for highly efficient data sharding
US20200177373A1 (en) * 2018-11-14 2020-06-04 Royal Bank Of Canada System and method for storing contract data structures on permissioned distributed ledgers
US11087017B1 (en) * 2021-03-09 2021-08-10 Technology Innovation Institute Systems, methods, and computer-readable media for utilizing anonymous sharding techniques to protect distributed data
US11394778B2 (en) * 2005-12-29 2022-07-19 Amazon Technologies, Inc. Distributed storage system with web services client interface

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11394778B2 (en) * 2005-12-29 2022-07-19 Amazon Technologies, Inc. Distributed storage system with web services client interface
US9619545B2 (en) * 2013-06-28 2017-04-11 Oracle International Corporation Naïve, client-side sharding with online addition of shards
US10423387B2 (en) * 2016-08-23 2019-09-24 Interana, Inc. Methods for highly efficient data sharding
US20200177373A1 (en) * 2018-11-14 2020-06-04 Royal Bank Of Canada System and method for storing contract data structures on permissioned distributed ledgers
US11087017B1 (en) * 2021-03-09 2021-08-10 Technology Innovation Institute Systems, methods, and computer-readable media for utilizing anonymous sharding techniques to protect distributed data

Similar Documents

Publication Publication Date Title
CN113168637B (zh) 用于交易验证期间的次级欺诈检测的方法和系统
US20240045989A1 (en) Systems and methods for secure data aggregation and computation
CN110009337B (zh) 一种基于区块链的数据处理方法和装置
CN110232565B (zh) 资源清算方法、装置、计算机设备和存储介质
US11062294B2 (en) Cognitive blockchain for customized interchange determination
US11257134B2 (en) Supplier invoice reconciliation and payment using event driven platform
CN109598505B (zh) 一种基于区块链的质量数据处理方法和装置
US20230098747A1 (en) Systems and methods for payment transactions, alerts, dispute settlement, and settlement payments, using multiple blockchains
US20080162202A1 (en) Detecting inappropriate activity by analysis of user interactions
US12033148B2 (en) Systems and methods for providing real-time warnings to merchants for data breaches
US12062051B2 (en) Systems and methods for using machine learning to predict events associated with transactions
WO2020119293A1 (fr) Procédé et appareil de poussée de contenu et dispositif électronique
CN111815420B (zh) 一种基于可信资产数据的匹配方法、装置及设备
US20220269956A1 (en) Systems for the prediction of recurring data entries, mitigation of recurring data entries and methods thereof
US20240311839A1 (en) Fraud detection and prevention system
CN109886676A (zh) 用于区块链网络的支付方法、计算设备、存储介质
US20200204553A1 (en) Method, apparatus and computer program product for exchanging messages across a network
US12045262B2 (en) Reconciliation systems and methods for unbounded streams
US20240053999A1 (en) Reconciliation systems and methods for unbounded streams
KR20210068039A (ko) 거래 시스템을 구현하는 네트워크 노드들의 서브세트 내의 컨텍스트 기반 필터링
CN116739596A (zh) 基于区块链的交易监管方法、装置、设备、介质和产品
WO2024035825A1 (fr) Systèmes et procédés de réconciliation pour flux non délimités
CN115423521B (zh) 大批量并发付款优化系统及其方法
US20220051232A1 (en) Payment information correlation system and method
US20240154800A1 (en) Token recovery

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23853339

Country of ref document: EP

Kind code of ref document: A1