US12067042B2 - System and method of enabling and managing proactive collaboration - Google Patents
System and method of enabling and managing proactive collaboration Download PDFInfo
- Publication number
- US12067042B2 US12067042B2 US17/715,895 US202217715895A US12067042B2 US 12067042 B2 US12067042 B2 US 12067042B2 US 202217715895 A US202217715895 A US 202217715895A US 12067042 B2 US12067042 B2 US 12067042B2
- Authority
- US
- United States
- Prior art keywords
- information
- operator
- class
- participating
- recommendation engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/041—Abduction
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/40—Business processes related to social networking or social networking services
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/214—Monitoring or handling of messages using selective forwarding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/1396—Protocols specially adapted for monitoring users' activity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/535—Tracking the activity of the user
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/096—Transfer learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/21—Monitoring or handling of messages
- H04L51/222—Monitoring or handling of messages using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/52—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail for supporting social networking services
Definitions
- the invention generally relates to enabling and managing collaboration between a plurality of operators.
- the operators may belong to the same organization or to different organizations and share a common interest. For instance, in some embodiments, these operators or organizations provide a variety of public services in a “Smart City” environment.
- managing the collaboration involves using artificial intelligence to selectively provide extracts from messages published by one operator to other operators.
- a first aspect pertains to a collaboration management system for managing collaboration between a plurality of participating operators.
- the system comprises
- the data distributor is configured to provide the pieces of information via a remote data network.
- the pieces of information comprise at least one of keywords and named entities.
- At least a subset of the electronic messages comprises at least one information in the form of textual information, image information, and/or spoken information.
- At least a subset of the electronic messages are messages that are published on one or more social-media platforms.
- At least a subset of the communicating agents are participating operators.
- at least a subset of the communicating agents are public organizations, wherein at least a subset of the participating operators are associated with one of the public organizations.
- an electronic device is assigned to each participating operator, and the data distributor is configured to provide the pieces of information to the electronic devices.
- the recommendation engine may be operatively coupled with the electronic devices and configured
- At least a subset of the electronic messages is sent from a mobile communication device together with geolocation information, and the message parser is configured to link the geolocation information with the extracted pieces of information.
- the message parser is configured to parse the received electronic messages to create a plurality of feature snippets based on the extracted pieces of information.
- the message parser may be configured to create at least a subset of the plurality of feature snippets based on at least two parsed electronic messages.
- the recommendation engine for deciding whether an extracted piece of information is considered class-relevant information the recommendation engine is configured to use either a rule-based approach, a machine-learning algorithm, a hybrid approach that includes the rule-based approach and a machine-learning algorithm, or a combination thereof.
- the recommendation engine is an unsupervised recommendation engine configured to use pre-trained models for deciding for each extracted piece of information whether it is class-relevant information.
- the recommendation engine may be configured
- an electronic device is assigned to each participating operator, the data distributor is configured to provide the pieces of information to the electronic devices, and the recommendation engine is operatively coupled with the electronic devices and configured to monitor the electronic devices to detect a behavior of the operators, and to train the machine-learning algorithm based on the detected behavior.
- the detected behavior may comprise information on whether a piece of information provided to the operator is considered relevant by the operator.
- the recommendation engine may be configured to perform clickstream analysis for detecting the behavior of the operators.
- the system further comprises a classification module operatively coupled with the message parser, wherein the classification module is configured to identify an information class of an extracted piece of information using trained artificial-intelligence agents (AI agents).
- AI agents may be trained using a re-trainable natural language processing (NLP) classifier based on a language model.
- NLP re-trainable natural language processing
- the language model may be a Bidirectional Encoder Representations from Transformers (BERT) language model.
- the recommendation engine is configured to derive a decision threshold as a basis for deciding whether or not an extracted piece of information is considered class-relevant information.
- the recommendation engine may be configured to derive the decision threshold using a pre-trained BERT language model, a logit function, a Softmax function and one or more datasets used for training the pre-trained model.
- a second aspect pertains to a method for managing collaboration between a plurality of participating operators.
- the method may be performed by a system according to the first aspect of the invention.
- the method comprises:
- a second aspect pertains to a computer program product including a non-transitory computer-readable medium having computer code thereon for managing collaboration between a plurality of participating operators.
- the computer program product may run on a system according to the first aspect to perform a method according to the second aspect.
- the computer code comprises:
- FIG. 1 shows a collaboration management system for managing collaboration between participating operators
- FIG. 2 shows steps of a method for managing collaboration between participating operators
- FIG. 3 shows a social graph illustrating a proactive approach to collaboration
- FIG. 4 illustrates the step of intent classification
- FIG. 5 illustrates the step of creating feature snippets
- FIGS. 6 a - c illustrate the step of deciding which information is relevant for which operator
- FIG. 7 shows a training module for AI agents
- FIG. 8 illustrates using the pool of agents for domain-specific tag generation
- FIG. 9 illustrates improving the agents with feedback from users.
- FIG. 1 shows a collaboration management system 10 for managing collaboration between a plurality of participating operators.
- the system 10 comprises an operator database 11 having stored information about the participating operators, the information of each operator comprising at least one operator class assigned to the respective operator.
- the system 10 also comprises a data receiver 12 that is configured to receive a data stream comprising a plurality of electronic messages from communicating organizations, each electronic message comprising at least one textual information.
- the communicating organizations may include the plurality of operators as well as operators relaying messages through commercial data feed providers or social media.
- the system 10 also comprises a message parser 13 operatively coupled with the data receiver 12 .
- the message parser 13 is configured to parse the received electronic messages to extract information of the parsed electronic messages and to classify the electronic messages according to one or more ontologies from a set of pre-defined ontologies.
- the system 10 also comprises a recommendation engine 14 operatively coupled with the message parser 13 and the database 11 .
- the recommendation engine 14 is configured to decide for each extracted information whether it is considered class-relevant information for one or more of the operator classes.
- At least a subset of the received electronic messages comprises information in the form of a text, and the message parser 13 is configured to extract textual information from the text.
- one or more electronic messages include information provided by audio including speech and/or by images or video, and the message parser 13 is configured to extract spoken information from the audio, e.g. by means of speech-to-text, or from the imaged information from the images or video, e.g. by image summarization.
- the system 10 also comprises a data distributor 15 operatively coupled with the recommendation engine 14 .
- the data distributor 15 is configured to provide, based on the assigned one or more operator classes, to each operator of at least a subset of the plurality of participating operators textual information that is considered class-relevant information for the respective operator.
- FIG. 2 shows a flow chart illustrating an exemplary embodiment of a computer-implemented method for managing collaboration between a plurality of participating operators. For instance, the method can be performed using the system of FIG. 1 .
- a data stream is continuously received 110 , the data stream comprising a plurality of electronic messages from a plurality of communicating agents.
- these agents may be public organizations.
- some or all of the participating operators may be communicating agents and/or associated with one of the public organizations.
- At least a subset of the electronic messages may be coming from social-media platforms.
- Each electronic message comprises at least one piece of information.
- the electronic messages may comprise text messages, and the at least one piece of information is provided in the form of textual information.
- some electronic messages may comprise the information in the form of images or audio messages, in particular speech, where the information content is then converted into textual features.
- a message parser is used to parse 120 the received electronic messages and to extract key bits of information from each message based on this parsing. For instance, these key bits may comprise keywords or named entities.
- the electronic messages are classified 130 according to a pre-defined ontology, or more precisely according to one or more ontologies from a set of pre-defined ontologies.
- Information about the participating operators that is stored in a database is accessed 140 , the stored information of each operator comprising at least one operator class assigned to the respective operator.
- the operator class of an operator might change. For instance, the operator may configure the class himself depending on a present case or assignment.
- an appropriate class may be detected automatically and dynamically based on the operator's behavior, for instance by analyzing the operator's clickstream.
- clickstream analysis is the tracking and analysis of visits to websites and comprises collecting, analyzing and reporting aggregate data about which pages a person opens and in what order.
- clickstream analysis may comprise collecting, analyzing and reporting aggregate data about what presented information an operator is interested in, e.g. in which order the operator clicks on said information.
- An algorithm decides 150 for each message and for one or more of the operator classes, based at least on the extracted textual information or the classification, whether a message should be considered class-relevant information for the operator class, i.e. whether it is relevant enough to be presented to the operators of that operator class. For instance, this decision may be based on
- the underlying intent of the extracted message may be identified using state-of-the-art natural language processing models such as decision trees, support vector machines (SVMs), neural networks or transformers (e.g. as illustrated in FIG. 4 ). It may be used in conjunction with extracted entities from the message such as “name of the person” (e.g. as illustrated in FIG. 5 ) to infer relevance of a given message for an operator class.
- state-of-the-art natural language processing models such as decision trees, support vector machines (SVMs), neural networks or transformers (e.g. as illustrated in FIG. 4 ).
- SVMs support vector machines
- neural networks or transformers e.g. as illustrated in FIG. 4
- the method continues from the top by continuously receiving 110 further electronic messages.
- the operators may also be provided with a suggestion 170 to connect with certain other participating operators or organizations.
- This suggestion fosters collaborations and can be based on their similar interests that can be inferred from their past behaviour, e.g. by using clickstream data or feedback on provided recommendations (see step 180 below).
- feedback may be received 180 from the operators, whether or not the provided information or suggestion was also considered relevant or useful by the respective operator.
- This feedback may be provided actively by the operators after receiving an information, e.g. by selecting one of the provided options “was relevant/useful” and “was not relevant/useful”.
- the operator's behaviour regarding the information may be monitored as a passive feedback, e.g. whether the information is read or discarded.
- This feedback may then be used in the step of deciding 150 for further messages whether they are relevant or not for a certain operator class.
- the feedback may be used to further train the artificial intelligence involved in the decision making. Clickstream analysis may be used to indirectly capture user feedback.
- FIG. 3 shows a social graph illustrating a proactive approach to collaboration according to embodiments of the present invention.
- the social graph is a model or representation of a social network involving a number of participating organizations, i.e. a graph that represents social relations between these organizations.
- Data linking comprises generating linked data, i.e. structured data which is interlinked with other data so it becomes more useful through semantic queries. Interactions and data are used to mine and re-surface effective collaborations and to connect people that share similar interests across and within organizations.
- Each communicating organization can be regarded as a communication node of a social graph.
- the communicating organizations comprise public organizations such as fire departments, hospitals, police departments, emergency medical services, utilities, social services and weather departments.
- the communication also relies on social networks, public records and location intelligence as further communication nodes.
- communications from police departments may include text streams related to crimes
- communications from weather departments may include text streams related to hazardous weather or similar natural disasters.
- a collaboration management system listens to the respective data streams (indicated by circles labelled 1 to 10), organizes and catalogues information and extracts meta data from the data streams. Related data streams across different organizations may be discovered, and the relations may be used as a proxy for overlap and for potential collaborations.
- FIG. 4 illustrates the step of intent classification as a part of a method for managing collaboration between a plurality of participating operators.
- incoming text streams from multiple sources are shown that are automatically tagged.
- all texts are related to a hurricane called “Sandy”.
- pre-defined ontologies are shown, that are assigned to the adjacent text streams of the right column. Free-form textual feeds from multiple sources are mapped according to the pre-defined ontologies. This may include visual screening of the textual feeds to enable intent-specific workflows.
- the assigned ontologies comprise the following intents: “Caution or Advice”, “Information Source”, “People: Missing/Found”, “Casualties and Damage”, and “Donations/Support”.
- a notification is raised wherever a pre-defined intent is detected, in the shown example the intent “Casualties and Damage”.
- FIG. 5 illustrates the step of creating feature snippets as a part of a method for managing collaboration between a plurality of participating operators.
- four incoming text streams of the text streams of FIG. 4 are shown.
- Class-relevant information is extracted from the free-form textual feeds to create feature snippets.
- a condensed summary of the events is created to enhance search results based on keywords and to enable intent-specific workflows.
- Named entities are extracted from a given textual information. Which entities are extracted from the free-form textual feeds may depend on the assigned ontology. For instance, if the intent is “Casualties and Damage”, “place” and “date” entities may always be included to the extracted feature snippet.
- FIGS. 6 a - c illustrate an exemplary approach for deciding which information might be relevant for which operator, i.e. to whom the snippets will be provided.
- a human-in-loop recommendation engine is used to discover similar events across multiple data sources. Similar events are mapped together so that a search criterion can be customized to include spatiotemporal features, alongside raw-content information.
- the pre-trained models can be used to enable an unsupervised recommendation engine.
- User feedback can be included to facilitate transition to a supervised approach (“transfer learning”).
- the decision-making may include similarity scoring to create a function that takes a pair of messages and produces a numerical score (“similarity score”) quantifying their relatedness. For such a pair of messages this score is high when the content of the pair covers the same topic.
- similarity score a numerical score quantifying their relatedness. For such a pair of messages this score is high when the content of the pair covers the same topic.
- Techniques for improving the quality of similarity scores include density adjustment, vector embedding, and co-occurrence and contingency tables.
- Step 1 comprises unsupervised similarity scoring.
- a user-defined, customized search criterion is used to determine similarities in a number of feature designs of a pair of messages.
- six feature designs are used, wherein four are raw-content-related, including underlying intent, extracted named entities, semantic representation and part-of-speech (POS) tags, and two are space- or time-related (geographical location and time scale).
- POS part-of-speech
- Each of these feature designs has a different contribution in generating final recommendations as indicated by the black bars. The operator may use these bars to adjust the specifics of the matching process.
- the underlying intent e.g. has a contribution of 94%.
- Step 2 comprises review recommendations based on user feedback.
- the user to whom a message has been provided may choose to provide feedback on whether the provided, i.e. recommended, message was useful or irrelevant.
- Step 3 comprises supervised similarity scoring, in which the AI models that are used for decision making are trained based on the raw input data and on the user feedback.
- FIG. 6 b illustrates the use of a search corpus 61 for recommending by the recommendation engine 14 similar content or proactively discover meaningful collaboration partners.
- the search corpus may include all previous messages, interactions and collaborations that are stored in a database. Together with input description 62 and a user-defined search criterion 63 , the search corpus 61 is provided to the recommendation engine 14 to allow similarity scoring.
- FIG. 6 c shows an example of a clickstream of an operator.
- User interaction of the operators may be captured for performing interest analysis from clickstream.
- Clickstream data is analysed to understand user preferences.
- the clickstream data may be complemented with pre-trained contextual models to understand the underlying dependencies and correlations.
- the clickstream analysis shows that the operator at first clicked on application 64 , showing the current number of high priority incidents (P0). The operator then transitioned to application 65 , showing a road closure suggestion, and to application 66 , showing a thematic map. Based on this captured user interaction, including both the watched applications and the transitions between them, further information may be suggested to the operator.
- the information is provided in the form of suggested media 67 that is related to the applications 64 , 65 , 66 previously clicked by the operator.
- the clickstream analysis also allows determining whether the operator is interested in viewing the suggested media 67 to enhance suggestions in the future.
- FIG. 7 shows a training module that can be used to train specific agents on any classification dataset with similar wording and statistics, e.g. intent and incident type identification.
- a pool of domain-specific AI agents based e.g. on natural language processing (NLP), can be used to allow creating re-trainable classifiers for intent classification.
- NLP natural language processing
- the pool of domain-specific AI agents may comprise one or more agents at each organization, e.g., using the example of FIG. 3 , a fire departments agent at the fire departments, a hospitals agent at the hospitals, a weather departments agent at the weather departments, a public records agent at the public records etc. These agents are usually trained on domain-specific datasets and used to implement the below mentioned use cases.
- the agents are trained on publicly available datasets that closely match the distribution of a domain-specific dataset, which might not be available.
- Several agents trained on publicly available data for a given domain are used together instead of an agent trained on a domain-specific dataset.
- a configured out-of-class threshold for each such agent ensures said agents trained on publicly available data are used in below use cases only, when the messages to be processed fall into their domain.
- the shown module comprises a common natural language processing (NLP) work package which is trained with messages from all available public and domain-specific datasets to produce trained agents for these datasets. Furthermore, all public datasets may be used to calculate and configure activity thresholds to partition agents into non-overlapping domains.
- the available public datasets comprise incidents, damage and alerts reports from a public emergency management agency such as the FEMA, relevant social media feeds and police reports.
- the pool of AI agents comprises one agent for each of the available public datasets.
- FIGS. 8 and 9 illustrate using the pool of agents for domain-specific tag (meta tag) generation and improving the agents with feedback from users.
- a domain of two messages of the incoming data stream is identified using the pool of agents, which here consists of six agents.
- Four domain-specific tags are created for each message.
- the generated tags may be used in conjunction with text-similarity agents to provide targeted recommendations to the operators.
- a user has given feedback regarding the domain-specific tags by approving two of the four tags as accurate. This feedback is added to existing databases, e.g. the module of FIG. 7 , where this feedback is used to trigger re-training of the AI agents.
- the disclosed apparatus and methods may be implemented as a computer program product for use with a computer system.
- Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk).
- the series of computer instructions can embody all or part of the functionality previously described herein with respect to the system.
- Such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems.
- such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies.
- such a computer program product may be distributed as a tangible removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).
- a computer system e.g., on system ROM or fixed disk
- a server or electronic bulletin board over the network
- some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware.
- Still other embodiments of the invention are implemented as entirely hardware, or entirely software.
- the embodiments described above are intended to be merely exemplary; numerous variations and modifications will be apparent to those skilled in the art. All such variations and modifications are intended to be within the scope of the present invention as defined in any appended claims.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Business, Economics & Management (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Tourism & Hospitality (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- Computational Linguistics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Operations Research (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Quality & Reliability (AREA)
- Entrepreneurship & Innovation (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
-
- a database having stored operator information about the participating operators, the operator information of each operator comprising at least one operator class assigned to the respective operator;
- a data receiver configured to receive a data stream comprising a plurality of electronic messages from a plurality of communicating agents, each electronic message comprising at least one piece of information;
- a message parser operatively coupled with the data receiver, the message parser configured to parse the received electronic messages to extract pieces of information of the parsed electronic messages and to classify the electronic messages according to one or more ontologies from a set of pre-defined ontologies;
- a recommendation engine operatively coupled with the message parser and the database, the recommendation engine configured to decide for each extracted piece of information whether it is considered class-relevant information for one or more of the operator classes; and
- a data distributor operatively coupled with the recommendation engine, the data distributor configured to provide, based on the assigned one or more operator classes, to each operator of at least a subset of the plurality of participating operators one or more pieces of information that are considered class-relevant information for the respective operator.
-
- to monitor the electronic devices to detect a behavior of the operators, to determine, based on the detected behavior, an appropriate operator class for the operator, and to dynamically assign the determined appropriate operator class as the operator class in the database; and/or
- to receive user inputs from the electronic devices, the user inputs comprising a selection from the operator of an operator class, and to assign the selected operator class in the database.
-
- to receive user feedback from the operators, the user feedback comprising information on whether a piece of information provided to the operator is considered relevant by the operator, and
- to train the machine-learning algorithm based on the user feedback.
-
- receiving a data stream comprising a plurality of electronic messages from the plurality of communicating agents, each electronic message comprising at least one piece of information;
- using a message parser to parse the received electronic messages and to extract pieces of information;
- classifying the electronic messages according to one or more ontologies from a set of pre-defined ontologies;
- accessing a database to retrieve information about the participating operators, the information of each operator comprising at least one operator class assigned to the respective operator;
- deciding for each extracted piece of information and for one or more of the operator classes, whether the extracted piece of information is considered class-relevant information for the operator class; and
- providing to each operator of at least a subset of the plurality of participating operators, based at least on the one or more operator classes assigned to the respective operator, pieces of information that are considered class-relevant information for the respective operator.
-
- program code for receiving a data stream comprising a plurality of electronic messages from a plurality of communicating agents, each electronic message comprising at least one piece of information;
- program code for parsing, by a message parser, the received electronic messages and to extract pieces of information;
- program code for classifying the electronic messages according to one or more ontologies from a set of pre-defined ontologies;
- program code for accessing a database to retrieve information about the participating operators, the information of each operator comprising at least one operator class assigned to the respective operator;
- program code for deciding for each extracted piece of information and for one or more of the operator classes, whether the extracted piece of information is considered class-relevant information for the operator class; and
- program code for providing to each operator of at least a subset of the plurality of participating operators, based at least on the one or more operator classes assigned to the respective operator, pieces of information that are considered class-relevant information for the respective operator.
-
- a) pre-defined operator rules, such as a match against a geo-temporal signature of the extracted message;
- b) machine learning and a pre-trained artificial intelligence; or
- c) a hybrid approach of (a) and (b).
-
- organizing data streams for each organization, e.g. to categorize if a given police department report concerns the “homicide” division or the “robbery” division;
- proactive recommendations on possible collaborations across organizations using the pool of agents, e.g. if the AI agent detected “accident” and “power-line breakdown” from a report, trigger a collaboration channel between police department and utilities department;
- highlighting important bits of information from the raw data streams for faster visual screening by operators;
- notifications for the operators regarding related content that might assist them with an ongoing investigation; and
- using operator interactions (e.g. clickstream data and ratings) as a proxy for human-in-loop feedback to update the social graph and data links.
Claims (20)
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/715,895 US12067042B2 (en) | 2022-04-07 | 2022-04-07 | System and method of enabling and managing proactive collaboration |
| CN202310356834.5A CN116896539A (en) | 2022-04-07 | 2023-04-04 | Systems and methods for enabling and managing proactive collaboration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/715,895 US12067042B2 (en) | 2022-04-07 | 2022-04-07 | System and method of enabling and managing proactive collaboration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| US20230325427A1 US20230325427A1 (en) | 2023-10-12 |
| US12067042B2 true US12067042B2 (en) | 2024-08-20 |
Family
ID=88239382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US17/715,895 Active 2042-06-30 US12067042B2 (en) | 2022-04-07 | 2022-04-07 | System and method of enabling and managing proactive collaboration |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US12067042B2 (en) |
| CN (1) | CN116896539A (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20240296340A1 (en) * | 2023-03-01 | 2024-09-05 | Mitsubishi Slectric Research Laboratories, Inc. | Methods to Improve Federated Learning Robustness in Internet of Vehicles |
Citations (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030649A (en) * | 2002-06-24 | 2004-01-29 | Internatl Business Mach Corp <Ibm> | Delivery of desired information to many users |
| FI118751B (en) * | 2005-03-24 | 2008-02-29 | First Hop Ltd | Information collection from traffic flow in a communication network |
| US20120259926A1 (en) * | 2011-04-05 | 2012-10-11 | Lockhart Kendall G | System and Method for Generating and Transmitting Interactive Multimedia Messages |
| US20120259852A1 (en) * | 2011-04-06 | 2012-10-11 | Nterop Corporation | Method and apparatus for pushing situationally relevant data |
| US20130226559A1 (en) * | 2012-02-24 | 2013-08-29 | Electronics And Telecommunications Research Institute | Apparatus and method for providing internet documents based on subject of interest to user |
| US20140280166A1 (en) * | 2013-03-15 | 2014-09-18 | Maritz Holdings Inc. | Systems and methods for classifying electronic documents |
| EP3187951A1 (en) * | 2016-01-04 | 2017-07-05 | Rockwell Automation Technologies, Inc. | Delivery of automated notifications by an industrial asset |
| GB2546797A (en) * | 2016-01-29 | 2017-08-02 | Waazon (Holdings) Ltd | Automated search method,apparatus and database |
| US20180129971A1 (en) * | 2016-11-10 | 2018-05-10 | Adobe Systems Incorporated | Learning user preferences using sequential user behavior data to predict user behavior and provide recommendations |
| US20180182015A1 (en) * | 2016-12-28 | 2018-06-28 | Google Inc. | System for Determining and Displaying Relevant Explanations for Recommended Content |
| US10021185B1 (en) * | 2013-08-22 | 2018-07-10 | Ca, Inc. | Optimized virtual storage fabric |
| US20190095495A1 (en) * | 2017-09-27 | 2019-03-28 | Oracle International Corporation | Computerized methods and systems for grouping data using data streams |
| US20190342415A1 (en) * | 2015-09-29 | 2019-11-07 | Tencent Technology (Shenzhen) Company Limited | Event information push method, event information push apparatus, and storage medium |
| US10565372B1 (en) * | 2017-08-03 | 2020-02-18 | Amazon Technologies, Inc. | Subscription-based multi-tenant threat intelligence service |
| US20200073932A1 (en) * | 2018-08-30 | 2020-03-05 | Intelligent Fusion Technology, Inc | Method and system for pattern discovery and real-time anomaly detection based on knowledge graph |
| US10705796B1 (en) * | 2017-04-27 | 2020-07-07 | Intuit Inc. | Methods, systems, and computer program product for implementing real-time or near real-time classification of digital data |
| CN111400595A (en) * | 2020-03-13 | 2020-07-10 | 上海传英信息技术有限公司 | Application content push method, device and storage medium |
| US20200272914A1 (en) * | 2019-02-25 | 2020-08-27 | Microsoft Technology Licensing, Llc | Context-based recommendations based on environment interactions |
| US20210004391A1 (en) * | 2019-07-02 | 2021-01-07 | Jpmorgan Chase Bank, N.A. | Methods and systems for classifying and filtering messages via natural language processing |
| WO2021032620A1 (en) * | 2019-08-16 | 2021-02-25 | Subfiber OÜ | Selecting items from a database |
| US20210056425A1 (en) * | 2019-08-23 | 2021-02-25 | Samsung Electronics Co., Ltd. | Method and system for hybrid model including machine learning model and rule-based model |
| US20210125098A1 (en) * | 2019-10-29 | 2021-04-29 | International Business Machines Corporation | Retraining individual-item models |
| US20210174423A1 (en) * | 2016-06-29 | 2021-06-10 | Go Daddy Operating Company, LLC | Hosting systems configured to generate optimizzed content based on dynamic tokens and methods of use thereof |
| US20210209109A1 (en) * | 2020-06-29 | 2021-07-08 | Beijing Baidu Netcom Science Technology Co., Ltd. | Method, apparatus, device, and storage medium for intention recommendation |
| US20210241299A1 (en) * | 2020-01-30 | 2021-08-05 | Walmart Apollo, Llc | Methods and apparatus for detecting and correcting item pricing anomalies |
| US11238410B1 (en) * | 2019-12-17 | 2022-02-01 | iCIMS, Inc. | Methods and systems for merging outputs of candidate and job-matching artificial intelligence engines executing machine learning-based models |
| US20220122731A1 (en) * | 2020-08-21 | 2022-04-21 | Cambia Health Solutions, Inc. | Systems and methods for generating and delivering personalized healthcare insights |
| US20220237374A1 (en) * | 2021-01-26 | 2022-07-28 | Microsoft Technology Licensing, Llc | Content element recommendation system |
| US20220236843A1 (en) * | 2021-01-26 | 2022-07-28 | Microsoft Technology Licensing, Llc | Collaborative content recommendation platform |
| US20220245557A1 (en) * | 2021-01-29 | 2022-08-04 | AmplifAI | Analyzing agent data and automatically delivering actions |
| US20220246286A1 (en) * | 2021-02-04 | 2022-08-04 | Unitedhealth Group Incorporated | Use of audio data for matching patients with healthcare providers |
| US11461067B2 (en) * | 2020-12-17 | 2022-10-04 | International Business Machines Corporation | Shared information fields with head mounted displays |
| US20230098882A1 (en) * | 2021-09-30 | 2023-03-30 | Adobe Inc. | Automated content selection based on multiple surface inputs, behavior and machine learning |
| US20230138836A1 (en) * | 2021-10-28 | 2023-05-04 | Microsoft Technology Licensing, Llc | Managing a collaborative workspace to identify relevant activities and present the relevant activities to a user |
| US20230141398A1 (en) * | 2021-11-11 | 2023-05-11 | Servicenow, Inc. | Data augmentation for intent classification |
| US20230169527A1 (en) * | 2021-11-30 | 2023-06-01 | Qualtrics, Llc | Utilizing a knowledge graph to implement a digital survey system |
| US20230267922A1 (en) * | 2022-02-18 | 2023-08-24 | Google Llc | Meeting speech biasing and/or document generation based on meeting content and/or related data |
| US11798675B2 (en) * | 2018-05-29 | 2023-10-24 | Otsuka America Pharmaceutical, Inc. | Generating and searching data structures that facilitate measurement-informed treatment recommendation |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10333995B2 (en) * | 2014-01-23 | 2019-06-25 | International Business Machines Corporation | Providing of recommendations determined from a collaboration session system and method |
| CN114201691A (en) * | 2021-12-17 | 2022-03-18 | 上海数禾信息科技有限公司 | Information recommendation method, apparatus, computer equipment and storage medium |
-
2022
- 2022-04-07 US US17/715,895 patent/US12067042B2/en active Active
-
2023
- 2023-04-04 CN CN202310356834.5A patent/CN116896539A/en active Pending
Patent Citations (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004030649A (en) * | 2002-06-24 | 2004-01-29 | Internatl Business Mach Corp <Ibm> | Delivery of desired information to many users |
| FI118751B (en) * | 2005-03-24 | 2008-02-29 | First Hop Ltd | Information collection from traffic flow in a communication network |
| US20120259926A1 (en) * | 2011-04-05 | 2012-10-11 | Lockhart Kendall G | System and Method for Generating and Transmitting Interactive Multimedia Messages |
| US20120259852A1 (en) * | 2011-04-06 | 2012-10-11 | Nterop Corporation | Method and apparatus for pushing situationally relevant data |
| US20130226559A1 (en) * | 2012-02-24 | 2013-08-29 | Electronics And Telecommunications Research Institute | Apparatus and method for providing internet documents based on subject of interest to user |
| US20140280166A1 (en) * | 2013-03-15 | 2014-09-18 | Maritz Holdings Inc. | Systems and methods for classifying electronic documents |
| US10021185B1 (en) * | 2013-08-22 | 2018-07-10 | Ca, Inc. | Optimized virtual storage fabric |
| US20190342415A1 (en) * | 2015-09-29 | 2019-11-07 | Tencent Technology (Shenzhen) Company Limited | Event information push method, event information push apparatus, and storage medium |
| EP3187951A1 (en) * | 2016-01-04 | 2017-07-05 | Rockwell Automation Technologies, Inc. | Delivery of automated notifications by an industrial asset |
| GB2546797A (en) * | 2016-01-29 | 2017-08-02 | Waazon (Holdings) Ltd | Automated search method,apparatus and database |
| US20210174423A1 (en) * | 2016-06-29 | 2021-06-10 | Go Daddy Operating Company, LLC | Hosting systems configured to generate optimizzed content based on dynamic tokens and methods of use thereof |
| US20180129971A1 (en) * | 2016-11-10 | 2018-05-10 | Adobe Systems Incorporated | Learning user preferences using sequential user behavior data to predict user behavior and provide recommendations |
| US20180182015A1 (en) * | 2016-12-28 | 2018-06-28 | Google Inc. | System for Determining and Displaying Relevant Explanations for Recommended Content |
| US10705796B1 (en) * | 2017-04-27 | 2020-07-07 | Intuit Inc. | Methods, systems, and computer program product for implementing real-time or near real-time classification of digital data |
| US10565372B1 (en) * | 2017-08-03 | 2020-02-18 | Amazon Technologies, Inc. | Subscription-based multi-tenant threat intelligence service |
| US20190095495A1 (en) * | 2017-09-27 | 2019-03-28 | Oracle International Corporation | Computerized methods and systems for grouping data using data streams |
| US11798675B2 (en) * | 2018-05-29 | 2023-10-24 | Otsuka America Pharmaceutical, Inc. | Generating and searching data structures that facilitate measurement-informed treatment recommendation |
| US20200073932A1 (en) * | 2018-08-30 | 2020-03-05 | Intelligent Fusion Technology, Inc | Method and system for pattern discovery and real-time anomaly detection based on knowledge graph |
| US20200272914A1 (en) * | 2019-02-25 | 2020-08-27 | Microsoft Technology Licensing, Llc | Context-based recommendations based on environment interactions |
| US20210004391A1 (en) * | 2019-07-02 | 2021-01-07 | Jpmorgan Chase Bank, N.A. | Methods and systems for classifying and filtering messages via natural language processing |
| WO2021032620A1 (en) * | 2019-08-16 | 2021-02-25 | Subfiber OÜ | Selecting items from a database |
| US20210056425A1 (en) * | 2019-08-23 | 2021-02-25 | Samsung Electronics Co., Ltd. | Method and system for hybrid model including machine learning model and rule-based model |
| US20210125098A1 (en) * | 2019-10-29 | 2021-04-29 | International Business Machines Corporation | Retraining individual-item models |
| US11238410B1 (en) * | 2019-12-17 | 2022-02-01 | iCIMS, Inc. | Methods and systems for merging outputs of candidate and job-matching artificial intelligence engines executing machine learning-based models |
| US20210241299A1 (en) * | 2020-01-30 | 2021-08-05 | Walmart Apollo, Llc | Methods and apparatus for detecting and correcting item pricing anomalies |
| CN111400595A (en) * | 2020-03-13 | 2020-07-10 | 上海传英信息技术有限公司 | Application content push method, device and storage medium |
| US20210209109A1 (en) * | 2020-06-29 | 2021-07-08 | Beijing Baidu Netcom Science Technology Co., Ltd. | Method, apparatus, device, and storage medium for intention recommendation |
| US20220122731A1 (en) * | 2020-08-21 | 2022-04-21 | Cambia Health Solutions, Inc. | Systems and methods for generating and delivering personalized healthcare insights |
| US11461067B2 (en) * | 2020-12-17 | 2022-10-04 | International Business Machines Corporation | Shared information fields with head mounted displays |
| US20220236843A1 (en) * | 2021-01-26 | 2022-07-28 | Microsoft Technology Licensing, Llc | Collaborative content recommendation platform |
| US20220237374A1 (en) * | 2021-01-26 | 2022-07-28 | Microsoft Technology Licensing, Llc | Content element recommendation system |
| US20220245557A1 (en) * | 2021-01-29 | 2022-08-04 | AmplifAI | Analyzing agent data and automatically delivering actions |
| US20220246286A1 (en) * | 2021-02-04 | 2022-08-04 | Unitedhealth Group Incorporated | Use of audio data for matching patients with healthcare providers |
| US20230098882A1 (en) * | 2021-09-30 | 2023-03-30 | Adobe Inc. | Automated content selection based on multiple surface inputs, behavior and machine learning |
| US20230138836A1 (en) * | 2021-10-28 | 2023-05-04 | Microsoft Technology Licensing, Llc | Managing a collaborative workspace to identify relevant activities and present the relevant activities to a user |
| US20230141398A1 (en) * | 2021-11-11 | 2023-05-11 | Servicenow, Inc. | Data augmentation for intent classification |
| US20230169527A1 (en) * | 2021-11-30 | 2023-06-01 | Qualtrics, Llc | Utilizing a knowledge graph to implement a digital survey system |
| US20230267922A1 (en) * | 2022-02-18 | 2023-08-24 | Google Llc | Meeting speech biasing and/or document generation based on meeting content and/or related data |
Non-Patent Citations (2)
| Title |
|---|
| Chanda et al., "A Parallel Processing Technique for Extracting and Storing User Specified Data", the 8th International Conference on Future Internet of Things and Cloud (FiCloud), 2021, pp. 241-249. (Year: 2021). * |
| Luo et al., "Design and Research of Intelligent Message Platform System in Hospital", the 2020 IEEE International Conference on E-Health Networking, Application & Services (HEALTHCOM), 2020, 5 pages. (Year: 2020). * |
Also Published As
| Publication number | Publication date |
|---|---|
| US20230325427A1 (en) | 2023-10-12 |
| CN116896539A (en) | 2023-10-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11093568B2 (en) | Systems and methods for content management | |
| Benefo et al. | Ethical, legal, social, and economic (ELSE) implications of artificial intelligence at a global level: a scientometrics approach | |
| US20210110475A1 (en) | Subscription-enabled news recommendation system | |
| US9923931B1 (en) | Systems and methods for identifying violation conditions from electronic communications | |
| US9535902B1 (en) | Systems and methods for entity resolution using attributes from structured and unstructured data | |
| US11204929B2 (en) | Evidence aggregation across heterogeneous links for intelligence gathering using a question answering system | |
| US11244113B2 (en) | Evaluating evidential links based on corroboration for intelligence analysis | |
| US11106662B2 (en) | Session-aware related search generation | |
| Lamsal et al. | Socially enhanced situation awareness from microblogs using artificial intelligence: A survey | |
| Khatoon et al. | Development of social media analytics system for emergency event detection and crisis management | |
| CN116991873B (en) | Target report data processing method and device based on knowledge graph and server | |
| CN119903159A (en) | A knowledge question-answering fast processing system based on artificial intelligence | |
| Vallejos et al. | Mining social networks to detect traffic incidents | |
| Thomas et al. | Semi‐supervised, knowledge‐integrated pattern learning approach for fact extraction from judicial text | |
| JP2017201437A (en) | News material extractor and program | |
| Shahsavari et al. | Event identification for supply chain risk management through news analysis by using large language models | |
| KR102228873B1 (en) | Construction system of criminal suspect knowledge network using public security information and Method thereof | |
| Song et al. | Social media crisis communication and public engagement during COVID-19 analyzing public health and news media organizations’ tweeting strategies | |
| US20230325427A1 (en) | System and method of enabling and managing proactive collaboration | |
| Parolin et al. | Come-ke: A new transformers based approach for knowledge extraction in conflict and mediation domain | |
| Teng et al. | Simplified data science approach to extract social and behavioural determinants: a retrospective chart review | |
| Pei et al. | A new method for early detection of mass concern about public health issues | |
| Long et al. | Automated crisis content categorization for covid-19 tweet streams | |
| Lovera et al. | An analytics approach to extracting location and sentiment insights from classified social media data | |
| Rezaei et al. | Image-based world-perceiving knowledge graph (wpkg) with imprecision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
| AS | Assignment |
Owner name: HEXAGON TECHNOLOGY CENTER GMBH, SWITZERLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BUDHRAJA, AKSHIT;BOESCH BORGARDS, TOBIAS;REIMANN, BERND;SIGNING DATES FROM 20220224 TO 20220228;REEL/FRAME:059564/0686 |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
| ZAAA | Notice of allowance and fees due |
Free format text: ORIGINAL CODE: NOA |
|
| ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
| STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
| AS | Assignment |
Owner name: HEXAGON INNOVATION HUB GMBH, SWITZERLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEXAGON TECHNOLOGY CENTER GMBH;REEL/FRAME:073833/0471 Effective date: 20251118 |