KR20020070491A - Candidate level multi-modal integration system - Google Patents

Abstract

The multi-modelity system includes sensing devices and a multi-modelity integration unit. The sensing devices provide a list of candidate pairs, each pair including a candidate characteristic and a probability indicative of the reliability of that candidate. The multi-modelity integration unit receives the lists from the sensing device and provides the multi-modelity context information to each sensing device in response to the lists. The sensing devices then repeatedly use the new information from the multi-modelity integration unit to change the sensing or other capabilities, repeatedly providing new lists of candidate pairs. A super-system comprising a hierarchy of these multiple modelability integration units can be configured.

Description

Candidate level multi-modal integration system

Background of the present invention

The field of multi-modelity integration has a number of applications. This will be discussed longer than in the detailed description. Several approaches have been adopted in prior art attempts at multi-modelity integration. "A multimodal biometric system using fingerprint, face and speech at the 2nd International Conference on Biological Human Authentication based on Audio and Video (Washington, USA, March 99). ) Is known by A. Jain et al. As an example. The general approach to this work will be called "decision level integration" here. 1 shows a conceptual diagram of decision level integration applied to data taken from " scene " The scene is detected and processed by at least two separate modules at 102 and 103. Each module includes a sensing operation 104, a feature extraction operation 105, and a recognition operation 106. Each module yields a uni-modal ("UM") "determination" 107 that characterizes or labels the data collected from the scene. Here, the term "characterization" is intended to be a general term that includes both concepts of "determination" and "label".

Feature extraction 105 generally involves applying mathematical data or data obtained in the step of sensing a predetermined algorithm. Recognition 106 generally involves a type of processing that requires some training, such as through the use of a neural network. Later, a multi-modal integration unit (“MMI”) applies multiple modelity heuristics and / or rules to determine how to produce the final multi-modelity decision, which is Characterize or label several aspects of the scene based on disparate data collected and processed in processes 102 and 103.

Decision level integration has the advantage of being simple to implement. It may include single modeling systems that are independently studied, developed, and updated. Thus, these systems can operate as pre-processors. In addition, the communication channels between single modeling systems and the MMI are unidirectional and have a small bandwidth.

However, decision level integration is limited at the level of cooperation that can be implemented between different modelities. In general, the correlation between modelities is not fully exploited, so information from one modelity cannot be used to improve the decisions made on the others. For example, if the determinations from two redundant modelities do not match, the most reliable one will be taken and the rest will be discarded and compete with the rest, so as not to drop the results obtained with one modelity, There is no overall improvement.

FIELD OF THE INVENTION

The present invention also relates to field integrating signals representing data sensed from multiple sensing modalities, also known as multi-modal integration, and in particular to data. It relates to the integration of data from multiple sensory modelities that preprocess and at least a tentative characterization or labeling of the data.

1 illustrates a prior art multi-modelity integration architecture.

2 illustrates a prior art multi-modelity integration architecture.

3 shows a system according to the invention.

4 shows a system according to the invention.

5 shows a list of symbols used to illustrate the present invention.

6 is a flow chart illustrating the development of a candidate list.

7 is a schematic diagram of a super-system including several MMI devices.

8 is a flowchart for explaining an operation of an MMI.

Enhancing multi-modelity integration by improving the collaborative use of data between single-modelity contributors for multiple-modelity systems and continuing to maintain the benefits of preprocessing from independent single-modelity systems Purpose.

This object is achieved in that independent single modeling systems generate sets of characterization pairs, each pair containing each candidate characteristic and confidence level. The MMI receives and processes sets of characteristic pairs and supplies at least one final characteristic of the signals. The final characteristic is selected from at least one of the characteristic pairs.

Alternatively, the object is achieved in that the MMI receives signals from a single modelity contributors that characterize the candidate and provides at least one control signal thereto. Control signals control processing and / or sensing. The control signal is derived from signals that characterize the candidate.

In a second alternative, the object is achieved with a training method. The method includes a training phase and a normal operating phase.

At the training phase, candidate characteristic signals and ground truths are received. Candidate characteristic signals are received from a number of previously trained sensing devices, the devices comprising trained processors, the candidate characteristic signals occurring as an initial physical reality setting. The training parameters are then adjusted to achieve ground true with respect to physical reality by evaluating optimal criteria and candidate characteristic signals.

At normal operation phase, other candidate characteristic signals are received from a number of previous trained sensing devices. A temporary final characteristic signal is generated. At least one control signal is then fed back to at least one of the sensing devices. The control signal is adapted to cause a change in the training and / or performance of the sensing device. The steps of normal operation phase are repeated until the characteristic criterion is met.

In addition, an object of the present invention is achieved in a single modelity sensing apparatus that provides characteristic information above the multi-modelity integration unit and receives the multi-modelity context information down in the multi-modelity integration unit.

In the related art, further cooperation between single modeling devices is achieved without preprocessing. This field will be referred to herein as "feature level integration." Examples of this field will be found in US Pat. No. 5,586,215. 2 illustrates a general concept of feature level processing. In addition, at 101, a scene appears. At 202, at least two different types of sensing occur. Then, at 203, all sensed data undergo some type of feature extraction to yield a feature vector. The feature vector is then processed at 204 to yield some kind of multi-modelity and at 205 a multi-modelity decision is output. In addition, feature extraction typically results from applying some kind of mathematical transformation or some algorithm to the sensed data; Recognition is an action that usually requires some kind of training, such as the use of neural networks.

Areas of application

There are a number of application areas for making multi-modal decisions. One is lip-reading, where audio data for phoneme recognition is used in combination with video data in an effort to understand the speaker. Similarly, identification of a person may include a combination of audio and video data.

Other areas in which multiple modelability information may be incorporated may be present in image processing, for example local or global 2-D shapes, color characterizations, gray level appearance, and Different image aspects, which are textural properties, may all be considered when characterizing an image. In general, the sensing video data is contained in a fairly large number of things, for example, fingerprints and facial images, including feature locations, feature appearance, and profile shapes. It may include information relating to gathering and characterizing.

One camera may be used to gather one or more types of data about a scene with different processing modules in the connected processor that use the data in different ways. Modules that then collect different types of information from the image effectively result in different sensing devices, even though they may be physically housed within a single processor.

In other applications, signals from different types of sensors may need to be combined. Other types of sensors that may be useful in multi-modelity integration applications include infrared and range sensors. In addition, user entry devices including keyboards and printer devices, such as, for example, mice, stylus type sensors, track balls, etc., may be used as single-modelity sensing devices. Can be used. Other areas where multi-modeling integration may be useful include acoustic localization through microphone arrays and the use of echo cancellation by direct input of known audio / noise sources. Even text data may be used in some applications.

In general, one skilled in the art can devise a fairly large number of applications for multi-modelity integration.

Architecture of the system according to the invention

3 and 4 are shown in relation to a system having two video and audio single-modelity systems, such as a video conferencing system. However, the concepts of candidate level integration may equally apply to many other applications, for example those listed in the preceding section.

3 shows the architecture of a system according to the invention. In addition, there is a scene 101 sensed by the sensors 301, 301 ′ and 302. While these sensors are shown as microphones and video cameras, they may be any sensors suitable for the desired area of application, including for example keyboards, mice, user writing devices that are touch screens or any other user writing device. have. At 303, 304, and 305, features are extracted from signals derived from sensors. Features extracted at 306, 307, and 308 are processed and recognized. At 310, 312, and 314, candidate decisions appear in MMI 317. At 309, 311, and 313, control signals are provided to back down boxes 306, 307, and 308 in the form of multi-modelity context information.

In this system, two sets of features are shown extracted from video data at 304 and 305. For example, facial feature data may be extracted at 305, while gesture feature data may be extracted at 304. Boxes 305 and 308 function together as separate sensing devices from boxes 304 and 307. Thus, the video camera 302 is actually connected to two sensing devices. In other words, a single sensing element can be connected with quite a large number of sensing devices.

In contrast to the video portion of the system, multiple microphones 301 and 301 ′ in the array function with a single pair of boxes 303 and 306. The boxes 303 and 306 thus function together as a third sensing device to collect position data, for example. Thus, one or more sensing elements can supply a single sensing device.

Additional sensing devices may be added whether or not coupled to existing sensing elements or additional sensing elements. There may be a significant number of sensing elements and sensing devices.

The control data fed back at 309, 311 and 313 will affect the performance and / or training of the respective sensing devices. For example, control signals for the video sensing device can bias what the sensing device sees as part of the picture.

In FIG. 3, the sensing devices are shown in the same processor 316 with the MMI 317. In FIG. 4, an alternative embodiment is shown, with the sensing devices 416, 417, and 418 received separately from the MMI 417. The connections 409-414 that supply the candidate decisions are now external leads.

Boxes 303-305 perform feature extraction on data received from the scene. The output of the boxes 303-305 will be in the form of feature vectors per formula (3) in FIG. 5.

Boxes 306-308 generate candidate lists in accordance with the present invention.

In the prior art, generally only a single decision is made based on the identification function or the value of the functions. The field of identification functions is well developed as described, for example, by K. Fukunaga in Introduction to Statistical Pattern Recognition (2nd edition, Academy Publishing, 10/99). If a single identification function is applied, it will typically have multiple local maxima. The next decision will be the largest of its local maximums. If multiple identification functions are applied or if a single identification function is applied repeatedly to various parts of the data, the decision will be the largest value received from all functions or applications of the function.

In a preferred embodiment, the identification functions will generally be probability distributions denoted here as "P". However, those skilled in the art will be able to devise other identification functions depending on their needs no matter what application area is selected.

In accordance with the present invention, it is desirable to supply a candidate list from sensing devices on lines 310, 312 or 314. Each sensing device will generate a candidate list for each formula (1) of FIG. 5, where

-Is a variable representing a candidate from a single modelity detection device,

k is an index variable that numbers single modelity sensing devices,

i is an index variable,

M _k is the number of candidates to be generated for a single modelity sensing device number K.

6 is a flow diagram illustrating further of the operation of the individual recognition units 306-308 within the sensing devices. The labels in the flowchart refer to official numbers from FIG. 5.

At 601, the list of multi-modelity context information of FIG. 5 is received from lines 313, 311 and 309 in the form of an initialized list of default values for formula (2). At 602, formula (5) is applied to get candidates (1). Formula (5) represents the product of the results of formula (2), received from MMI, with an identification function based on probability, per formula (4). At 603, several criteria are evaluated. The criterion may be some fixed number of iterations completed, or any other suitable criterion in which no change in the candidate list 6 has been achieved since the last iteration, or designed by a skilled technician. If the criteria are not met, then at 604 the current list of candidate pairs per formula (6) is sent to MMI 317, 417. The candidate pair list includes candidates from formula (1) with confidence levels from formula (4). The candidate pair list is an example of the term “characterization pairs” used elsewhere herein and is provided in the MMI on lines 310, 312 and 314. At 606, the new multi-modelity context information is received from the MMI at 606 in the form of formula (2), based on the newly proposed candidate list and control is returned to 602.

If the criteria are met, then at 605 the final set of candidates in the form of formula (6) is sent to the MMI. MMI 317 and 417 alternately perform evaluation of all combinations of candidates from single modelity sensing devices. 8 shows a flowchart of the operation of the MMI.

At 801, candidate pair lists per formula (6) are received from single modelity sensing devices. Each single modelability sensing device k generates a list of candidate pairs per formula (6).

At 802, a list of combinations of single modelity candidates is formed as shown in equation (7). The total number of combinations is L and the index numbering the combinations is c.

Each combination of candidates generally includes one single modelity candidate from each single modelity sensing apparatus. Each combination of single modelity candidates is used to generate multiple modelability properties c ^* of the scene. The multi-modelity feature may be the same as one of the features 1 from the single modelity sensing devices. Alternatively, the multi-modelity feature can be characterized by several combination patterns derived from patterns recognized by single modeling devices.

Multi-modelity properties are analyzed according to the multi-modelity identification function 8. This function includes: a) the product of the super multi-modelity context information P (c); And b) for each formula (4) evaluate the product of all probabilities of all monomodelity decisions and the product of the probability function applied to each combination. Similarly, using a single modeling system, the super multi-modelity contextual information P (c) will be initialized with some default values first. Fortunately, the value of P (c) can then be modified based on the information received at a higher level from the MMI. This modified value will then be supplied as new super multi-modelity context information from a higher level.

Based on the analysis described in formula (8), super candidates are selected to be supplied from the MMI. This is a subset {c ^* } of possible combinations 7. Super candidates will be provided as another list of feature pairs. The property pairs will then have the format of formula (9).

Next, at 803, the criteria are tested. This criterion may be multiple iterations, lack of change in output 2 since the last iteration, lack of change in multiple modelability candidate pairs 9 since the last iteration, or any other suitable criterion devised by a skilled technician. have. If the criteria are not met, then per model (2) multi-modelity context information is sent to the individual single modeling devices at 804. The values sent to a single modelity device will typically vary depending on what type of data the device collects.

7 shows a system with a super MMI 701. In this case, there are three MMIs 702-704, each corresponding to the MMIs 317, 417 discussed previously. Each MMI is connected with a number of single modelity sensing devices 705. The MMIs 702-704 send the super candidate lists, i.e., property pairs, per formula 9 to the super MMI 701 per 707, and the super multi-modelity context information P via the 706 from the super MMI 701. (c) is received. The super MMI may generate different feature pairs at 708, and thus may be part of a super-super-MMI system, with different levels of hierarchy. Super MMI 70 operates similarly to MMI treating MMIs as MMIs treat single modelity sensing devices.

In FIG. 7, there are three MMIs 702 each having three single modelity sensing devices 705. However, those skilled in the art will appreciate that there may be other numbers of components. For example, a super MMI can be coupled with at least one MMI and at least one free-standing single biounit sensing device. There may alternatively be two MMIs, each supplied by two single modelity sensing devices or the like.

Upon reading this disclosure, other modifications will become apparent to those skilled in the art. Such modifications may already be known in the use of design, manufacture and recognition of sensed data, and may involve other features in addition to the features that may be used or described herein. Although the claims are formulated in this application with specific combinations of features,

The scope of the present application also includes any new features or novel combinations of features described herein, explicitly or implicitly, or any generalization thereof, whether this mitigates any or all technical problems equivalent to those of carrying out the present invention. This will be understood. Applicants note that new claims may be formulated with these features during the prosecution of the present application or any other application derived therefrom.

The words "comprising", "comprise" or "comprises" as used herein should not be viewed as excluding additional elements. The singular article “a” or “an”, as used herein, should not be regarded as excluding a number of elements.

Claims (18)

In the multi-modal integration unit 317, 417, 702, 703, 704, From each of the plurality of sensing devices 416-418, 303-308, a set of characterization pairs ((6)) containing each candidate characteristic (1) and each candidate characteristic Means (310, 312, 314, 410, 412, 414) for receiving an indication of each trust (4) relating, wherein said pairs of characteristics result from preprocessing in sensing devices, Means for processing sets of characteristic pairs and for supplying 315 a final characteristic (9) of at least one of the signals received at the sensing devices, the final characteristic being selected from at least one of the characteristic pairs; Means for multi-modeling integration. In a data processing system, At least one sensing element 301, 301 ′, 302 adapted to receive input signals indicative of physical reality, and A plurality of processors or processes 303-308, 416-418, adapted to supply respective characteristic signals for each sensing device that characterize the input signals, wherein the respective characteristic signals from each sensing device are the pairs of characteristic pairs. A plurality of processors or processes (303-308, 416-418), each set comprising a set, each pair of characteristics including each candidate characteristic and each confidence indication for each candidate characteristic; With a plurality of sensing devices, including A data processing system comprising a multi-modeling integration unit as claimed in claim 1. 3. The system of claim 2, wherein the multi-modeling integration unit employs a discriminating function to process sets of character pairs. 4. The system of claim 3, wherein the identification function is a probability distribution. In a super-system, At least one system as claimed in claim 2, wherein the at least one final characteristic for each system comprises a different set of respective pairs of characteristics (9); If there is only one such system, it is at least one other uni-modal sensing device, Receive and process different sets of feature pairs along with signals from at least one other sensing device, Adapted to supply a super-final characteristic 708 of at least one of the signals At least one super-multiple modelity integration unit 701, wherein the super-final feature comprises the at least one super-multiple modelability integration unit selected from at least one feature pair from another set of feature pairs , Super-system. In the multi-modelity integration unit 702-704, 317, 417, Receive respective candidate characteristic signals (1), (6) from each of a plurality of sensing devices 303-308, 416-418 having preprocessing capability (314, 312, 310, 410, 412, 414). Candidate reception signals are characterized by physical existence, Means 409, 411, 413, 309, 311, 313 for supplying at least one control signal to the sensing devices for controlling processing and / or sensing at the sensing devices, and Means for processing the candidate characteristic signals to derive at least one final characteristic signal and at least one control signal therefrom. In a data processing system, At least one sensing element 301, 301 ′, 302 adapted to receive input signals indicative of physical existence, and Provide at least one candidate characteristic signal representing a characteristic of the physical entity based on input signals (414, 412, 410), Adapted to receive (413, 411, 409) control signals for controlling processing and / or sensing Each comprising at least one processor or process (303-308, 416-418) Multiple sensing devices, A data processing system comprising the multi-modelity integration unit of claim 6. 8. The system of claim 7, wherein the control signals relate to biasing the selection of a physical entity. 8. The system of claim 7, wherein the control signals are in the form of feedback from the multi-modelity integration unit to the sensing devices. 8. The system of claim 7, wherein each candidate characteristic signal comprises a respective candidate list ((1)) from each sensing device. 11. The method of claim 10, wherein each candidate list is a set of feature pairs (6), each feature pair comprising each candidate feature (1) and each candidate representation (4) relating to each candidate feature. Including, system. 8. The system of claim 7, wherein the multi-modelity integration unit employs an identification function for processing each candidate characteristic signals. 13. The system of claim 12, wherein the identification function is a probability distribution. In a super-system, At least one systems as claimed in claim 12, If there is only one system, at least one other single modelity sensing device, Receive and process any signals from at least one final characteristic signal (9), 707 from any other single modelity sensing device from at least one system, Suitable for deriving from at least one super-final characteristic signal 701 At least one super multi-modelity integration unit (701). A sensing device suitable for use in a multi-modelity integration system as claimed in claim 7, wherein the sensing device comprises: Receive signals indicative of physical existence from at least one sensing element 301, 301 ′, 302, For bidirectional communication with multi-modelity integration unit (409-414, 309-314) Connection means, Receive control signals (409, 411, 413, 309, 311, 313) from multiple modeling integration units 317, 417, 702-704 to control processing and / or sensing; Provide signals 410, 412, 414, 310, 312, 314 representing a list of candidate physical properties for the multiple modeling integration unit in response to control signals. At least one processor or process (303-308, 416-418) adapted. The apparatus of claim 15, wherein the data representing the physical entity comprises video data. The apparatus of claim 16, wherein the control signals bias the video data to a portion of the field of view. In the method of training a data processing system, The following operations on at least one data processing device: Receiving candidate characteristic signals from a plurality of previous trained sensing devices, wherein the devices include trained processors, the candidate characteristic signals being derived from an initial physical reality setting; Searching for signals representing ground truths about physical reality, Adjusting training parameters to achieve ground true by evaluating optimal criteria and candidate characteristic signals, Fill out the training pace, After completing the training pace, Receiving other candidate characteristic signals from a plurality of previous trained sensing devices, Generating a temporary final characteristic signal, Feedbacking at least one control signal to at least one of the sensing devices, wherein the control signal is adapted to cause a change in training and / or performance of at least one of the sensing devices, Performing writing a normal operation phase comprising repeating receiving, generating, and feeding back another candidate characteristic signal until the characteristic criteria are met.