JP5544508B2 - Action identification system - Google Patents

Description

  The present invention relates to an action identification system, and more particularly to an action identification system that identifies an action performed by a nurse in a medical field based on sensor information from a sensor and time information from a clock.

  As a conventional system of this type, a system described in Patent Document 1 is known. In this background art, an accelerometer is attached to a subject (nurse) to collect acceleration data, while an experimenter accompanies the subject with action time (start time and end time) and label (identification information). Record. For example, when the subject walks, the experimenter identifies the time and gives the label “walking”. The time specified by the experimenter and the given label are input to the server together with the acceleration data collected by the acceleration sensor, and the server cuts out the portion belonging to the time from the acceleration data and calculates its characteristics. The labels are associated with each other and recorded in a database (hereinafter “DB”).

By repeating such processing, information (dictionary information) indicating the correspondence between features and labels is accumulated in the DB. The server can identify the action of the nurse based on the acceleration data by referring to the dictionary DB generated in this way.
JP 2009-39466 A

  However, in the background art, for example, the behavior feature labeled “hand washing” includes behavior features (outliers) different from hand washing, and the accuracy of the dictionary DB may be reduced.

  In the background art, the experimenter specifies the time and labels according to the subject, and it has not been possible to perform these afterwards based on the acceleration data.

  Therefore, the main object of the present invention is to provide a novel behavior identification system.

  Another object of the present invention is to provide an action identification system that can delete outliers from a DB such as a dictionary interactively (in accordance with an instruction by presenting information to an experimenter).

  Another object of the present invention is to provide an action identification system capable of performing label addition based on sensor information such as acceleration and registration of the added label in a DB afterwards and conversationally.

  The present invention employs the following configuration in order to solve the above problems. Note that reference numerals in parentheses, supplementary explanations, and the like indicate correspondence relationships with embodiments described later to help understanding of the present invention, and do not limit the present invention in any way.

1st invention is the action identification system which identifies a test subject's action based on the sensor information from a sensor, and the time information from a clock, Comprising: Sensor DB which accumulated sensor information and time information, Each action which a test subject can perform A dictionary DB that stores dictionary information indicating features and labels, sensor information and time information are input, and features are extracted from the sensor information. Based on a comparison between the extracted features and the dictionary DB, labels corresponding to the features are extracted. A time corresponding to the selected and extracted feature is specified based on time information, and action identification means for outputting action information indicating the extracted feature, the selected label and the specified time, action output from the action identification means An annotation DB that accumulates the label and time indicated by the information as annotation information, and a sensor data based on the sensor DB Added based annotation information based on the annotation DB at time information on a screen, the first display means for displaying the annotated sensor data screen on the monitor, and the feature indicated by the action information output from the action identification unit first This is performed by an input device with reference to the second display means for displaying on the monitor a first scatter diagram image that is converted into one vector and indicating the scatter state of each first vector, and the first scatter diagram image displayed on the monitor. and a first receiving means for receiving the outliers delete operation, the features indicated by the action information output from the action identification unit, the labels and time, depending on the outliers delete operation, deletes from the dictionary DB and annotation DB, Conversational deletion means is provided.

In the first invention, the action identification system (100) includes a sensor DB (16a), a dictionary DB (16c), an action identification means (14), an annotation DB (16b), a first display means (20), and conversational deletion. Means (18, S1-S39, S51-S71) are provided. The sensor DB stores sensor information from the sensor (12) and time information from the clock (12T). Dictionary DB accumulate | stores the dictionary information which shows the characteristic and label about each action which a test subject can perform. The behavior identification means receives sensor information and time information, extracts a feature from the sensor information, selects a label corresponding to the feature based on a comparison between the extracted feature and the dictionary DB, and corresponds to the extracted feature The time is specified based on the time information, and action information indicating the extracted feature, the selected label, and the specified time is output. The annotation DB accumulates the label and time indicated by the behavior information output from the behavior identification means as annotation information. The first display means adds annotation information based on the annotation DB to the sensor data screen based on the sensor DB based on the time information, and displays the sensor data screen with annotation (FIG. 2) on the monitor (52). The conversational deletion means converts the feature indicated by the action information output from the action identification means into a first vector, and shows a first scatter diagram image (W1: FIG. 5 (A), FIG. 5 (B), FIG. 6 (A), FIG. 6 (B)) on the monitor, and an input device (S5 to S19) with reference to the first scatter diagram image displayed on the monitor. and a first receiving means for receiving the outlier delete operation to be performed (S21, S27) at 54), the features shown behavioral information output from the action identification unit, the labels and time, in accordance with the outliers removed operation Delete from the dictionary DB and annotation DB.

  According to the first invention, outliers can be deleted conversationally from the dictionary data DB and the annotation DB.

A second invention is an action identification system subordinate to the first invention, wherein the first receiving means performs an outlier selection operation performed on the input device while referring to the first scatter diagram image displayed on the monitor. accepted before outliers delete operation, the first display means, a portion corresponding to the outlier selection operation annotated sensor data screen highlight, outliers deletion further annotated sensor data screen This is done while referring to the highlighted part .

In the second invention, the first accepting means accepts an outlier selection operation performed on the input device with reference to the first scatter diagram image displayed on the monitor before the outlier removal operation, and the first display means The portion corresponding to the outlier selection operation in the annotated sensor data screen is highlighted . The outlier deletion operation is further performed while referring to the highlighted portion in the annotated sensor data screen .

According to the second invention, by highlighting a portion corresponding to the outlier selection operation annotated sensor data screen, the annotation of the experimenter determines whether the first vector is an outlier This can be done based on a comparison between the highlighted portion in the attached sensor data screen and the other portions .

A third invention is an action identification system according to the first invention, wherein the second display means performs an outlier determination process on each first vector, and the determination result deviates from the first vector indicating the outlier. Highlighted as a value candidate (FIGS. 6A and 6B).

  According to the third aspect, by highlighting outlier candidates, the experimenter can easily determine whether or not each first vector is an outlier.

  A fourth invention is an action identification system subordinate to the third invention, wherein the second display means further determines the presence or absence of temporal proximity for each first vector, and the determination result is temporal proximity. The highlight display mode is made different between the first vector indicating presence and the first vector indicating no temporal proximity (FIG. 6B).

  According to the fourth invention, since the highlight display mode (color, brightness, blinking cycle, shape, size, etc.) varies depending on the presence or absence of temporal proximity, the experimenter determines that each first vector is an outlier. This makes it easier to determine whether or not.

A fifth invention is an action identification system according to any one of the first to fourth inventions, wherein the feature indicated by the action information output from the action identification means is converted into a second vector and each second vector is converted. And a third display means for displaying on the monitor a second scatter diagram image showing the scatter state of each second vector together with the clustering information, and the input device while referring to the second scatter diagram image displayed on the monitor. and a second receiving means for receiving the label application operation dividing characteristic indicated by the action information output from the action identification unit, the labels and time, in accordance with the label application operation, is added to the dictionary DB and annotation DB, conversations The method further includes an additional means.

In 5th invention, an action identification system is further provided with a conversational addition means (22, S81-S119). The conversational adding means converts the feature indicated by the action information output from the action identifying means into a second vector, clusters each second vector, and shows a scatter state of each second vector together with clustering information. Third display means (S85 to S99) for displaying the image (FIG. 7) on the monitor, and second reception means for receiving the label addition operation performed on the input device while referring to the second scatter diagram image displayed on the monitor ( S101, S107) and a characteristic indicated by the action information output from the action identification unit, the labels and time, in accordance with the label application operation, is added to the dictionary DB and annotation DB.

  According to the fifth invention, label addition based on sensor information and registration of the added label in the annotation DB and dictionary DB can be performed afterwards and conversationally.

A sixth invention is an action identification system according to the fifth invention, wherein the conversational adding means further includes fourth display means for displaying a label candidate image relating to a cluster included in the second scatter diagram image on the monitor. In addition, the label addition operation is further performed with reference to the label candidate image.

In the sixth invention, conversational additional means further includes a fourth display hand stage. The fourth display means displays the label candidate image (W 3 : FIG. 8 ) on the monitor. The label addition operation is further performed with reference to the label candidate image.

According to the sixth invention, the label candidates for the cluster in the second scatterplot image by table Shimesuru, the experimenter can easily impart label.

The seventh invention is an action identification system subordinate to the fifth invention, wherein the third display means further determines the presence or absence of temporal proximity for each cluster, and the determination result indicates that there is temporal proximity. The indicated cluster is highlighted as a label addition candidate.

  According to the seventh aspect, since the cluster having temporal proximity is highlighted as a label addition candidate, the experimenter can easily recognize the cluster corresponding to the action to which the label is to be added.

An eighth invention is a behavior identification system for identifying a subject's behavior based on sensor information from a sensor and time information from a clock, the sensor DB storing sensor information and time information, and each behavior that a subject can perform A dictionary DB that stores dictionary information indicating features and labels, sensor information and time information are input, and features are extracted from the sensor information. Based on a comparison between the extracted features and the dictionary DB, labels corresponding to the features are extracted. A time corresponding to the selected and extracted feature is specified based on time information, and action identification means for outputting action information indicating the extracted feature, the selected label and the specified time, action output from the action identification means An annotation DB that accumulates the label and time indicated by the information as annotation information, and a sensor data based on the sensor DB Added based annotation information based on the annotation DB at time information on a screen, and displays the annotated sensor data screen on the monitor, the first display means, and the characteristics indicated by the action information output from the action identification unit A second display means for clustering each vector by converting it into a vector and displaying a scatter diagram image showing the scatter state of each vector together with clustering information on the monitor; and an input device with reference to the scatter diagram image displayed on the monitor and a receiving means for receiving the label application operation to be performed, the features indicated by the action information output from the action identification unit, the labels and time, in accordance with the label application operation, is added to the dictionary DB and annotation DB, conversational Additional means are provided.

In the eighth invention, the behavior identification system (100) includes a sensor DB (16a), a dictionary DB (16c), a behavior identification means (14), an annotation DB (16b), a first display means (20), and conversational addition. Means (22, S81 to S119) are provided. The sensor DB stores sensor information from the sensor (12) and time information from the clock (12T). Dictionary DB accumulate | stores the dictionary information which shows the characteristic and label about each action which a test subject can perform. The behavior identification means receives sensor information and time information, extracts a feature from the sensor information, selects a label corresponding to the feature based on a comparison between the extracted feature and the dictionary DB, and corresponds to the extracted feature The time is specified based on the time information, and action information indicating the extracted feature, the selected label, and the specified time is output. The annotation DB accumulates the label and time indicated by the behavior information output from the behavior identification means as annotation information. The first display means adds annotation information based on the annotation DB to the sensor data screen based on the sensor DB based on the time information, and displays the sensor data screen with annotation (FIG. 2) on the monitor (52). The conversational addition means converts the feature indicated by the action information output from the action identification means into a vector, clusters each vector, and uses a scatter diagram image (FIG. 7) showing the scatter state of each vector together with clustering information on the monitor. A second display means (S85 to S99) for displaying, and a receiving means (S101, S107) for receiving a label addition operation performed by the input device (54) while referring to the scatter diagram image displayed on the monitor. features indicated by action information output from the identification means, the labels and time, in accordance with the label application operation, is added to the dictionary DB and annotation DB.

  According to the eighth invention, label addition based on sensor information and registration of the added label in the annotation DB and dictionary DB can be performed afterwards and conversationally.

A ninth invention is a behavioral identification system that depends on the eighth invention, conversational additional means further includes a third display means for displaying the label candidate image on the clusters included in the scatterplot image on the monitor The label addition operation is further performed with reference to the label candidate image.

In the ninth invention, conversational additional means further includes a third display hand stage. Third display means displays the label candidate image on the clusters included in the scatterplot image image (FIG. 8) on the monitor. The label addition operation is further performed with reference to the label candidate image.

According to the ninth invention, the label candidates for clusters included in scatterplot image by Table Shimesuru, the experimenter can easily impart label.

The tenth invention is an action identification system according to the eighth invention, wherein the second display means further determines the presence or absence of temporal proximity for each cluster, and the determination result indicates that there is temporal proximity. The indicated cluster is highlighted as a label addition candidate.

  According to the tenth invention, the cluster having temporal proximity is highlighted as the label addition candidate, so that the experimenter can easily recognize the cluster corresponding to the action to which the label should be added.

  An eleventh invention is a behavior identification system according to any one of the first to tenth inventions, wherein the video DB stores video image information from a video camera and time information from a second clock synchronized with the clock. The first display means further displays the video image based on the video image information and the time information on the monitor, and the outlier deletion operation and / or the label addition operation is further performed while referring to the video image.

  In the eleventh aspect, the behavior identification system further includes a video DB (16d). The video DB stores video image information from the video camera (24) and time information from the second clock (24T) synchronized with the clock. The first display means further displays the video image (FIG. 4) based on the video image information and the time information on the monitor, and the outlier deletion operation and / or the label addition operation is further performed with reference to the video image.

  According to the eleventh aspect, the experimenter can more easily determine the outlier removal and the label addition by referring to the video image.

  According to this invention, the action identification system which can delete an outlier conversationally from DB is implement | achieved. In addition, an action identification system is realized in which label addition based on sensor information and DB registration of the added label can be performed afterwards and conversationally. As a result, a highly accurate DB can be easily constructed, and the identification accuracy of the action identification system can be increased.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the structure of the action identification system which is one Example of this invention. It is an illustration figure which shows an example of a sensor data (acceleration data) viewer screen. It is an illustration figure which shows an example of an annotation data viewer screen. It is an illustration figure which shows an example of a video image viewer screen. (A) is an illustration figure which shows an example of a scatter diagram window, (B) is an illustration figure which shows another example of a scatter diagram window. (A) is an illustrative view showing another example of the scatter diagram window, (B) is an illustrative view showing still another example of the scatter diagram window. It is an illustration figure which shows another example of a scatter diagram window. It is an illustration figure which shows an example of a label candidate window. It is a block diagram which shows the structural example of PC. It is an illustration figure which shows the memory map of the main memory of PC. It is a flowchart which shows a part of CPU operation | movement of PC. It is a flowchart which shows a part of other CPU operation | movement of PC. It is a flowchart which shows the other part of CPU operation | movement of PC. FIG. 10 is a flowchart showing yet another part of the CPU operation of the PC. It is a flowchart which shows a part of other CPU operation | movement of PC.

  Referring to FIG. 1, an action identification system 100 (hereinafter “system 100”) according to an embodiment of the present invention includes a PC 10, a sensor 12, and a video camera 24. The sensor 12 and the video camera 24 are provided with clocks 12T and 24T, respectively. The clocks 12T and 24T are synchronized with a clock 10T on the PC 10 side (see FIG. 9: described later). The sensor 12 and the video camera 24 are further provided with wireless LAN modules 12L and 24L. The wireless LAN modules 12L and 24L communicate with the wireless LAN module 10L on the PC 10 side (see FIG. 9: described later). I do.

  The sensor 12 includes, for example, four three-axis acceleration sensors, and each three-axis acceleration sensor is attached to the left upper arm, upper right arm, chest and waist of a subject (nurse or the like). From the sensor 12, a total of 12 component acceleration data based on these four 3-axis acceleration sensors is transmitted to the PC 10 through the wireless LAN module 12L together with a time stamp based on the clock 12T. That is, the sensor data output from the sensor 12 includes 12-component acceleration data and a time stamp.

  The video camera 24 is installed in the vicinity of a sink where the subject performs predetermined actions such as hand washing. From the video camera 24, video data that captures the vicinity of the installation location is transmitted to the PC 10 through the wireless LAN module 24L together with a time stamp based on the clock 24T. In other words, the video data output from the video camera 24 also includes a time stamp.

  The configuration of the system 100 shown here is merely an example, and various changes can be made. For example, an infrared sensor for position measurement or a proximity sensor (such as a Bluetooth (registered trademark) module) for specifying a proximity object (patient, nursing equipment, etc.) of the subject as a component of the sensor 12 may be added. Good. Further, for each connection between the PC 10 and the sensor 12 and the video camera 24, a short-range wireless communication method other than the wireless LAN such as Bluetooth or ZigBee (registered trademark) may be used. A wired connection may be used between the PC 10 and the video camera 24.

  The PC 10 includes a behavior identification engine 14, a data manager 16, a conversational outlier deletion application 18, a data viewer 20, and a conversational label addition application 22. The data manager 16 includes a sensor DB 16a, an annotation DB 16b, a dictionary DB 16c, and a video DB 16d. The functions of these components (14 to 22, 16a to 16d) are the hardware resources shown in FIG. 9, that is, the clock 10T, the wireless LAN module 10L, the CPU 50, the LCD monitor 52, the key input device 54, and the main memory 56. And software resources (a part of which is shown in FIG. 10) stored in the main memory 56.

  The sensor data from the sensor 12 and the video data from the video camera 24 are given to the data manager 16, and the data manager 16 records the given sensor data and video data in the sensor DB 16a and the video DB 16d, respectively. Therefore, time-stamped acceleration data (hereinafter referred to as “acceleration data”) is stored in the sensor DB 16a, and time-stamped video data (hereinafter referred to as “video data”) is stored in the video DB 16d.

  The acceleration data stored in the sensor DB 16a is given to the behavior identification engine 14 through the data manager 16, and the behavior identification engine 14 compares the given acceleration data with the dictionary DB 16c to identify the subject's behavior, and the identification result is obtained. The behavior data shown is returned to the data manager 16. In the behavior data, features, labels, and times (start time and end time) of the recognized behavior are described.

  More specifically, the dictionary DB 16c stores dictionary data indicating features and labels for each action that can be performed by the subject. Here, the feature is a feature of each behavior extracted from the past acceleration data by the behavior identification engine 14, and has, for example, a form of waveform data. On the other hand, the label is identification information given by the experimenter to the extracted feature of each action through the conversational label addition application 22 (described later). For example, characters such as “hand wash” and “hand wipe” are used. It has a data aspect.

  The behavior identification engine 14 performs feature extraction on the given acceleration data, searches the dictionary DB 16c with the extracted feature, and obtains a label corresponding to the feature from the dictionary DB 16c, while calculating the time corresponding to the extracted feature. The time stamp attached to the acceleration data is specified with reference to the time stamp. Then, the extracted feature, the acquired label, and the specified time are output as an identification result (behavior data) regarding given acceleration data.

  The data manager 16 records the label and time included in the behavior data from the behavior identification engine 14 in the annotation DB 16b as annotation data for the acceleration data in the sensor DB 16a. The annotation data and the acceleration data are associated with each other through a time stamp.

  If there is any label and feature registered in the annotation DB 16b that does not exist in the dictionary DB 16c, the data manager 16 also creates dictionary data indicating the label and feature via the interactive label addition application 22 (described later). Add to DB16c.

  The data viewer 20 acquires data from various DBs 16 a to 16 d via the data manager 16 and visualizes the data. Specifically, a sensor data viewer screen as shown in FIG. 2 is displayed mainly based on sensor data from the sensor DB 16a, and an annotation data viewer screen as shown in FIG. 3 is displayed based on annotation data from the annotation DB 16b. Then, the video image viewer screen as shown in FIG. 4 is displayed based on the video image data from the video DB 16d. It is also possible to display the label registered in the dictionary DB 16c as a label candidate window W3 as shown in FIG.

  On the sensor data screen of FIG. 2, the acceleration waveforms of the total 12 components described above are drawn along a common time axis. On the sensor data screen, annotation information based on annotation data from the annotation DB 16b, that is, characters indicating labels, bands and numbers indicating time, and the like are also drawn.

  The annotation data viewer screen of FIG. 3 is a high-level label indicating “time” indicating the start or end of time, and comprehensive actions such as “hand-washing: start”, “hand-washing: end”, and “hand-wiping: start”. “Label 1” and “Label 2”, which is a subordinate label indicating detailed actions such as “hand wetting: start”, “hand wetting: end”, and “hand washing: start”, are listed. In other embodiments, the labels may be hierarchized into three or more layers, or may not be hierarchized at all.

  The video image viewer screen of FIG. 4 displays a video image of a scene where the subject is washing his hands and a number indicating a time stamp. In the label candidate window W3 in FIG. 8, characters indicating a part of the registered label of the dictionary DB 16c are displayed with an indent ("." Or "..") indicating a hierarchy.

  The conversational outlier deletion application 18 interactively removes data corresponding to the outlier from the annotation DB 16b and the dictionary DB 16c through the scatter diagram window W1 as shown in FIGS. Specifically, the scatter diagram window W1 is displayed in response to the user specifying two components of one or more labels and feature vectors regarding the sensor data in the sensor DB 16a. In any of the scatter plot windows W1 in FIG. 5 and FIG. 6, “hand wipe”, “hand wash”, “gloves”, and “disinfection” are designated for the labels. The two components are “1” and “2” in the scatter diagram window W1 in FIG. 5A, and “1” and “3” in FIGS. 5B, 6A, and 6B. Are specified respectively.

  The conversational outlier deletion application 18 acquires sensor data from the sensor DB 16a, and cuts out a portion corresponding to the specified label from the acquired sensor data. Then, a feature extraction process is performed on the extracted sensor data, and points corresponding to the extracted features are displayed in different manners (here, shapes: circles, squares, triangles, etc.) for each label. Plot display inside.

  5 and 6, each point in scatter diagram window W1 forms a cluster for each label. However, if the extracted feature includes an outlier, a point isolated from the cluster appears as shown in FIG. The conversational outlier deletion application 18 highlights a point isolated from the cluster as an outlier candidate so as to distinguish it from each point in the cluster (for example, increase brightness, blink, etc.).

  Such isolated points may or may not have temporal proximity to the cluster, and the conversational outlier deletion application 18 may use each isolated point as shown in FIG. Can be highlighted in a different manner (eg, size, color, etc.) depending on the presence or absence of temporal proximity.

  The outlier candidates are simply highlighted as shown in FIG. 6A, or the outlier candidates are highlighted in different modes (different sizes here) depending on the presence or absence of temporal proximity as shown in FIG. 6B. The display is determined based on the experimenter's selection.

  On the other hand, when the experimenter selects an arbitrary point in the scatter diagram window W1 with the cursor C as shown in FIG. 5A, for example, the various viewer screens shown in FIGS. Updated to be displayed. Note that on the updated sensor data viewer screen and the updated annotation data viewer screen, a portion corresponding to the point (dotted line frame portion in FIGS. 2 and 3) may be highlighted.

  Further, when the experimenter performs a deletion operation on the selected point, data corresponding to the point is deleted from the annotation DB 16b and the dictionary DB 16c, and the sensor data viewer screen and the annotation data viewer screen are further updated accordingly. . That is, “hand finger wash” is deleted from the sensor data viewer screen of FIG. 2, and “09: 02: 20: 800” and “hand finger wash: start” and “09: “02: 21: 700” and “hand cross-washing: end” are deleted.

  Therefore, the experimenter confirms whether or not there is an outlier candidate (isolated point) through the scatter diagram window W1 shown in FIGS. 5 and 6, and if there is an outlier candidate, sensor data or video corresponding to the isolated point is displayed. Check the images on the various viewer screens shown in FIGS. 2 to 4, and if necessary, compare such sensor data and video images between isolated points and points in the cluster. The deletion operation can be performed after determining whether the value is an outlier.

  The interactive label addition application 22 adds new data to the annotation DB 16b and the dictionary DB 16c interactively through the scatter diagram window W2 as shown in FIG. 7 and the label candidate window W3 as shown in FIG. Specifically, the scatter diagram window W2 and the label candidate window W3 are displayed in response to the user specifying two components of one or more labels and feature vectors regarding the sensor data in the sensor DB 16a.

  In the scatter diagram window W2 in FIG. 7, as in FIG. 5 (A), “hand wipe”, “hand wash”, “gloves”, “disinfection” as labels, and “1” and “2” as two components, respectively. The points indicating the features corresponding to these specified labels and components are plotted. Again, each point forms a cluster for each label, but unlike that of FIG. 5A, there are many points (stars) isolated from each cluster. Among these isolated points, those having temporal proximity form a new cluster, and this new “cluster having temporal proximity” is highlighted as a label addition candidate.

  When one of the clusters highlighted as the label addition candidate in the scatter diagram window W2 as shown in FIG. 7 is designated by the experimenter, the interactive label addition application 22 displays the label candidate window as shown in FIG. W3 is displayed. In this label candidate window W3, the designated label (that is, “hand wipe”, “hand wash”,...) And its lower labels (for example, “hand wet”, “hand wash”,..., “Paper towel removal”,...) Are displayed. Is done.

  When the experimenter specifies one of the label candidates displayed in the label candidate window W3 as shown in FIG. 8, the interactive label addition application 22 is included in the cluster previously specified in the scatter diagram window W2. A label designated here is added to a feature corresponding to (i.e., an unknown feature), and a time corresponding to the feature is specified. Then, annotation data indicating these features and time is added to the annotation DB 16b, while dictionary data indicating these features and labels is added to the dictionary DB 16c.

  Among the operations of the system 100 as described above, information processing performed by the PC 10 to realize the conversational outlier deletion application 18 and the conversational label addition application 22 will be described next.

  As shown in FIG. 9, the PC 10 includes a wireless LAN module 10L, a clock 10T, a CPU 50, an LCD monitor 52, a key input device 54, and a main memory 56. The wireless LAN module 10L performs wireless LAN communication with the wireless LAN modules 12L and 24L provided in the sensor 12 and the video camera 24, respectively, under the control of the CPU 50. The timepiece 10T is synchronized with the timepieces 12T and 24T provided in the sensor 12 and the video camera 24, respectively, and gives necessary time information to the CPU 50.

  The LCD monitor 52 displays images as shown in FIGS. 2 to 8 under the control of the CPU 50. The key input device 54 receives the operation of the experimenter and gives a corresponding command to the CPU 50. The main memory 56 stores a program and data for the CPU 50 to perform processing. Specifically, as shown in FIG. 10, a program area 56a and a data area 56b are formed in the main memory 56. The program area 56a has a conversational outlier deletion program 60, a conversational label addition program 62, a display control program 64, and A communication control program 66 and the like are stored.

  The conversational outlier deletion program 60 is a main program for realizing the conversational outlier deletion application 18, and flowcharts corresponding to the main program are shown in FIGS. The conversational label addition program 62 is a main program for realizing the conversational label addition application 22, and flowcharts corresponding to the main program are shown in FIGS. 14 to 15. The display control program 64 is a subprogram used by the conversational outlier deletion program 60 and the conversational label addition program 62, thereby realizing display of images as shown in FIGS. 2 to 8. The communication control program 66 is a subprogram used by the conversational outlier deletion program 60 and the conversational label addition program 62, thereby realizing wireless communication between the PC 10 and the sensor 12 or the video camera 24.

  The data area 56b includes a sensor data area 70, an annotation data area 72, a dictionary data area 74, a video image data area 76, a variable / parameter area 78, and the like. The various data areas 70 to 78 are areas for storing various data read from the various DBs 16a to 16d. The variable / parameter area 74 includes various variables (feature vector X, time vector T, canonical vector C, principal component vector P,...) And parameters (class, α, Flag, dt, param,...) Used in the program. ) Is an area for storing.

  Referring to FIG. 11, when conversational outlier deletion program 60 is activated, CPU 50 first reads various data from various DBs 16a to 16d into various data areas 70 to 78 in step S1, and in step S3. Various viewer screens as shown in FIGS. 2 to 4 are displayed on the LCD monitor 52 based on the various data.

  Next, in step S5, an operation for designating two components of a label and a feature vector is accepted through the key input device 54. In step S7, feature extraction processing is performed on the designated label and sensor data, and the feature vector X and A time vector T is obtained. In step S9, a canonical correlation conversion process is performed on the feature vector X to obtain a canonical vector C. In step S11, the specified two components (corresponding points) of the obtained canonical vector C are plotted and displayed in the scatter diagram window W1 (see FIGS. 5 and 6).

  Next, in step S13, parameters are set based on the operation of the key input device 54. As a result, the target class is “class”, the outlier coefficient is “α”, the temporal proximity consideration flag is “Flag”, and the threshold value for temporal proximity is “dt”. Is set. In step S15, outlier processing (see FIG. 13: described later) is performed based on the canonical vector C, the time vector T, and the parameters (class, α, Flag, dt) to obtain “Outlier” and “Continuity”. Here, “Outlier” is an array of canonical vectors that are outliers, and “Continuity” is an array related to the presence / absence of temporal proximity of outliers (True / False). In step S17, “Outlier” is classified into “Continuity” that is True (“Outlier_cont”) and False (“Outlier_discont”). In step S19, “Outlier_cont” and “Outlier_discount” are highlighted in different modes (size and color).

  Next, in step S21, an operation for designating any point in the scatter diagram window W1 is accepted through the key input device 54 (see FIG. 5A). In step S23, the time corresponding to the designated point is set. calculate. In step S25, various viewer screens (see FIGS. 2 to 4) are updated and displayed at the calculated times. The time interval may be highlighted on the sensor data viewer screen and the annotation data viewer screen. In step S27, an outlier deletion operation for the previously designated point is accepted through the key input device 54. In step S29, it is determined whether or not an outlier deletion operation has been performed. If NO, the process returns to step S13 to repeat the same processing.

  If “YES” in the step S29, the canonical vector C and the feature vector X corresponding to the point are deleted in a step S31. In step S33, the scatter diagram window W1 is updated and displayed. In step S35, the annotation DB 16b and the dictionary DB 16c are updated. In step S37, various viewer screens are updated and displayed. In step S39, it is determined whether or not to end. If NO, the process returns to step S5 to repeat the same processing. When the end operation is performed by the key input device 54, it is determined YES in step S39, and this process is ended.

  The outlier processing in step S15 is executed according to the subroutine of FIG. In step S51, the canonical sample vector C * = C_s. Corresponding to the sample specified by “class” in the canonical vector C. t. For (class level = class), the average vector μi (1 ≦ i ≦ d) and the diagonal component σi (1 ≦ i ≦ d) of the covariance matrix, that is, [σ1, σ2,..., Σd] = diag (Cov (C *)). Further, the number of samples of C * (that is, the number of samples having the label designated by “class” in the canonical vector C) is obtained, and this is set to “K”. Further, the average of the times of the sample vectors in the vicinity of the average vector μi is obtained, and this is set as “T *”.

  In step S53, the counter is initialized (k = 1). In step S55, it is determined whether or not processing has been completed for all data (k> K?). If “YES” here, the process returns to the upper layer routine, and if “NO”, the process proceeds to a step S57. In step S57, the difference Di from the average vector μi is calculated for each element C * (k) i of the canonical sample vector C * (k) (Di = | C * (k) i−μi |). In step S59, it is determined whether or not there is an element satisfying Di> α × σi in the calculation result. If YES (if any), the process proceeds to step S71. If NO (NO), there is no one. Eg) Proceed to step S61.

  In step S61, k is added to “Outlier”. In step S63, it is determined by “Flag” whether or not the temporal proximity is considered. If Flag = True, the determination is YES and the process proceeds to step S71. If Flag = False, NO is determined and the process proceeds to step S65.

  In step S65, it is determined whether or not the absolute value difference between the time T (k) and the time T * exceeds the previously specified threshold, that is, “dt” (| T (k) −T * |> dt?). To do. If “YES” here, after setting “True” to “Continuity [k]” in a step S67, the process proceeds to a step S71. On the other hand, if “NO”, “False” is set in “Continuity [k]” in step S69, and then the process proceeds to step S71.

  In step S71, the counter is incremented (k = k + 1), and then the process returns to step S55 to repeat the same processing.

  Referring to FIG. 14, when conversational label addition program 62 is activated, CPU 50 first reads various data from various DBs 16a to 16d into various data areas 70 to 78 in step S81, and in step S83, the data is read. Various viewer screens as shown in FIGS. 2 to 4 are displayed on the LCD monitor 52 based on the various data.

  Next, in step S85, an operation for designating two components of a label and a feature vector is accepted through the key input device 54. In step S87, feature extraction processing is performed on the designated label and sensor data, and the feature vector X A time vector T is obtained. In step S89, the principal component analysis process is performed on the feature vector X to obtain the principal component vector P. In step S91, the designated two components (corresponding points) of the obtained principal component vector P are plotted and displayed in the scatter diagram window W2 (see FIG. 7).

  Next, in step S93, parameters are set based on the operation of the key input device 54. Accordingly, a clustering parameter is set in “param”, a temporal proximity consideration flag is set in “Flag”, and a threshold value related to temporal proximity is set in “dt”. In step S95, clustering processing is performed based on the principal component vector C, the time vector T, and the parameters (param, Flag, dt) to obtain “Cluster” and “Continuity”. Here, “Cluster” is an array of clusters as a result of the clustering process, and “Continuity” is an array related to the presence / absence of cluster proximity (True / False). For the clustering process, a clustering method based on an EM (Expectation Maximization) algorithm assuming a multivariate mixed normal distribution, a K-means (K-means) which is a standard clustering method, or the like can be used.

  In step S97, “Cluster” is classified into “Continuity” with True (“Cluster_cont”) and False (“Cluster_discount”). In step S99, “Cluster_cont” and “Outlier_discount” are highlighted in different modes (size and color).

  Next, in step S101, an operation of the cursor C for designating any cluster in the scatter diagram window W2 is accepted through the key input device 54 (see FIG. 7), and in step S103, a time interval corresponding to the designated cluster. Is calculated. In step S105, various viewer screens (see FIGS. 2 to 4) are updated and displayed in the calculated time section. The time interval may be highlighted on the sensor data viewer screen and the annotation data viewer screen. In step S107, a label candidate window W3 as shown in FIG. 8 is displayed on the LCD monitor 52, and a label assignment operation for the previously designated cluster is accepted through the key input device. Specifically, the experimenter selects one of the labels shown in the label candidate window W3 by a cursor operation, or inputs a new label not in the label candidate window W3 by a character key operation. In step S109, the presence / absence of a labeling operation is determined. If NO, the process returns to step S93 to repeat the same processing.

  If “YES” in the step S109, a label is added to the principal component vector P corresponding to the cluster and the feature vector X in a step S111. In step S113, the scatter diagram window W2 is updated and displayed. In step S115, the annotation DB 16b is updated. In step S117, various viewer screens are updated and displayed. In step S119, it is determined whether or not to end. If NO, the process returns to step S85 to repeat the same processing. When the end operation is performed by the key input device 54, YES is determined in the step S119, and this process is ended.

  As is clear from the above, the behavior identification system 100 of this embodiment includes a PC 10, which is a sensor DB 16a, a dictionary DB 16c, a behavior identification engine 14, an annotation DB 16b, a conversational outlier deletion application 18, a data viewer 20, and a conversational label. Each function of the additional application 22 is realized. The sensor DB 16a stores sensor information from the sensor 12 and time information from the clock 12T. The dictionary DB 16c stores dictionary information indicating features and labels for each action that can be performed by the subject. The behavior identification engine 14 receives sensor information and time information, extracts a feature from the sensor information, selects a label corresponding to the feature based on a comparison between the extracted feature and the dictionary DB 16c, and corresponds to the extracted feature. The time to perform is specified based on the time information, and the extracted feature, the selected label, and the action information indicating the specified time are output. The annotation DB 16b accumulates the label and time indicated by the behavior information output from the behavior identification engine as annotation information. The data viewer 20 adds annotation information based on the annotation DB 16 b to the sensor data screen based on the sensor DB 16 a based on the time information, and displays the sensor data screen with annotation (FIG. 2) on the LCD monitor 52.

  Then, the conversational deletion application 18 performs the feature, label, and time indicated by the behavior information output from the behavior identification engine 14 by using the key input device 54 while referring to the annotated sensor data screen displayed on the LCD monitor 52. In response to the outlier deletion operation, the dictionary DB 16c and the annotation DB 16b are deleted (S1 to S39, S51 to S71). In this way, outliers can be deleted conversationally from the sensor DB 16a and the annotation DB 16b.

  On the other hand, the conversational label addition application 22 uses the key input device 54 while referring to the annotated sensor data screen displayed on the LCD monitor 52 for the features, labels, and times indicated by the behavior information output from the behavior identification engine 14. Depending on the label addition operation to be performed, it is added to the dictionary DB 16c and the annotation DB 16b (S81 to S119). In this way, label addition based on sensor information and registration of the added label in the sensor DB 16a and annotation DB 16b can be performed afterwards and conversationally.

  In this embodiment, the sensor information stored in the sensor DB 16a is input to the behavior identification engine 14, but the sensor information output from the sensor 12 may be input in real time.

  In this embodiment, four three-axis acceleration sensors are used as the sensor 12, but in addition to this, a position sensor using an infrared transmitter / receiver or a proximity sensor using a Bluetooth (registered trademark) module or the like is used. It may be used. In this case, the timepieces of the position sensor and the proximity sensor are also synchronized with the timepiece 10T of the PC 10, and the sensor DB 16a accumulates the position information from the position sensor and the proximity information from the proximity sensor together with the time information. The action identification engine 14 performs action identification based on acceleration information, referring to position information and proximity information as appropriate.

  In the above, an example of application to nursing work has been described. However, the present invention can be applied to other work such as manufacturing and distribution.

10 ... PC
12 ... Sensor (3-axis acceleration sensor x 4)
DESCRIPTION OF SYMBOLS 14 ... Action identification engine 16 ... Data manager 18 ... Conversational outlier deletion application 20 ... Data viewer 22 ... Conversational label addition application 24 ... Video camera 50 ... CPU
52 ... LCD monitor 54 ... Key input device 56 ... Main memory 10T, 12T, 24T ... Clock 16a ... Sensor DB
16b Annotation DB
16c ... Dictionary DB
16d ... Video DB
100 ... Action identification system

Claims (11)

A behavior identification system for identifying a subject's behavior based on sensor information from a sensor and time information from a clock,
A sensor DB that stores the sensor information and the time information;
Dictionary DB that stores dictionary information indicating features and labels for each action that can be performed by the subject,
The sensor information and the time information are input, a feature is extracted from the sensor information, a label corresponding to the feature is selected based on a comparison between the extracted feature and the dictionary DB, and a time corresponding to the extracted feature is determined. Action identification means for specifying the extracted time based on the time information and outputting action information indicating the extracted feature, the selected label and the specified time;
An annotation DB that accumulates the label and time indicated by the behavior information output from the behavior identification means as annotation information,
First display means for adding annotation information based on the annotation DB to the sensor data screen based on the sensor DB based on the time information, and displaying the sensor data screen with annotation on a monitor; and
Second display means for converting the characteristics indicated by the action information output from the action identification means into a first vector and displaying on the monitor a first scatter diagram image showing a scatter state of each of the first vectors; and the monitor A first accepting unit that accepts an outlier deletion operation performed by the input device with reference to the first scatter diagram image displayed on the screen , and the feature, label, and time indicated by the behavior information output from the behavior identifying unit in response to the outlier delete operation, it deletes from the dictionary DB and the annotation DB, comprises a conversational deletion means, behavioral identification system. The first accepting unit accepts an outlier selection operation performed on the input device with reference to the first scatter diagram image displayed on the monitor before the outlier removal operation,
The first display means highlights a portion corresponding to the outlier selection operation in the annotated sensor data screen,
Before Kihazure value delete operation is carried out with reference also more highlighted portions of the annotated sensor data screen, behavior identification system of claim 1, wherein. The second display means, highlighting a first vector indicating a determination result outlier subjected to outlier determination processing to each of said first vector as an outlier candidate, behavior identification system of claim 1, wherein.   The second display means further determines the presence or absence of temporal proximity for each of the first vectors, and a first vector indicating that the determination result is temporal proximity and a first vector indicating no temporal proximity; The behavior identification system according to claim 3, wherein the highlight display is made different. The second scatter diagram image showing the scatter state of each second vector together with clustering information is converted into a second vector by converting the feature indicated by the behavior information output from the behavior identification means into a second vector. 3rd display means to display on a monitor, and 2nd reception means to receive the label addition operation performed with the said input device, referring the 2nd scatter diagram image displayed on the said monitor, It outputs from the said action identification means 5. The apparatus according to claim 1, further comprising a conversation addition unit that adds the feature, the label, and the time indicated by the action information to be added to the dictionary DB and the annotation DB according to the label addition operation. Behavior identification system. The interactive addition means further includes fourth display means for displaying on the monitor a label candidate image relating to a cluster included in the second scatter diagram image, and the label addition operation is performed while further referring to the label candidate image. The behavior identification system according to claim 5. 6. The behavior according to claim 5 , wherein the third display means further determines whether or not there is temporal proximity for each of the clusters, and highlights a cluster indicating that the determination result indicates temporal proximity as a label addition candidate. Identification system. A behavior identification system for identifying a subject's behavior based on sensor information from a sensor and time information from a clock,
A sensor DB that stores the sensor information and the time information;
Dictionary DB that stores dictionary information indicating features and labels for each action that can be performed by the subject,
The sensor information and the time information are input, a feature is extracted from the sensor information, a label corresponding to the feature is selected based on a comparison between the extracted feature and the dictionary DB, and a time corresponding to the extracted feature is determined. Action identification means for specifying the extracted time based on the time information and outputting action information indicating the extracted feature, the selected label and the specified time;
An annotation DB that accumulates the label and time indicated by the behavior information output from the behavior identification means as annotation information,
First display means for adding annotation information based on the annotation DB to the sensor data screen based on the sensor DB based on the time information and displaying the sensor data screen with annotation on a monitor; and
Second display means for converting the feature indicated by the action information output from the action identifying means into a vector, clustering each vector, and displaying a scatter diagram image showing the scatter state of each vector together with clustering information on the monitor And receiving means for receiving a label addition operation performed by an input device while referring to a scatter diagram image displayed on the monitor, and the feature, label, and time indicated by the action information output from the action identifying means, An action identification system comprising conversational addition means for adding to the dictionary DB and the annotation DB in accordance with the label addition operation. The conversational additional means further includes a third display means for displaying the label candidate image on the clusters included in the scatter diagram image on the monitor, the label application operation yet the label candidate image is also carried out with reference to, The behavior identification system according to claim 8. 9. The behavior according to claim 8 , wherein the second display means further determines the presence or absence of temporal proximity for each of the clusters, and highlights the cluster indicating that the determination result indicates temporal proximity as a label addition candidate. Identification system. A video DB that stores video image information from a video camera and time information from a second clock synchronized with the clock;
The first display means further displays a video image based on the video image information and the time information on the monitor, and the outlier deletion operation and / or the label addition operation is further performed with reference to the video image. The action identification system according to any one of claims 1 to 10.

Images