JP2019046304A - Event series extraction apparatus, event series extraction method, and event extraction program - Google Patents

Abstract

To provide an event series extraction apparatus capable of extracting event series on the basis of a sensor signal so as to generalize it to unknown places, while taking account of contexts of an event.SOLUTION: An event series extraction apparatus 1 stores data about a subject 2 in a storage unit 40, with a behavior label, an object label, and a position label defined as a time line and extracts event series. The event series extraction apparatus 1 extracts a most likelihood combination, as an event, out of a plurality of candidates of combinations of the object label and position label in a prescribed time window with the behavior label used as a standard, according to the most likelihood that the behavior label, object label, and position label corresponding to the candidates may be present in a language corpus.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for extracting an event sequence from information of a subject's perceptual experience.

  With the advancement of information communication technology, a technology has been proposed in which meta information (typically, a label) is added and classified to various data generated as each individual lives (for example, See Patent Document 1, Non-Patent Document 1, etc.). Among various pieces of data collected in this manner, search and extraction are performed on target data using the provided meta information as a keyword.

  That is, automatically recognizing one's current state and surrounding circumstances is an important task in understanding human daily life. Such technology is called "action recognition", and its purpose is to detect and identify human behavior patterns based on a wide variety of sensor signals. With the development of sensor technology, the technology of behavior recognition is applied in a wide range of fields such as monitoring of Alzheimer's disease patients, detection of behavior pattern in smart home, nurse's behavior recognition for medical improvement, life log .

  In many conventional technologies, the problem of classifying time-series signals acquired from sensors into labels of basic actions (daily life actions) performed in performing life such as "walk", "eat", "drink" and "read" For example, Non-Patent Document 2).

  However, as our daily life "make coffee in the kitchen, move to the living room and then drink coffee in the living room", we work on various objects in various places and perform daily life actions sequentially. . In order to recognize such a series of actions representing who, when, where, and what in detail, in addition to recognizing individual action labels, a plurality of semantic concepts such as actions, objects, places, etc. It is necessary to recognize “events” that are combinations and to correctly recognize a series of temporally consecutive events.

JP, 2017-58729, A

J. Gemmell, G. Bell, and R. Lueder, MyLifeBits: A Personal Data atabase for Everything, Communications of the ACM, 49 (1): 88-95, 2006.

Ord'o "nez, F. J., and Roggen, D." Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. "Sensors 16 (1): 115, 2016.

  When trying to execute a method of generating an event sequence (event timeline) relating to daily life actions from a large number of semantic concepts estimated from semantic signals estimated from sensor signals, there are the following problems.

  In other words, each meaning concept is an action such as "take off", "wear", "walk" or "sit" observed in a real environment, an object such as "shoes", "slippers", "sofa" or "TV", "door entrance" "living room" Represents the location where the observed start and end times are given. However, in a real environment, it is difficult to generate events by combining concepts on a rule basis, since multiple semantic concepts (concept streams) appear on the time axis, and the observed times of these semantic concepts do not match. It is. Also, predicting a semantic concept from a sensor signal does not mean that only the correct semantic concept is predicted, so it is necessary to select only appropriate semantic concepts. Furthermore, researches for providing language to real-world behavior, which have been proposed conventionally, have been conducted in a limited space where the place does not change. However, since events occur sequentially in various places in daily life, it is necessary to create an event timeline so as to generalize to unknown places while considering the context of the events.

  The present invention has been made to solve the above-mentioned problems, and its object is to provide an event timeline (event sequence) so as to generalize to an unknown place while considering the context of the event. An event sequence extraction device, an event sequence extraction method, and an event extraction program capable of extracting the signal sequence based on a sensor signal.

  According to one aspect of the present invention, there is provided an event sequence extraction device, a first sensor for acquiring data for identifying an object of a subject's perceptual experience, and a subject's perceptual experience when the subject has a perceptual experience. A second sensor for acquiring position information for identifying the action and the position of the subject, a storage device for storing the information, and for identifying each of the action, the object, and the position of the subject. The action label, the object label, and the position label are stored in the storage device as a timeline, and the event sequence extraction unit extracts an event sequence. The event sequence extraction unit includes an activity label and an object label for each predetermined sliding window. Concept estimation processing means that predicts position labels and combines and arranges identical labels on the timeline Candidates for combinations of object labels and position labels within a predetermined time window with reference to the action label, for the combined result of the action label, the object label and the position label on the timeline, candidates in the language corpus Language model contrasting means for extracting the most likely combination as an event according to the likelihood that the action label, the object label and the position label corresponding to the event exist, and the event for extracting an event sequence from the time series of the extracted event And sequence selection means.

  Preferably, the language model contrast means uses a likely combination as an event according to the likelihood existing in the language corpus and the likelihood due to the closeness of time between the combined action label, the object label and the position label. Extract.

  Preferably, the concept estimation processing means includes a gated recursive unit that predicts activity labels, object labels and position labels using sequence labeling for the first and second sensor signals in each sliding window.

  Preferably, the language model comparison means includes a gated recursive unit which receives as input a word embedding vector in consideration of word similarity.

  Preferably, the event sequence selection means extracts an event sequence according to a hidden state model in which the dissimilarity of the vector representing the event is the transition probability of the event.

According to another aspect of the present invention, there is provided an event sequence extraction method for extracting information on a sequence of events generated in a subject from sensor data in a real environment, which identifies an object of perceptual experience of the subject. Acquiring data of the subject by the first sensor; acquiring position information for identifying the subject's behavior and the subject's position by the second sensor at the time of the subject's perceptual experience; and predetermined sliding Predicting, for each window, an action label, an object label and a position label for identifying each of the action, the object, and the position, and combining and arranging the same labels on the timeline; Storing the action label, the object label and the position label as a timeline for the subject in the storage device;
Predicting an action label, an object label and a position label for each predetermined sliding window, and performing a process of combining and arranging the same label on the timeline; an action label, an object label and a position label on the timeline The existence of action labels, object labels and position labels corresponding to the candidates in the language corpus for a plurality of combinations of object labels and position labels within a predetermined time window relative to the action labels, for the combined result of According to the likelihood, the step of extracting the most likely combination as an event, and the step of extracting an event sequence from the extracted time series of events are provided.

According to still another aspect of the present invention, an event sequence extraction program for causing a computer to execute an event sequence extraction process for extracting information on a sequence of events generated in a subject from sensor data in a real environment. The event sequence extraction program causes the computer to acquire data for identifying an object of the subject's perceptual experience with the first sensor, and the subject to perform the subject's perceptual experience with the second sensor. Acquiring position information for specifying the behavior of the subject and the position of the subject, an action label, an object label and a position label for identifying each of the action, the object and the position for each predetermined sliding window Execute the process of combining and arranging identical labels on the timeline by predicting And storing the action label, the object label and the position label in the storage device as a timeline for the subject, and the action label on the basis of the combination result of the action label, the object label and the position label on the timeline. For a plurality of combinations of object labels and position labels within a predetermined time window, the most likely combination is determined according to the likelihood of the action label, object label and position label corresponding to the candidate in the language corpus A step of extracting as an event and a step of extracting an event sequence from the extracted time series of events are executed.
(Significance of terms)
In the present specification, the meanings of the terms are as follows.

  1) The "action label" refers to the term when the subject's action corresponding to the sensor signal is associated with the "defined term indicating the action" based on the sensor signal worn by the subject. Say.

  2) "Semantic concept" refers to information identifying the subject's behavior identified by the subject's action label, an object recognized as a subject related to the subject's action, and the place where the subject is present. Action labels such as "take off", "wear", "walk" and "sit" observed in a real environment, objects as subjects related to actions such as "shoes", "slippers", "sofa" and "TV", "door entrance" "living room" And the observed start and end times.

  3) "Concept stream" means a plurality of semantic concepts aligned on the time axis.

  4) "Event" refers to one that consists of multiple semantic concepts and is recognized in everyday life as having a single meaning as a series of actions of the subject.

  According to the event sequence extraction apparatus, the event sequence extraction method, and the event extraction program of the present invention, it is possible to extract an event timeline (event sequence) based on a sensor signal so as to generalize to an unknown place.

It is a schematic diagram which shows an example of the whole structure of the event sequence extraction apparatus 1 according to this Embodiment. It is a schematic diagram which shows the structure of the wearable camera 10 shown in FIG. It is a schematic diagram which shows the structure of the motion sensor 20 shown in FIG. It is a schematic diagram which shows the structure of the information processing apparatus 50 shown in FIG. It is a figure which shows the example which created the event timeline by combining the semantic concept estimated from the sensor signal. It is a 1st conceptual diagram for demonstrating the procedure which produces | generates an event sequence. It is a 2nd conceptual diagram for demonstrating the procedure which produces | generates an event sequence. It is a 3rd conceptual diagram for demonstrating the procedure which produces | generates an event series. It is a table showing a concept name for each concept type of action, object, and place. It is a conceptual diagram which shows the structure of the RNN language model which used word embedding vector and GRU. It is a conceptual diagram which shows the example of the transition of an event. It is a figure which shows the house plan used by experiment, and 20 site-specific actions which a test subject performed. It is a figure which shows the evaluation result of the extracted event timeline.

Hereinafter, the configuration of an event sequence extraction apparatus, an event sequence extraction method, and an event extraction program according to an embodiment of the present invention will be described with reference to the drawings. In the following embodiments, components and processing steps denoted by the same reference numerals are the same or equivalent, and the description thereof will not be repeated if not required.
Embodiment
In the present specification, the “perceived experience” of the subject may be any perception (for example, vision, hearing, smell, taste, touch) as long as it can be obtained or measured using any device. . That is, any perception may be used as long as it can be stored as some data and reproduced later.

  If it is visual, it can be digitized as a still image or a moving image captured using a camera. Then, the image can be reproduced on a television monitor or the like. If it is a hearing, it can be digitized as an acquired voice. Then, the voice can be reproduced by a speaker or the like. The sense of smell and taste can be converted to data using an odor sensor and a taste sensor, respectively. If it is a tactile sense, it can be data-ized using a contact pressure sensor etc. These perceptions can also be reproduced by using a specific device.

  The event sequence extraction device according to the present embodiment generates a timeline of events related to daily life, taking into consideration at least three characteristics of the subject's behavior, objects, and position in the real world. First of all, by modeling the temporal closeness relationship between the time of human action, the time of observing the object to be acted, and the time of being in the place of action, the time difference of observation of the concept is dealt with Do. Also, in order to find a combination of concepts that is most likely among multiple concepts, we learn combinations of concepts that may occur in the real world from an external language corpus that reflects real-world common sense knowledge. Furthermore, a recurrent neural network is used to obtain an information expression vector of the event, and this vector is used to express sequential changes in the event. Finally, we integrate these models using hidden Markov models to generate plausible event lime lines that represent everyday life.

  The external time change that occurs as the subject's "action" includes a behavior that the subject performed spontaneously or passively. For example, it includes movements of hands, head, body, legs, etc. that occur in daily life. In addition, internal time changes that occur in the subject as "action" include changes that occur spontaneously or passively in the subject's body. For example, it includes changes in blood flow patterns in the brain, changes in body temperature patterns, changes in sweating patterns, and the like.

  In the following description, as one embodiment, an image (including a still image or a moving image) perceived by the subject's vision, an action as an external time change generated in the subject, and position information are associated with each other. A process of storing in the storage device and extracting as an event sequence will be described.

  Therefore, in the following description, the behavior corresponding to the “action” is a concept including a temporally generated change by operating a part of the subject's body. Thus, the perceptual experience in the present embodiment includes an image perceived by the subject's vision, and the temporal change produced in the subject includes information indicating the movement of a part of the subject's body.

[System configuration]
Next, an overall configuration of a search system according to the present embodiment will be illustrated.

  FIG. 1 is a schematic diagram showing an example of the entire configuration of an event sequence extraction device 1 according to the present embodiment. Referring to FIG. 1, event sequence extraction apparatus 1 according to the present embodiment basically has an event sequence extraction function.

  In the event sequence extraction function of the event sequence extraction device 1, the subject 2 wears the wearable camera 10 in order to collect the vision of the subject 2, that is, the image seen by the subject 2. At the same time, in order to collect the behavior of the subject 2, the subject 2 also wears the motion sensor 20. In order to measure the behavior of the subject 2 more accurately, a plurality of motion sensors 20 may be attached. In this case, for example, the subject may be worn on three places, for example, the arms of the subject (for example, the right wrist and the left wrist) and the head.

  The wearable camera 10 sends the captured video data at predetermined intervals or events. Alternatively, the captured video data may be stored in a memory incorporated in the wearable camera 10. The motion sensor 20 sends out positional information of the subject 2 at predetermined intervals along with motion data (for example, acceleration, angular velocity, etc.) which is information indicating the behavior of the subject 2. Alternatively, motion data and position information acquired in a memory incorporated in the motion sensor 20 may be stored for a predetermined period in a predetermined cycle, and these may be collectively transmitted later. In the following description, a case where accelerations for three axes and angular velocities for three axes are used as motion data will be mainly described.

However, since the video data sent from the wearable camera 10 and the motion data sent from the motion sensor 20 need to be synchronized with each other, for example, a time including a time or a counter value output from the same timer A stamp is attached to the data to be collected.

  The event sequence extraction device 1 has, as an event sequence extraction function, an image collection unit 30, an image feature amount calculation unit 32, an exercise information collection unit 34, an exercise feature amount calculation unit 36, a position information collection unit 35, and a position information A feature amount calculation unit 37, an event sequence extraction unit 38, and a data storage unit 40 are included.

  The image collection unit 30 and the motion information collection unit 34 provide the function of acquiring the perceptual experience of the subject 2 and the external or internal temporal change that has occurred during the perceptual experience of the subject 2. That is, the video collecting unit 30 receives the video data transmitted by the wearable camera 10, and the motion information collecting unit 34 receives the motion data transmitted by the motion sensor 20.

  In addition, the position information collecting unit 35 receives position information at the same time as each exercise data.

  The video feature quantity calculating unit 32, the motion feature quantity calculating unit 36, the position information feature calculating unit 37, and the event sequence extracting unit 38 provide a function of generating identification information based on a time change occurring in the subject 2. In the present embodiment, as a preferable form, identification information is generated based on the feature value calculated from the time change occurring in the subject 2.

  More specifically, in order to distinguish the feature amount included in the image data received by the image collection unit 30 (the feature amount included in the movement amount data, the image feature amount calculation unit 32 may Also referred to as Details of the method of calculating the image feature amount and the method of use will be described later. The motion feature quantity calculation unit 36 includes feature quantities included in the motion data received by the motion information collection unit 34 (hereinafter, also referred to as "motion feature quantities" to distinguish from the above-described "image feature quantities"). calculate. Details of the calculation method and usage method of the movement feature quantity will be described later. The position feature quantity calculation unit 37 also refers to the position quantity received by the position information collection unit 35 as a feature quantity (hereinafter, also referred to as “position feature quantity” in order to distinguish it from the “image feature quantity” and “motion feature quantity” described above. ). Details of the calculation method and usage method of the position feature amount will be described later.

  The event sequence extraction unit 38 refers to motion feature quantities calculated for motion data 40.1 from the sensors, and determines an action label 40.2 as identification information to be associated with each motion data 40.1. That is, the event sequence extraction unit 38 uses the action label as information characterizing the behavior performed by the subject 2 when the exercise data 40.1 is acquired based on the movement feature amount calculated by the movement feature amount calculation unit 36. Determine 40.2 and associate the determined action label 40.2 with the corresponding exercise data 40.1. Then, the data storage unit 40 stores a set of exercise data 40.1 and a corresponding action label 40.2. That is, the data storage unit 40 provides a function of associating and storing sensor information characterizing the exercise of the subject 2 and information characterizing the corresponding action.

  The event sequence extraction unit 38 determines an object label 42.2 as identification information to be associated with each video data 42.1 with reference to the video feature amount calculated for each video data 42.1. That is, the event sequence extraction unit 38 estimates the type of the video data 42.1 based on the video feature amount calculated by the video feature amount calculation unit 32, and when the video data is captured, the subject 2 The object label 42.2 is determined as information characterizing the object to be targeted in the behavior performed by the user, and the determined object label 42.2 is associated with the corresponding video data 42.1. Then, the data storage unit 40 stores a set of the video data 42.1 and the corresponding object label 42.2. That is, the data storage unit 40 provides a function of associating and storing information characterizing the object which is the target of the action of the subject 2 and the corresponding perceptual experience.

  Further, the event sequence extraction unit 38 identifies the location where the subject 2 is located when the exercise data is collected, based on the feature data calculated by the position feature data calculation unit 37 from the position data 44.1. The data storage unit 40 stores a set of the position data 44.1 and the corresponding position label 44.2.

  Furthermore, the data storage unit 40 also stores language corpus data 48 as described later.

[Hardware configuration]
Next, hardware used for the event sequence extraction device 1 according to the present embodiment will be described.
(Wearable camera 10)
FIG. 2 is a schematic view showing the configuration of the wearable camera 10 shown in FIG. Referring to FIG. 2, wearable camera 10 includes an imaging unit 102, a control unit 104, a communication unit 108, and a battery 110 for supplying power to each unit. The control unit 104 gives a control command to the imaging unit 102 to acquire video data at a target cycle or timing, and sends out the acquired video data via the communication unit 108. The control unit 104 has a timer 106 and adds information indicating the timing at which each video data is acquired to the acquired video data.

Although not particularly limited, for example, for the largest object in the field of view of wearable camera 10, image feature amount calculation unit 32 extracts this object from the image, calculates the feature amount, and event sequence extraction unit 38 It is also possible to perform the process of attaching an "object label" by executing recognition processing using a reoccurring neural network as described later and classifying it to this and attaching label information.
(Motion sensor 20)
FIG. 3 is a schematic view showing the configuration of the motion sensor 20 shown in FIG. Referring to FIG. 3, motion sensor 20 is a battery that supplies power to acceleration sensor 202, gyro sensor 204, position sensor 205, control unit 206, communication unit 210, control unit 206 and communication unit 210. And 212. The control unit 206 acquires the acceleration data output from the acceleration sensor 202, the angular velocity data output from the gyro sensor 204, and the position data output from the position sensor 205, and via the communication unit 210, Send data The control unit 206 has a timer 208, and adds information indicating the timing at which each data is acquired to the data to be sent.

In addition, if it is the outdoors, as a position sensor, it should just be a sensor which outputs positional information, such as the latitude longitude measured by positioning devices, such as GPS. Also, if indoors, position information from a beacon transmitter using a Bluetooth (registered trademark) signal or the like provided in each room, position information from a wireless tag device using a signal of a predetermined frequency band, Alternatively, it is possible to use a sensor or the like that outputs position information calculated based on the radio wave intensity from a plurality of wireless LAN access points. However, as a method of acquisition of position information, it is not limited to these methods.
(Information processing device 50)
FIG. 4 is a schematic view showing the configuration of the information processing apparatus 50 shown in FIG. The information processing apparatus 50 typically employs a computer conforming to a general-purpose architecture. More specifically, referring to FIG. 4, the information processing device 50 includes a processor 502, a main memory 504, a network interface 506, a communication interface 508, an input unit 510, an output unit 512, and a secondary And a storage unit 520. Each of these components is communicably connected to each other via a bus 514.

  The processor 502 executes the program codes expanded in the main memory 504 in the specified order to realize various processes as described later. As the processor 502, either a single core configuration or a multi-core configuration may be adopted, or a plurality of processors may be used. As the main memory 504, typically, a volatile storage device such as a dynamic random access memory (DRAM) is used.

  The network interface 506 exchanges data with other devices via a LAN (Local Area Network) or the like. The communication interface 508 exchanges data with the wearable camera 10 and the motion sensor 20 shown in FIG. Typically, the communication interface 508 employs a device that exchanges packets by wire or wirelessly such as Ethernet (registered trademark). As the communication interface 508, a wireless device such as Blutooth (registered trademark) is preferably employed. However, the network interface 506 and the communication interface 508 may be realized by a common device.

  The input unit 510 is a device that receives an operation from a user, and is configured of, for example, a keyboard, a mouse, a touch panel, and the like. The output unit 512 is a device that presents various information to the user or outputs various data to another device, and includes, for example, a display, various indicators, and a printer.

  The secondary storage unit 520 stores the collected video data and motion data, and stores an operating system (OS) and application programs executed by the processor 502. For example, the secondary storage unit 520 includes the functions of the video collection unit 30, the motion information collection unit 34, the position information collection unit 35, and the video feature quantity calculation unit 32, the motion feature quantity calculation unit 36, and the position feature quantity calculation unit 37. And an event sequence extraction program 524 for realizing the event sequence extraction function. The feature amount calculation program 522 and the event sequence extraction program 524 may execute a target program using a library provided by the OS. Even in this case, these programs can be included in the scope of the present invention.

  Typically, the feature value calculation program 522 and the event sequence extraction program 524 are distributed integrally or separately via a recording medium such as an optical disc. Alternatively, the feature value calculation program 522 and the event sequence extraction program 524 may be distributed in the form of download via an internetwork or the like. In this case, the recording medium itself storing the feature quantity calculation program 522 and / or the event sequence extraction program 524 may be included in the scope of the present invention.

  FIG. 5 is a diagram showing an example of creating an event timeline by combining semantic concepts predicted from sensor signals.

  In the example shown in FIG. 5, the event sequence extraction unit 38 estimates the semantic concept (walk, sit_down, coffee, cracker, kitchen, living_room) to which the time stamp is given from the sensor signal, and from among a plurality of concepts, Choose an appropriate combination and generate a plausible event sequence. For example, the following event sequence may be generated.

[Walk, *, living_room] → [put, craker, living_room] → [eat, craker, living_room])
That is, the event sequence extraction unit 38 generates a temporally ordered event sequence related to daily living behavior from given sensor data. The event sequence extraction unit 38 combines a plurality of semantic concepts generated from sensor signals and finally creates a timeline of events.

  FIG. 6 is a first conceptual diagram for explaining a procedure of generating an event sequence by the event sequence extraction unit 38, and FIG. 7 is a second conceptual diagram for explaining a procedure of generating an event sequence. FIG. 8 is a third conceptual diagram for describing a procedure for generating an event sequence.

  First, as shown in FIG. 6, the event sequence extraction unit 38 converts a signal obtained by sensing a real environment into a semantic concept representing its content. The meaning concept consists of the concept name, the concept type, and the start and end times at which the concept was observed.

  FIG. 6 shows the process executed by the event sequence extraction unit 38 based on exercise data, and the concept type is "action", and the exercise label is calculated for each predetermined time interval (sliding window). The concept names “walk”, “sit down”, and “drink” are assigned as concept names.

  More generally, the concept name is a label indicating the meaning of the sensor signal, and the concept type indicates the action / object / location category.

  For example, the sensor signal when you drink something is converted into a semantic concept that represents an action. Once the sensor signal has been converted to a semantic concept, any environmental or wearable sensor can be used to predict the event sequence.

  FIG. 7 is a diagram showing a procedure for generating an event candidate.

  Next, the event sequence extraction unit 38 generates event candidates by selecting a combination of a semantic concept of each action and a concept temporally close from a plurality of semantic concepts within a predetermined time window.

  FIG. 8 is a diagram for explaining a method of generating an event sequence.

Among the event candidates obtained as shown in FIG. 7, the event sequence extraction unit 38 relates to temporal relationships between semantic concepts, external language resources, and models of changes between events as described later. Select an event sequence likely to be clues.
[Process of Event Sequence Extraction Unit 38]
In the following, as an example of the processing performed by the event sequence extraction unit 38, a framework for generating a timeline of events related to daily living behavior in the indoor environment will be described.

  This procedure consists of three components: creation of semantic concept from sensor signal, creation of event candidate, learning of Hidden Markov Model (HMM) transition and output probability using real world characteristics, and prediction of event sequence .

  In the event sequence extraction unit 38, the real world state is defined as the event sequence e as follows.

In each event e _i , the semantic concept c is represented by a tuple (c ₁ ,..., C _N ). Here, "tuple" means a set of a plurality of components.

  Here, each meaning concept c is a time stamp c.t representing the time when the meaning concept was observed and a concept name c. w, concept type c. It has an attribute of type ∈ {a, o, p}, where a represents the action taken by the subject, o represents an object observed by the subject, and p represents a place where the subject is present.

In this embodiment, concept names are arranged side by side in the order of (c ₁ .type = a, c ₂ .type = o, c ₃ .type = p), and each element is an action, an object, or a place. Represents a semantic concept name. That is, in the present embodiment, a tuple (c ₁ , c ₂ , c ₃ ) of _three semantic concept names is defined as the content of the event. That is, the content of one event represents (c ₁ .w, c ₂ .w, c ₃ .w), and each event is defined by this and a time stamp et indicating the start time of the event.

However, when the semantic concept c ₂ representing the object is not observed (for example, “walk” or “stand”), the symbol “*” is used instead. Also, the concept name is determined in advance by the concept type c.type.

  FIG. 9 is a table showing concept names according to action types, objects, and place concept types.

As shown in FIG. 9, since each event represents a daily life activity, the time et of the event is defined as the start time of the concept representing the activity in the event. For example, in the case where “man drinks coffee (coffee) in the living room (living room) at 8:31:55” (drink), e = (c1.w = drink, c2.w = coffee, c3 It can be expressed as .w = living_room) and et = 8: 31: 55. Note that all the events are arranged in chronological order by the time stamp et so as to constitute a timeline.
(Meaning concept estimation process)
Next, according to FIG. 6, the procedure for the event sequence extraction unit 38 to create semantic concepts of actions, objects and places that may constitute an event from sensor signals will be described in more detail.

  Since the sensor signal is a continuous signal, a sliding window method is used for feature extraction. In the sliding window method (sliding window method), labels are predicted using sequence labeling for sensor signals within each time window (the number of window size samples is N).

  Although not particularly limited, for example, a gated recurrent unit (GRU), which is a type of recurrent neural network, can be used for sequence labeling.

  Such GRUs are disclosed, for example, in the following documents.

Known documents 1: Chung, J. et al. Cho, K .; and Bengio, Y. "Empirical evaluation of gated recurrent neural networks on sequence modeling. G" ulccehre, cC. ”In The NIPS 2014 Deep Learning and Representation Learning Workshop.
Here, it is assumed that the GRU input data x _t and the hidden state h ^t in each step t. The GRU function is defined as follows.

  Here, σ represents a sigmoid function, and ○ represents a Hadamard product, which is defined as follows.

Here, the dimension n _I is the size of the input vector, the dimension n _H is the size of the hidden vector, and b ^(Z) , b ^(r) and b ^(h) are bias terms. The above function is expressed as ht = GRU (x _t , h _t-1 ).

Also, since the concept is observed at any time, after predicting labels of each time window using GRU, they are combined in time adjacent and one if they are the same label. As a result, as shown in FIGS. 5 and 6, it is possible to generate a plurality of semantic concepts (meaning concept streams) arranged on a time axis.
(Event candidate generation process)
Below, according to FIG. 7, the event sequence extraction part 38 introduces the method of producing | generating the candidate of the event showing a real world by combining and combining the combination of a concept from a semantic concept stream.

In the meaning concept stream, it is not obvious which combination of concepts will be an event. Therefore, using the sliding window, event candidates are generated from the concept in the window. First, create windows w = (w ₁ ,... W _{| w |} ) for each concept of action.

  Here, the number of windows | w | is equal to the number of observed actions. FIG. 7 is an example in which the event sequence extraction unit 38 generates event candidates using a sliding window in the semantic concept stream.

Here, a fixed-length time window (the number of samples of the window size is M) is used with the start time ct of the semantic concept of action as the center of the window. Find concepts overlapping in time in this time window, and generate candidate events that satisfy the tuple (c ₁ , c ₂ , c ₃ ).

  For example, when the meaning concept of action "drink" is provided with a time window from 8:31:55 to 8:31:58, "TV (TV)", "coffee" overlapping with the window , "Table" is the semantic concept of the object in the event candidate, and "living room" is the semantic concept of the place in the event candidate.

As a result, as the multiple event candidates e = (c ₁ , c ₂ , c ₃ ), (c _1. W, c _2. W, c _3. W) = (drink, TV, living_room), (drink, table) , Living_room), (drink, coffee, living_room), (drink, *, living_room) are generated for this window.

  Next, we introduce a method for estimating likely event sequences from event candidates using the properties of the real world.

A Hidden Markov Model (HMM) is used to efficiently find a probable event sequence e ^* from the event candidates of each window. The HMM is composed of transition probabilities between hidden states and output probabilities of the hidden states and output states. The HMM here estimates the optimal event sequence e ^* from among the event candidate group by maximizing the following function.

  Here, w in the HMM corresponds to the hidden state, and e corresponds to the output. This output w has a binary value, and is 1 when there is an event candidate, and 0 otherwise.

Further, P (e _i | w _i ) is an output probability and represents the probability that the event e _i is output from the window w _i . _{Also, P (w i | w i} -1) corresponds to a transition probability, represents a transition ease between the window containing event candidate. Furthermore, it is assumed that the time property of the event and the linguistic likelihood of the event are independent, and let P (e | w) = P (e _l | w) P (e _t | w). Here, P (e _l | w) represents a language model of an event, and P (e _t | w) represents a model of time. If there is no event candidate in the window, state w has no output. As described below, selecting a more likely event candidate by considering P (e _t | w) as P (e _i | w _i ) in addition to P (e _l | w) Can.

The definitions of each probability are introduced below.
(Time relation model)
First, a time-related model P (et _t ) is defined.

Here, let et = (c1 _. T, c2 _. T, c3 _. T). This represents the time when the semantic concept of human actions and objects / places was observed. Here, it is assumed that when an object or a place is observed near the time when the action is performed, there is a relation between the two. For example, when a person acts to drink coffee, he sees an object called a coffee cup. By considering the temporal relationship between the concepts, it is possible to remove event candidates including objects and places unrelated to the action.

  The temporal relationship between the semantic concepts is defined by the following equation.

  Here, e.t represents the time of the event, and c.t represents the start time at which the concept was observed.

Let P (et _t ) be one of the output probabilities of the HMM representing the event series model.
(Event language model)
We use linguistic knowledge mined from language corpus to express the likelihood of combination of concepts.

  Since the language corpus reflects common-sense knowledge in the real world, this property is used to express the likelihood of human actions and semantic concepts representing objects and places.

  For example, people do not drink crackers, but sometimes drink coffee. Also, people are more likely to drink coffee in the living room than to drink coffee in the bathroom. In order to express such real-world common sense knowledge, a language model is used from a language corpus.

The language model is a probability distribution for a word string, and the language model of the event is P (e _l | w). Here, let e1 = (c1 _. W, c2 _. W, c3 _. W). As a language model of an event, a Recurrent Neural Network (RNN) language model is used.

  This RNN language model is one in which a sequence of input word embedding vectors is trained to a sequence of output word embedding vectors through internal state vectors.

  GRU can be used for the RNN language model. Language models using GRU are known to perform better than standard RNN language models using tanh functions, as also described in the following document.

Known document 2: Chung, J. et al. G "ulcehre, C .; Cho, K .; and Bengio, Y. 2015." Gated feedback recurrent neural networks. In ICML, 2067-2075.
FIG. 10 is a conceptual diagram showing the configuration of an RNN language model using word embedding vectors and GRUs.

  In the RNN language model, the probability that the k-th word appears is defined as follows.

  Here, the k-th word is expressed as a one-hot vector (a vector in which the location of the corresponding word is 1).

  However, concepts observed in the real world do not always appear in the language corpus for learning. Therefore, this problem is dealt with in consideration of the semantic similarity of words. In order to consider the similarity of words, the word embedding vector learned by GloVe disclosed in the following document is used for GRU.

Well-known reference 3: Pennington, J .; Socher, R .; and Manning, CD 2014. Glove: Global vectors for word representation. In EMNLP, 1532-1543.
Assuming that the concept name ci.w in the event is a word and the word embedding vector of this word is x _k , the joint probability when all the word embedding vectors corresponding to the concept in the event are read is as follows.

This is the likelihood of the language model.
(Event series model)
In daily life, in order to perform different actions sequentially, events representing daily life actions change little by little before and after. For example, people often wear shoes rather than taking off their slippers and putting on their slippers again. In this embodiment, this change in event is used for the transition probability of the HMM.

The dissimilarity of a vector representing an event is represented as the transition probability P (e _i | e _i-1 ) of the event as follows.

Here, q _i represents the hidden state vector of the GRU when the embedded vector of the end word in the event e _i is read. P (e _i | e _i-1 ) is the transition probability of the event series model by the HMM.

  FIG. 11 is a conceptual diagram showing an example of event transition.

  For example, event route (remove, shoe, entrance) → (wear, slipper, entrance) → (walk, *, living_room), other event route (remove, shoe, entrance) → (wear, shoe, entrance) → It is considered to be a more likely route than (walk, slipper, entrance).

The reason is that the vector representation of the event (wear, slipper, entrance) is less similar to (remove, shoe, entrance) compared to the vector representation of (wear, shoe, entrance). Considering these event transitions, select a likely sequence of events.
(Select event series)
Next, referring to FIG. 8, the procedure of the event sequence extraction unit 38 selecting an event sequence will be described in more detail.

  There are multiple event candidates in each window of the semantic concept stream, and multiple paths of event candidates can be considered between the windows. FIG. 8 shows the lattice structure of the HMM representing the path of the event.

As described above, in this embodiment, since the HMM is used for the series of events, the transition probability P (e _i | e _i-1 ) between the events calculated so far and the normalized output probability P (e | By using w) = P (e _l | w) P (e _t | w), the Viterbi algorithm can efficiently find the optimal sequence of events. This optimal event sequence is the event timeline.

By the configuration of the event sequence extraction unit 38 as described above, it is possible to extract an event timeline (event sequence) based on a sensor signal so as to generalize to an unknown place.
[Experimental result]
In the following, the event sequence generation method of the present embodiment is evaluated in comparison with a baseline using a daily life activity data set. This daily living behavior data set is a verbalized result of simulating daily living behavior indoors by a plurality of subjects.

  In order to evaluate the operation of the event sequence extraction unit 38 of the present embodiment, a daily life activity data set is used.

This data set is a data set used for searching for daily life behavior, and is composed of motion signals (acceleration and gyro) and a first-person viewpoint image captured by a head mounted camera.
Since this data set uses wearable sensors for sensing daily living behavior, it is possible to sequentially track the details of the subject's behavior even if the location is moved. In this data set, eight subjects wear motion sensors and wearable cameras, and 20 daily living activities are considered as one session, and the place-specific behavior described in the worksheet at different places is arbitrary according to the subject's intention. Do it at the timing.

  For example, a subject sits on a sofa, drinks coffee while watching TV in a living room, and then moves to the kitchen to wash dishes, etc. This series of actions is repeated 10 times, and 10 sessions per person, totaling about 17 hours worth of data are collected. Data collection by this semi-natural experimental protocol is frequently used in behavioral recognition research to collect sensor data on various behaviors in a limited environment.

  FIG. 12 is a diagram showing house layouts used in the experiment and 20 place-specific actions performed by the subject.

  FIG. 12 (a) shows a floor plan of a house, and FIG. 12 (b) shows an action.

  For example, as labels for places, there are "kitchen", "living room", "bedroom", "door entrance", "toilet" and "lavatory".

  Also, as a label of action, there are "put on shoes" and "remove shoes" at "entrance", for example.

  Or, in the "kitchen", "make a sandwich cookie" "make coffee" "wash hands" "wash the dishes" and so on.

  In the living room, drink coffee, eat a sandwich cookie, mop the floor, turn on / off the air conditioner (turn on / off A / C), "turn on / off TV", etc.

  In each of the "bedroom", the "bedroom", and the "washroom", specific actions are defined in advance.

  In order to evaluate the generation method of the event timeline, an event sentence representing an action, an object, and a place is described in this daily life activity data set.

  How to make a data set is as follows. First, two workers manually label the action concept of 27 actions, 39 objects, and 6 places while watching the image from the wearable camera attached to the subject's head. Do.

  Next, using the crowdsourcing service, six workers selected the semantic concepts that make up the event from the semantic concepts that were labeled, and manually rearranged the semantic concepts so that they would be correct events. .

  For example, when the drink (action), coffee (object) and living_room (place) concepts were observed, the concepts were rearranged so that "one person drinks coffee in the living room".

  As a result, a total of 550 unique events and 11,501 action concepts, 13,087 object concepts, 931 location concepts, and 11,280 events were created.

  The event sequence extraction unit 38 of the present embodiment automatically generates an event timeline that is a sequence of events from the given semantic concept. The performance of the event sequence extraction unit 38 of the present embodiment is evaluated by looking at the degree of coincidence between the generated event sequence and the event sequence manually labeled.

(Experimental conditions)
Create a semantic concept from sensor data acquired from the wearable sensor. The method of feature extraction was performed in the same manner as disclosed in the following document using the same data set.

Known materials: Miyanishi, T .; Hirayama, J.-i .; Maekawa, T .; Kong, Q .; Moriya, H .; and Suyama, T. 2016. Egocentric video search via physical interactions. In AAAI, 330- 336.
As feature extraction of motion data, feature extraction was performed using short-time Fourier transform by shifting one sample at a window size of 75 samples to data of 25 Hz.

  For feature extraction of moving images, feature extraction of images was first performed using the value of the activation function of the final layer of the VGG model pre-learned with ImageNet for each image, and moving feature was extracted using a sliding window.

  The VGG model is disclosed in the following document.

Known documents: Simonyan, K., and Zisserman, A. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.
Furthermore, motion data and image features of VGG are downsampled every 0.2 seconds and combined into one vector to create a vector within a 3 second window, and from this vector of sensor data, a concept of a semantic concept is labeled using GRU. Made a prediction. Since the semantic concepts of behavior and place do not overlap in time, we used cross-entropy loss to predict labels in multiclass classification. On the other hand, since the semantic concept of the object is observed more than once, it may overlap in time. Therefore, we used multi-label classification to predict labels by using one-to-all loss based on maximum entropy to predict labels of objects. Finally, we combined adjacent labels to create a semantic concept with arbitrary duration. As a result, the semantic concepts of 14,444 actions, 34,923 objects, and 1,113 places were predicted.
(Generation of event series)
The semantic concept estimated from the sensor signal is used to create a sequence of plausible events. The Montreal Video Annotation Dataset (M-VAD) corpus was used to learn the language model of the event. From this M-VAD corpus, we extracted verbs and their dependencies using the Stanford dependency analyzer.

  As the target of the word to be used, I was limited to the main part of the verb and the complement and XCMOP. The vocabulary of the event component chose the nouns of the top 4,000 main parts, 4,000 verbs, and 6,000 verbs who were frequently used. The training data of the event language model used a total of 51,564 events and used 6,871 events as verification data. Moreover, the embedded vector of the word used for the language model of an event was created using Glove learned with the English version of Wikipedia.

  Here, the dimension of the hidden layer of Glove was 300, and the vocabulary used 400,000 words. The other words were learned by replacing them with unknown tokens. We used GRU to learn the language model, and using Adam as the optimization method, we set the learning rate to 0.01, the batch size to 32, and lost learning at 10 epochs to see the loss of verification data. The GRU 300-dimensional hidden layer was initialized according to the normal distribution N (0, 0.01). Regarding event timeline generation, the event sequence generation method of this embodiment is based on the cross-validation of leave-one-place-out with test data for an event sequence in one place and training data for an event sequence in another place. Parameters N and M were adjusted and evaluated.

  In order to verify the superiority of the event sequence generation method of the present embodiment, a baseline method is prepared. FullRandom creates an event sequence by randomly selecting events from all event candidates. WindowRandom creates an event sequence by randomly selecting an event from the event candidates of each window. Unigram (MVAD) creates an event sequence by ranking event candidates of each window based on the likelihood of unigrams trained in the MVAD corpus. Bigram (MVAD) creates an event sequence by ranking the event candidates of each window based on the likelihood of the bigram trained in the MVAD corpus. Video2TextGenerate generates an event directly from given sensor data using sequence-to-sequence. This is the same method as. Video2TextRanking creates an event sequence by ranking event candidates of each window using sequence-to-sequence likelihood.

  In the event sequence generation method of the present embodiment, RNN-Lang, RNN-Lang + Time, and RNN-Lang + Time + Context are prepared to see the effect of each component of the model. RNN-Lang generates an event sequence by ranking event candidates of each window based on the likelihood of the RNN language model learned in the MVAD corpus. RNN-Lang + Time adds a temporal model to RNN-Lang, adds the temporal model score to the likelihood of the RNN language model, ranks the events based on the total score, and generates an event sequence . RNN-Lang + Time + Context adds an event change model to RNN-Lang + Time, and uses HMM, taking into account transition probability between events in RNN language model likelihood and time model score Generate an event sequence.

(Event generation performance)
In the following, baseline Unigram (MVAD) and Bigram (MVAD), which use word statistical information, and event-generating methods using sequence-to-sequece, which are often used for generating explanatory sentences for moving pictures, are compared with Video2TextGenerate and Video2TextRanking. Then, we evaluate the performance of the proposed event timeline generation method RNN-Lang + Time + Context.

  The prepared baseline does not use real-world properties such as temporal models between concepts and changes between events.

  In evaluating the event timeline, two kinds of evaluation are performed: conditions for generating an event sequence from manually labeled semantic concepts and conditions for generating an event sequence from semantic concepts predicted by GRU. These two conditions are respectively described as a correct concept (TrueConcept) and a prediction concept (PredConcept). By using the semantic concept that is labeled manually, it is possible to evaluate the performance of only event timeline generation, and by predicting the event sequence using the predicted semantic concept, it is possible to make the condition more noisy and closer to reality Can evaluate the event series method.

  BLEU, METEOR, CIDER are used as indices for evaluating the event timeline generation method. These indices are typically used to evaluate techniques for generating image captions.

  These indices are disclosed in the following documents.

Known documents: Fang, H. et al. Gupta, S .; Iandola, F .; Srivastava, R. K ,; Deng, L .; Doll'ar, P .; Gao, J, He, X .; Mitchell, M .; Platt, J. C .; From captions to visual concepts and back. In CVPR, 1473-1482.
In particular, BLEU and METEOR are used as standard in the evaluation of machine translation systems, and BLEU looks at the degree of coincidence between the predicted event and the correct N-gram. CIDEr is used to evaluate an explanatory text generated from an image, and evaluates how much a word in a predicted sentence is included in a word set of a correct sentence.

  The score of BLEU, METEOR, CIDEr is calculated by comparing the event sequence generated by the event sequence generation method of the present embodiment with the baseline and the event sequence generated manually. The higher these scores indicate that the performance of the event timeline generation method is higher.

  In the following, we will assess the overall performance of the event timeline.

  FIG. 13 is a diagram showing the evaluation result of the extracted event timeline.

FIG. 13 shows the overall results of the baseline and the event sequence generation method of the present embodiment when using the correct concept (TrueConcept) and the predictive concept (PredConcept).
Based on the results of the correct concept, although these methods use the same language corpus to construct language models, RNN-Lang is more effective than Unigram (MVAD) and Bigram (MVAD) for all metrics. It can be seen that the This result suggests that RNN-Lang using RNN language model that can express word string of arbitrary length can more accurately represent real-world state compared with standard unigram and bigram language models. ing.

  Furthermore, RNN-Lang is higher in all evaluation indexes compared to Video2TextGenerate and Video2TextRanking using sequence-to-sequence.

  This is because the method using sequence-to-sequence over-fits the sensor data collected in a specific environment, and can not predict events in other places.

  In addition, Video2TextGenerate has almost the same result as a model (FullRandom) randomly selected from event candidates. In order to generalize the generation performance of event sequences in different environments, this is not learning with a parallel corpus of sensors and their explanatory sentences, but word strings that represent real-world states using external language resources Suggests that you need to learn.

  That is, the sequence-to-sequence used in the description generation of a moving image is not suitable for generating an event timeline in an environment where the location changes. From the results using the correct answer concept, the proposed event sequence generation methods RNN-Lang, RNN-Lang + Time, RNN-Lang + Context, and RNN-Lang + Time + Context exceed the baseline.

  This indicates that it is effective to incorporate the characteristic nature of the real environment in order to generate an event sequence related to daily living behavior. In particular, RNN-Lang + Time shows better performance than RNN-Lang, so it can be seen that considering the closeness in time between concepts is important in generating real-world events . Moreover, since the performance of RNN-Lang + Time + Context exceeds RNN-Lang + Time, it was found that a more correct event sequence can be generated by using a model in consideration of event changes. Since the performance of RNN-Lang + Time + Context surpasses RNN-Lang and RNN-Lang + Time, not only common sense knowledge expressing likelihood of combination of concepts, but also concept expressing action and object / place It can be seen that modeling by adding transitions between events representing temporal living behavior and events that change with time is effective in generating an event timeline for everyday life.

  Next, we introduce the results of generating event sequences using the prediction concept.

  Naturally, because there is an error in the prediction concept, the result using the prediction concept is lower in the value of the evaluation index than in the case where the correct concept is used.

  From FIG. 13, the proposed RNN-Lang exceeds the baseline Unigram (MVAD), but the RNN-Lang and Bigram (MVAD) are comparable.

  It is considered that this is because the performance of RNN-Lang is degraded because an event candidate includes an incorrectly predicted concept. However, RNN-Lang + Time which added the model of time to common sense knowledge can improve the performance of event sequence generation compared with RNN-Lang, and as a result, Unigram (MVAD), Bigram (MVAD) It exceeds the performance of).

  This indicates that the performance of event generation can be improved by considering the temporal closeness of concepts even under more practical conditions. Furthermore, as with the correct answer concept, RNN-Lang + Time + Context exceeds RNN-Lang and RNN-Lang + Time.

  This result shows that it is easy to select the correct event sequence that represents the state of the real world by considering the change of the event even in noisy conditions. In addition, the method using RNN-Lang + Time + Context of the event sequence generation method of the present embodiment outperforms all methods in all evaluation indexes. The results show that modeling the characteristics of the real world enables accurate event timelines to be generated using data sensed in the real environment.

  As described above, in the event sequence extraction unit 38 according to the present embodiment, the characteristics of the real world such as temporal proximity of semantic concepts, linguistic likelihood of combination of concepts, temporal change of events, etc. I am modeling. As a result of conducting evaluation experiments using a data set related to daily living behavior, by modeling the features of the real world, it is possible to correctly generate a series of events that change sequentially.

  The embodiment disclosed this time is an illustration of a configuration for specifically implementing the present invention, and does not limit the technical scope of the present invention. The technical scope of the present invention is indicated not by the description of the embodiment but by the scope of claims, and includes modifications within the scope of wording and meaning of the scope of claims. Is intended.

  DESCRIPTION OF SYMBOLS 1 event sequence extraction apparatus, 2 test subjects, 10 wearable cameras, 20 motion sensors, 30 image collecting units, 32 image feature amount calculating units, 34 exercise information collecting units, 35 position information collecting units, 36 exercise feature amount calculating units, 37 positions Feature amount calculation unit, 38 event sequence extraction unit, 40 data storage unit.

Claims (7)

A first sensor for acquiring data for identifying an object of a subject's perceptual experience;
A second sensor that acquires position information for specifying the behavior of the subject and the position of the subject during the perceptual experience of the subject;
A storage device for storing information;
For the subject, an action sequence for storing an action label, an object label, and a position label for identifying each of the action, the object, and the position as a timeline in the storage device and extracting an event sequence And an extraction means,
The event sequence extraction unit
Concept estimation processing means for predicting the action label, the object label, and the position label for each predetermined sliding window, and executing a process of combining and arranging identical labels on the timeline;
Regarding the combination result of the action label, the object label and the position label on the timeline, a plurality of candidates of combinations of the object label and the position label within a predetermined time window based on the action label Language model comparison means for extracting the most likely combination as an event according to the likelihood that the action label corresponding to the candidate, the object label, and the position label exist in the language corpus;
And an event sequence selection unit that extracts the event sequence from the extracted time series of events.   The language model comparing means may select a combination according to the likelihood existing in the language corpus and the likelihood due to the closeness of time between the combined action label, the object label and the position label. The event sequence extraction device according to claim 1, which is extracted as an event.   The concept estimation processing means is a gated recursive unit that predicts the action label, the object label, and the position label using sequence labeling for the first and second sensor signals in each sliding window. The event sequence extraction device according to claim 1 or 2, comprising   The event series extraction device according to any one of claims 1 to 3, wherein the language model comparison means includes a gated recursive unit which receives a word embedding vector in consideration of word similarity.   The event sequence according to any one of claims 1 to 4, wherein the event sequence selection means extracts the event sequence by a hidden state model in which the degree of dissimilarity of the vector representing the event is a transition probability of the event. Extraction device. An event sequence extraction method for extracting information on a series of events occurring in a subject from sensor data in a real environment, comprising:
Acquiring, with a first sensor, data for identifying an object of the perceptual experience of the subject;
Obtaining the behavior of the subject and position information for specifying the position of the subject by a second sensor when the perceptual experience of the subject is performed;
For each predetermined sliding window, an action label, an object label and a position label for identifying each of the action, the object and the position are predicted, and the same label is combined and arranged on the timeline. Performing the process;
Storing the action label, the object label and the position label as the time line in the storage device for the subject;
Predicting the action label, the object label, and the position label for each predetermined sliding window, and combining and arranging the same labels on the timeline;
Regarding the combination result of the action label, the object label and the position label on the timeline, a plurality of candidates of combinations of the object label and the position label within a predetermined time window based on the action label Extracting a most likely combination as an event according to the likelihood that the action label, the object label, and the position label corresponding to the candidate exist in a language corpus;
Extracting an event series from the extracted time series of events. An event sequence extraction program for causing a computer to execute an event sequence extraction process for extracting information on a sequence of events generated in a subject from sensor data in a real environment, the event sequence extraction program comprising: ,
Acquiring, with a first sensor, data for identifying an object of the perceptual experience of the subject;
Obtaining the behavior of the subject and position information for specifying the position of the subject by a second sensor when the perceptual experience of the subject is performed;
For each predetermined sliding window, an action label, an object label and a position label for identifying each of the action, the object and the position are predicted, and the same label is combined and arranged on the timeline. Performing the process;
Storing the action label, the object label and the position label as the time line in the storage device for the subject;
Regarding the combination result of the action label, the object label and the position label on the timeline, a plurality of candidates of combinations of the object label and the position label within a predetermined time window based on the action label Extracting a most likely combination as an event according to the likelihood that the action label, the object label, and the position label corresponding to the candidate exist in a language corpus;
An event sequence extraction program that executes an event sequence extraction step from the time series of the extracted events.