JP6829368B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present disclosure relates to information processing devices, information processing methods, and programs, and more particularly to information processing devices, information processing methods, and programs that enable effective extraction of desired classification results from a large amount of data.

In recent years, the position of a person has been used by using an RGB-D sensor that acquires color (RGB) and depth (Depth) images using a color camera and an infrared camera, and a laser range finder that measures the reflection intensity of an infrared laser. It is possible to measure.

For example, the present inventors have proposed a method of identifying a plurality of children by using two RGB-D sensors and tracking the position of each child to extract a movement trajectory (). For example, see Non-Patent Document 1).

Zhang Yi, Tomoaki Nakamura, Kasumi Abe, Attami Mimhammad, Takayuki Nagai, Takashi Omori, Natsuki Oka, Masahide Kaneko, "Children's Behavior Tracking and Personal Authentication Using Multiple Kinects", Japanese Society for Artificial Intelligence National Convention, 4K4-1 ， 2016

By the way, in order to effectively utilize a large amount of data (position, velocity, etc.) acquired by continuously performing the above-mentioned measurement, the large amount of data is classified and a desired classification result is extracted. There is a need. However, in order to extract a desired classification result from a large amount of data, for example, it becomes troublesome to preset what kind of classification result is required, and the desired classification result is effectively extracted. It was very difficult to do.

The present disclosure has been made in view of such a situation, and makes it possible to effectively extract a desired classification result from a large amount of data.

The information processing apparatus of one aspect of the present disclosure classifies the time-series data for each observation object into a first classification item by unsupervised clustering of time-series data obtained by observing a plurality of observation objects. By unsupervised clustering of the first classification unit to be used and the feature quantities representing the characteristics of the set of the first classification items for the entire observation target, the first classification unit for the entire observation target is performed. It is provided with a second classification unit that classifies a set of one classification item into a second classification item.

The information processing method or program of one aspect of the present disclosure unsupervised clusters the time-series data obtained by observing a plurality of observation objects, thereby classifying the time-series data for each observation object into a first classification item. By unsupervised clustering the feature quantities representing the characteristics of the set of the first classification items for the entire observation target, the first classification item for the entire observation target is obtained. Includes a step of classifying the set of items into a second category.

In one aspect of the present disclosure, by unsupervised clustering of time-series data obtained by observing a plurality of observation objects, the time-series data for each observation object is classified into a first classification item, and a plurality of observations are performed. By unsupervised clustering of features representing the characteristics of the set of first classification items for the entire object, the set of first classification items for the entire observation target is classified into the second classification item. Will be done.

According to one aspect of the present disclosure, a desired classification result can be effectively extracted from a large amount of data.

It is a figure explaining the outline of the activity classification process to which this technique is applied. It is a block diagram which shows the structural example of one Embodiment of the information processing system to which this technology is applied. It is a figure explaining the classification of behavior. It is a figure explaining the generation of activity feature quantity. It is a figure explaining the classification of activity. It is a flowchart explaining the activity classification process. It is a block diagram which shows the structural example of one Embodiment of the computer to which this technique is applied.

Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

<Outline of activity classification process>

First, the outline of the activity classification process will be described with reference to FIG.

In the present embodiment, for example, as in the observation scene shown in FIG. 1, children are observed to be playing freely, and what kind of activity (play) is being performed from the observation results. The activity classification process for classifying is described.

First, a plurality of children in the place where the observation is performed are identified as observation targets, the position of each observation target is measured, and the movement locus that tracks the position where the observation target moves at each time is extracted. Then, the position and velocity of each observation target are unsupervised clustered by HDP-HMM (Hierarchical Drichlet Process-Hidden Markov Model). As a result, the position and velocity of each observation target can be classified (discretized) into local behaviors.

Further, the action frequency histogram is calculated by counting the frequency at which each action is performed for each arbitrary time zone. For example, if a set of actions for the entire observation target is regarded as an activity, the action frequency histogram can be used as an activity feature quantity representing the characteristics of each activity. Then, the behavior frequency histogram, which is an activity feature amount, can be classified by activity by unsupervised clustering by LDA (Latent Dirichlet Allocation) or HDP-LDA (Hierarchical Drichlet Process --LDA).

In this way, by performing two-step unsupervised clustering using the observation results of children in the observation scene, a set of behaviors of a plurality of children can be classified as a specific activity.

<Configuration example of information processing device>

FIG. 2 is a block diagram showing a configuration example of an embodiment of an information processing system to which the present technology is applied.

As shown in FIG. 2, the information processing system 11 includes an observation device 12, an input device 13, a storage device 14, and an activity classification processing device 15. Further, as shown in the figure, the information processing system 11 has a connection configuration in which the observation device 12, the input device 13, and the storage device 14 are connected to the activity classification processing device 15, respectively.

The observation device 12 is configured to include, for example, a plurality of RGB-D sensors, and supplies a color image and a depth image obtained by capturing an observation scene from a plurality of directions to the activity classification processing device 15. The observation device 12 may have a configuration capable of extracting a movement locus as described later, and the type and number of sensors used in the observation device 12 are particularly limited to the description in the present embodiment. There is nothing.

The input device 13 is composed of, for example, a keyboard, a mouse, or the like, and inputs various input values (for example, a window width described later) according to an operation by the user to the activity classification processing device 15.

The storage device 14 is composed of, for example, a hard disk drive or a memory, and various data temporarily stored when the activity classification processing device 15 performs the activity classification processing and the activity classification processing device 15 perform the activity classification processing. The classification result obtained as a result is stored.

The activity classification processing device 15 includes a movement locus extraction unit 21, a behavior classification unit 22, an activity feature amount generation unit 23, and an activity classification unit 24. Further, as shown in the figure, in the activity classification processing device 15, the movement locus extraction unit 21 is connected to the behavior classification unit 22, the behavior classification unit 22 is connected to the activity feature amount generation unit 23, and the activity feature amount generation unit 23 It has a connection configuration connected to the activity classification unit 24.

The motion locus extraction unit 21 identifies each observation target based on the color image and the depth image supplied from the observation device 12, and identifies the position of each observation target. Then, the movement locus extraction unit 21 extracts the movement locus for each observation target by tracking the change in the position due to the movement of each observation target, and supplies the movement locus to the behavior classification unit 22.

For example, the movement locus extraction unit 21 performs object recognition processing on the color image supplied from the observation device 12, and identifies each child based on the child's face, clothes color, etc. reflected in the color image. can do. Further, the movement locus extraction unit 21 can specify the position (xy coordinates) for each child identified in the color image according to the distance from the depth image supplied from the observation device 12 to the child. Here, the position for each child is specified by using, for example, the coordinate position on the color image, and when the position for each child can be specified in the real space, the coordinate position in the real space is used. May be specified. Further, the method of specifying the position of each child is not particularly limited to these.

The behavior classification unit 22 unsupervised clusters the positions and velocities for each observation target obtained from the movement locus supplied from the movement locus extraction unit 21 by HDP-HMM. As a result, the behavior classification unit 22 classifies each observation target into actions (classification items) composed of their respective positions and speeds, and supplies the behavior for each observation target to the activity feature amount generation unit 23.

Here, HDP-HMM is a hierarchical Dirichlet process hidden Markov model, which is one of the models expressed by the hidden state and the stochastic transition between the hidden states. For example, the training data can be obtained without determining the number of states in advance. The optimum number of states can be estimated according to the complexity.

For example, as shown in FIG. 3, the movement locus extraction unit 21 clusters the position and the velocity for each time t of the movement locus of the first observation target and the second observation target, so that the first observation target And the action ID (Identification) that identifies the action performed by the second observation target is dynamically assigned. As a result, for example, the same action ID is assigned to similar positions and velocities (range of the broken line shown in FIG. 3) in the movement trajectories of the first observation target and the second observation target. In this way, the behavior classification unit 22 can classify similar positions and velocities for the movement loci of each observation target into corresponding behaviors.

The activity feature amount generation unit 23 generates an activity feature amount representing the activity characteristics of the plurality of observation objects as a whole based on the behaviors of the plurality of observation objects supplied from the behavior classification unit 22, and the activity classification unit 24 Supply to.

For example, as shown in FIG. 4, the activity feature amount generation unit 23 has an arbitrary window width (time zone) in which the user operates the input device 13 to input the actions of the first observation target and the second observation target. ). Then, the activity feature amount generation unit 23 counts the number of times that the action ID assigned in each window width appears, and generates an action frequency histogram (fixed length vector) obtained by this as the activity feature amount. The window width may be set continuously as shown in FIG. 4, or may be set so as to overlap with a predetermined width, for example.

The activity classification unit 24 activates (classifies) a set of actions for the entire plurality of observation objects by unsupervised clustering the activity features supplied from the activity feature generation unit 23 by LDA, HDP-LDA, or the like. Item). Then, the activity classification unit 24 stores the classification result obtained as a result of the classification in the storage device 14.

Here, LDA is a latent Dirichlet allocation method that estimates latent states (topics) for documents and words. For example, the number of latent states is given in advance, the activity feature is set as "document", and the action is performed. The activity (latent state) can be estimated as a "word". In HDP-LDA, the required number of latent states is automatically determined according to the complexity of the data.

For example, as shown in FIG. 5, the activity classification unit 24 is based on the similarity of each activity feature amount (the shape similarity of the histogram as shown) for each time zone in which the activity feature amount is generated. Therefore, the same activity ID can be dynamically assigned to similar activity features. The number of classifications (number of activity IDs) for which the activity classification unit 24 classifies the activities may be input by the user by operating the input device 13, or the activity classification unit 24 is appropriate from all the activity feature quantities. The number of classifications may be estimated.

In this way, the information processing system 11 is configured, and by performing unsupervised clustering by the behavior classification unit 22 and unsupervised clustering by the activity classification unit 24 in two stages, various activities by the entire plurality of observation objects are performed. The classification result for each can be effectively extracted. As a result, the information processing system 11 semi-automates the annotation of activities by a plurality of observation targets (that is, the assignment of activity IDs is automatically performed, but the meaning of each activity is not automatically performed. It is assumed that the meaning of the activity is performed by an observer or another system based on the above-mentioned activity ID depending on the situation).

For example, in the past, when performing a process of classifying activities by a plurality of observation targets, it was necessary to preset the behavior to be classified as teacher data, and various troubles such as how to set the teacher data have occurred. Therefore, the processing could not be easily performed.

On the other hand, the information processing system 11 can dynamically assign the activity ID from the movement trajectories of a plurality of observation targets without setting the teacher data in advance, and can effectively extract the classification result. .. For each activity ID, what kind of activity it is can be defined by a person who sees an image or the like after classifying the activity.

Specifically, the information processing system 11 can automatically classify, for example, what kind of play the children in the nursery school are playing, and then the nursery teacher can classify the content of the play (for example,). It can mean hide-and-seek, tag, etc.). Furthermore, by classifying the activities of children over a long period of time by the information processing system 11, it is possible to observe the growth of those children as a group.

Further, the information processing system 11 can search for similar activities (observation scenes) observed so far by referring to the classification results stored in the storage device 14, for example.

In particular, the information processing system 11 uses the activity features of a plurality of observation targets as a whole, so that, for example, the identification accuracy for individually identifying the observation targets is low, or the number of observation targets is not completely grasped. Even so, the activity of multiple observation objects as a whole can be accurately classified.

<Flowchart of activity classification process>

Next, the activity classification process executed in the activity classification processing device 15 will be described with reference to the flowchart shown in FIG.

For example, processing is started when a color image and a depth image for a certain period of time are supplied from the observation device 12, and in step S11, the motion locus extraction unit 21 is based on the color image and the depth image supplied from the observation device 12. Then, the movement trajectory for each observation target is extracted. The process of extracting the movement locus for each observation target by the movement locus extraction unit 21 is described in detail in Non-Patent Document 1 described above.

In step S12, the action classification unit 22 uses the movement locus extracted by the movement locus extraction unit 21 in step S11 to perform unsupervised clustering by HDP-HMM to set the position and speed of each observation target for each action. Classify into.

In step S13, the activity feature amount generation unit 23 generates a histogram showing the frequency of each action classified by the action classification unit 22 in step S12 as an activity feature amount showing the activity feature of the entire observation target. To do.

In step S14, the activity classification unit 24 classifies the activity feature amount generated by the activity feature amount generation unit 23 in step S13 for each activity by unsupervised clustering by LDA, HDP-LDA, or the like, and obtains the result. The activity ID to be performed is output as a classification result.

As described above, the activity classification processing device 15 classifies the movement trajectories (a large amount of data) of a plurality of observation targets into activities by the entire observation target by performing unsupervised clustering in two stages. Can be effectively extracted. Further, the activity classification processing device 15 can perform such clustering in real time according to, for example, the timing at which the color image and the depth image during imaging are continuously supplied. Of course, the activity classification processing device 15 may perform processing using the already recorded color image and depth image.

In addition to classifying children's play as described above, this technology can be used to classify sports performed in gymnasiums and manage the operation of gymnasiums. In addition, this technology can be useful for crime prevention, for example, by classifying the movements of people in a specific area and extracting people who behave abnormally.

Further, in the present embodiment, the position and velocity of each observation target (processing result obtained by performing the process of obtaining the temporal change of the position with respect to the movement locus) have been described as the target of clustering. , The movement locus itself may be the target of clustering for each observation target. In addition, the acceleration obtained by performing the process of obtaining the temporal change of the velocity with respect to the movement locus of each observation object, and the relationship obtained by performing the process of obtaining the mutual relationship between the movement trajectories of a plurality of observation objects. Processing results such as sex (for example, correlation coefficient) can be targeted for clustering.

Further, the information processing system 11 may use time-series data other than the movement locus of the observation target, and by clustering the data values and processing results of the time-series data without supervised learning, for example, desired other than actions and activities. It is possible to extract the classification results classified into the classification items. Further, the activity feature amount used in the information processing system 11 is not limited to the behavior frequency histogram as described above as long as it represents the activity feature in the whole of the plurality of observation objects.

It should be noted that each process described with reference to the above flowchart does not necessarily have to be processed in chronological order in the order described as the flowchart, and is a process executed in parallel or individually (for example, a parallel process or an object). Processing by) is also included. Further, the program may be processed by a single CPU or may be distributed by a plurality of CPUs.

Further, the series of processes (information processing method) described above can be executed by hardware or by software. When a series of processes are executed by software, the programs that make up the software execute various functions by installing a computer embedded in dedicated hardware or various programs. It is installed from a program recording medium on which a program is recorded, for example, on a general-purpose personal computer.

FIG. 7 is a block diagram showing an example of hardware configuration of a computer that executes the above-mentioned series of processes programmatically.

In a computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are connected to each other by a bus 104.

An input / output interface 105 is further connected to the bus 104. The input / output interface 105 includes an input unit 106 composed of a keyboard, a mouse, a microphone, etc., an output unit 107 composed of a display, a speaker, etc., a storage unit 108 composed of a hard disk, a non-volatile memory, etc., and a communication unit 109 composed of a network interface, etc. , A drive 110 for driving a removable medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

In the computer configured as described above, the CPU 101 loads the program stored in the storage unit 108 into the RAM 103 via the input / output interface 105 and the bus 104 and executes the above-described series. Is processed.

The program executed by the computer (CPU101) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. It is recorded on removable media 111, which is a package media including memory, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

Then, the program can be installed in the storage unit 108 via the input / output interface 105 by mounting the removable media 111 in the drive 110. Further, the program can be received by the communication unit 109 and installed in the storage unit 108 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 102 or the storage unit 108.

The present embodiment is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present disclosure.

11 Information processing system, 12 Observation device, 13 Input device, 14 Storage device, 15 Activity classification processing device, 21 Movement locus extraction unit, 22 Behavior classification unit, 23 Activity feature amount generation unit, 24 Activity classification unit

Claims (8)

A first classification unit that classifies the time-series data for each observation target into a first classification item by unsupervised clustering of time-series data obtained by observing a plurality of observation objects.
By unsupervised clustering the feature quantities representing the characteristics of the set of the first classification items for the whole of the plurality of observation objects, the set of the first classification items for the whole of the plurality of observation objects can be obtained. An information processing device including a second classification unit for classifying into a second classification item. The information processing apparatus according to claim 1, wherein the first classification item is composed of a data value of the time-series data for each observation target and a processing result obtained by processing the time-series data. Based on the first classification item for each observation object classified by the first classification unit, the feature amount generation unit that generates the feature amount for the entire plurality of observation objects at arbitrary time zones. The information processing apparatus according to claim 1 or 2, further comprising. The information according to claim 3, wherein the feature amount generation unit generates a histogram as the feature amount, which counts the number of times the first classification item appears for each of the plurality of observation objects in an arbitrary time zone. Processing equipment. The information processing apparatus according to any one of claims 1 to 4, further comprising an extraction unit that extracts as time-series data a movement locus that tracks a position where a plurality of observation objects move. The first classification unit classifies the behavior of each observation object as the first classification item based on the movement locus of each observation object extracted by the extraction unit.
The information processing device according to claim 5, wherein the second classification unit classifies the activities of the plurality of observation objects as a whole as the second classification item based on the set of actions. By unsupervised clustering the time-series data obtained by observing a plurality of observation objects, the time-series data for each observation object is classified into the first classification item.
By unsupervised clustering the feature quantities representing the characteristics of the set of the first classification items for the whole of the plurality of observation objects, the set of the first classification items for the whole of the plurality of observation objects can be obtained. An information processing method that includes a step of classifying into a second classification item. By unsupervised clustering the time-series data obtained by observing a plurality of observation objects, the time-series data for each observation object is classified into the first classification item.
By unsupervised clustering the feature quantities representing the characteristics of the set of the first classification items for the whole of the plurality of observation objects, the set of the first classification items for the whole of the plurality of observation objects can be obtained. A program that causes a computer to execute information processing including a step of classifying into a second classification item.

Images