JP2018198028A - Movement state recognition model learning device, movement state recognition device, method, and program - Google Patents

Abstract

To efficiently extract and combine information from both time series of image data and time series of sensor data and realize highly accurate movement state recognition with a small amount of training data.SOLUTION: A movement state recognition semi-supervised DNN model construction unit 42 is a DNN model for recognizing a movement state of a user and further constructs a DNN model for decoding time series of decoded image data and time series of decoded sensor data. A movement state recognition DNN model unsupervised learning unit 44 learns parameters of the DNN model so that the time series of the decoded image data and the time series of the decoded sensor data coincide with input data. A movement state recognition DNN supervised learning unit 46 learns the parameters of the DNN model so that a movement state recognized by the DNN model coincides with a movement state indicated by an annotation.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a movement situation recognition model learning apparatus, a movement situation recognition apparatus, a method, and a program, and in particular, movement situation recognition model learning for automatically recognizing a user's movement situation from video and sensor data acquired by the user. The present invention relates to a device, a movement status recognition device, a method, and a program.

  With the downsizing of video imaging devices and the power saving of GPS and gyro sensors, user behavior can be easily recorded as various data such as video, position information, and acceleration. Detailed analysis of user behavior from these data is useful for various applications. For example, using the first-person video acquired through glassesware, etc., and the acceleration data acquired by the wearable sensor, automatically recognize and analyze the situation of window shopping and the situation of crossing a pedestrian crossing. If possible, it can be used for various purposes such as personalization of services.

  Conventionally, as a technique for automatically recognizing a user's movement status from sensor information, there is a technique for estimating a user's moving means from GPS position information and speed information (Non-Patent Document 1). In addition, the development of technology that analyzes walking, jogging, ascending / descending stairs, etc. using information such as acceleration acquired from a smartphone has been undertaken (Non-Patent Document 2).

Zheng, Y., Liu, L., Wang, L., and Xie, X .: Learning transportation mode from raw GPS data for geographic applications on the web.In Proc. Of World Wide Web 2008, pp. 247-256, 2008. Jennifer R. Kwapisz, Gary M. Weiss, Samuel A. Moore: Activity Recognition using Cell Phone Accelerometers, Proc. Of SensorKDD 2010.

  However, since the conventional method uses only sensor information, the user's movement status cannot be recognized in consideration of video information. For example, when trying to grasp the user's movement status from wearable sensor data, even if he understands that he is walking, he / she knows that he / she is walking, but it is a detailed user such as a window shopping situation or a situation where a pedestrian crossing is crossed. It is difficult to automatically recognize the situation from only sensor data. On the other hand, even if a simple classification model such as Support Vector Machine (SVM), which is one of machine learning technologies, is used by combining the input of video data and sensor data, the degree of abstraction of the information of the video data and sensor data can be reduced. Due to the difference, it was difficult to recognize the movement situation with high accuracy. In addition to SVM, many machine learning techniques require a large amount of training data to construct a highly accurate classification model. In order to prepare training data for recognizing a movement situation, an annotation operation is required in which an appropriate movement situation (training label) is manually assigned to a combination of video data and sensor data. It is difficult for a user to perform annotation work on a combination of a large amount of video data and sensor data.

  The present invention has been made in view of the above circumstances, and realizes highly accurate movement state recognition with a small amount of training data by efficiently extracting and combining information from both video data and sensor data. An object of the present invention is to provide a moving state recognition model learning apparatus, method, and program capable of performing the above.

  It is another object of the present invention to provide a movement status recognition apparatus, method, and program capable of recognizing a movement status with high accuracy from both video data and sensor data.

  To achieve the above object, a moving situation recognition model learning apparatus according to the present invention inputs a time series of image data of a camera mounted on a moving body and a time series of sensor data of a sensor mounted on the moving body. And extracting each feature of the image data and each feature of the sensor data, and recognizing the movement state of the moving object from the data obtained by abstracting each feature of the image data and each feature of the sensor data. It is a DNN (Deep Neural Network) model, and further decodes a time series of decoded image data and a time series of decoded sensor data from data obtained by abstracting each feature of the image data and each feature of the sensor data. A moving situation recognition semi-supervised DNNN model construction unit for constructing a DNN model, a time series of the image data, and the sensor data The time series of the decoded image data and the time series of the decoded sensor data output from the DNN model when the time series of the image data and the time series of the sensor data are input based on the time series of A movement situation recognition DNN model unsupervised learning unit that learns parameters of the DNN model so that the time series of the image data and the time series of the sensor data match, and the parameters of the learned DNN model; Based on the time series of the image data and the time series of the sensor data, and the time series of the image data, based on the time series of the image data and the annotation indicating the movement situation previously given to the time series of the sensor data. The movement situation recognized by the DNN model when the time series of the series and the sensor data is input is the annotate. ® As emission matches the moving condition shown, is configured to include a, a movement condition recognition DNN supervised learning unit for learning the parameters of the DNN model.

  Further, in the moving situation recognition model learning method according to the present invention, the movement situation recognition semi-supervised DNN model construction unit performs a time series of image data of a camera mounted on the moving body and a sensor of the sensor mounted on the moving body. The time series of the data is input, each feature of the image data and each feature of the sensor data are extracted, and the movement of the moving body is obtained from the data obtained by abstracting each feature of the image data and each feature of the sensor data. A DNN (Deep Neural Network) model for recognizing a situation, and further, a time series of decoded image data and a decoding sensor from data obtained by abstracting each feature of the image data and each feature of the sensor data A DNN model that decodes a time series of data is constructed, and the movement state recognition DNN model unsupervised learning unit performs the time series of the image data. Based on the sequence and the time series of the sensor data, the time series of the decoded image data and the decoded sensor data output from the DNN model when the time series of the image data and the time series of the sensor data are input. And the time series of the image data and the time series of the sensor data are learned so that the parameters of the DNN model are learned, and the movement state recognition DNN supervised learning unit performs the learning of the DNN Based on the model parameters, the time series of the image data and the time series of the sensor data, the time series of the image data and the annotation indicating the movement situation previously given to the time series of the sensor data, Movement status recognized by the DNN model when the time series of the image data and the time series of the sensor data are input , To match the moving condition of the annotation shown learns the parameters of the DNN model.

  The movement situation recognition apparatus according to the present invention inputs a time series of image data and a time series of sensor data about a moving object to be recognized into the DNN model learned by the movement situation recognition model learning apparatus. And a movement status recognition unit for recognizing the movement status of the moving body.

  In the movement situation recognition method according to the present invention, the movement situation recognition unit learns the time series of the image data and the time series of the sensor data of the moving object to be recognized by the above-described DNN learning method. It inputs into a model and recognizes the movement condition of the said mobile body.

  The program according to the present invention is a program for causing a computer to function as each part of the above-described movement situation recognition model learning apparatus or the above movement situation recognition apparatus.

  According to the moving state recognition model learning apparatus, method, and program of the present invention, a DNN for recognizing the moving state of the moving object from data obtained by abstracting each feature of image data and each feature of sensor data. A DNN model that decodes a time series of decoded image data and a time series of decoded sensor data from data obtained by abstracting each characteristic of the image data and each characteristic of the sensor data. The time series of the decoded image data and the time series of the decoded sensor data output from the DNN model when the time series of the image data and the time series of the sensor data are input, and the time series of the image data And the parameters of the DNN model are learned so that the time series of the sensor data matches, and the time series of the image data and the time series of the image data By learning the parameters of the DNN model so that the movement situation recognized by the DNN model when the time series of sensor data is input matches the movement situation indicated by the annotation, the time series of the image data and the sensor An effect is obtained that information can be efficiently extracted and combined from both time series of data, and highly accurate movement state recognition can be realized with a small amount of training data.

  Further, according to the movement status recognition apparatus, method, and program of the present invention, information is efficiently extracted and combined from both the time series of image data and the time series of sensor data, and highly accurate movement status recognition is performed. The effect that it is realizable is acquired.

It is a block diagram which shows the structure of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the memory | storage format of video data DB. It is a figure which shows an example of the memory format of sensor data DB. It is a figure which shows an example of the storage format of annotation DB. It is a flowchart which shows the flow of a process of the video data pre-processing part of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the time series of the image data which the video data pre-processing part produced | generated from video data. It is a flowchart which shows the flow of a process of the sensor data pre-processing part of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the network structure of a DNN model. It is a flowchart which shows the flow of a process of the movement condition recognition DNN model unsupervised learning part of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process of the movement condition recognition DNN model supervised learning part of the movement condition recognition model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the memory | storage format of a movement condition recognition DNN model DB. It is a block diagram which shows the structure of the movement condition recognition apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process of the movement condition recognition apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process of the movement condition recognition part of the movement condition recognition apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment of the present invention, a case where the present invention is applied to a movement situation recognition model learning apparatus corresponding to a learning phase and a movement situation recognition apparatus corresponding to a recognition phase will be described as an example.

<Configuration of Moving Situation Recognition Model Learning Device According to Embodiment of the Present Invention>
First, the configuration of the movement situation recognition model learning device according to the embodiment of the present invention will be described. As shown in FIG. 1, the movement situation recognition model learning device 10 according to the embodiment of the present invention includes an input unit 20, a calculation unit 30, and an output unit 50.

  The calculation unit 30 includes a video data DB 32, a sensor data DB 34, a video data pre-processing unit 36, a sensor data pre-processing unit 38, an annotation DB 40, a movement status recognition semi-supervised DNNN model construction unit 42, and a movement status. A recognition DNNN unsupervised learning unit 44, a movement situation recognition DNNN supervised learning unit 46, and a movement situation recognition DNN model DB 48 are provided, and DNN for movement situation recognition using information of each DB. The model is output by the output unit 50. Here, it is assumed that the video data DB 32 and the sensor data DB 34 are constructed in advance so that the video data associated with the data ID can be associated with the sensor data in time series. Regarding the construction processing of the video data DB 32 and the sensor data DB 34, for example, the input unit 20 receives time-series pairs of video data and sensor data input by the system operator, and sets IDs that uniquely identify these pairs as data. The video data and sensor data input as IDs may be given to the time series and stored in the video data DB 32 and sensor data DB 34, respectively. The annotation DB 40 stores an annotation name for each data ID. Here, for example, the annotation is assumed to explain the movement status of the first-person viewpoint video data acquired by the glassware, and corresponds to window shopping, crossing a pedestrian crossing, or the like. As for the construction process of the annotation DB 40, as in the construction process of the video data DB 32 and the sensor data DB 34, for example, the input unit 20 accepts an annotation for each data ID input by the system operator and stores the input result in the DB. You just have to do it.

  In the embodiment of the present invention, the operation of the components shown in FIG. 1 is constructed as a program, and is installed and executed on a computer used as a movement situation recognition model learning device.

  The video data preprocessing unit 36 performs sampling and normalization on the time series of the image data represented by the video data stored in the video data DB 32.

  The sensor data preprocessing unit 38 performs normalization and feature vectorization on the time series of sensor data stored in the sensor data DB 34.

  The movement state recognition semi-supervised DNN model construction unit 42 receives the time series of the image data and the time series of the sensor data, extracts each feature of the image data and each feature of the sensor data, and extracts each feature of the image data. A DNN model for recognizing a movement situation from data abstracted from features and sensor data, and further from data abstracted from features of image data and features of sensor data, A DNN model for decoding the time series of decoded image data and the time series of decoded sensor data is constructed.

  The movement state recognition DNN model unsupervised learning unit 44 is based on the time series of image data that is the processing result of the video data preprocessing unit 36 and the time series of sensor data that is the processing result of the sensor data preprocessing unit 38. When the time series of image data and the time series of sensor data are input, the time series of decoded image data and the time series of decoded sensor data, the time series of image data, and the time series of sensor data are output from the model. The parameters of the DNN model are learned so as to match.

  The movement situation recognition DNN supervised learning section 46 includes model parameters learned by the movement situation recognition DNN model unsupervised learning section 44, a time series of image data that is a processing result of the video data preprocessing section 36, and sensor data. Based on the time series of the sensor data that is the processing result of the pre-processing unit 38, the time series of the image data, and the annotation indicating the movement status given in advance to the time series of the sensor data, The movement status recognized by the DNN model when the time series of sensor data is input matches the movement status indicated by the annotation, and is output from the model when the time series of image data and the time series of sensor data are input. Decoded image data time series and decoded sensor data time series, image data time series and sensor data time So that the string matches, to learn the parameters of the DNN model. The learned parameters of the DNN model are stored in the movement situation recognition DNN model DB 48.

<Operation of the movement situation recognition model learning device according to the embodiment of the present invention>
FIG. 2 is a flowchart of a model learning process routine executed by the movement situation recognition model learning device 10 according to the embodiment of the present invention. This will be specifically described below.

<Model learning processing routine>
In step S100, the video data preprocessing unit 36 receives data from the video data DB 32 and processes the data. Details of the processing will be described later. FIG. 3 shows an example of the data storage format of the video data DB 32. The video data is stored in a file compressed in the Mpeg4 format or the like, and each is associated with a data ID for associating with sensor data as described above. The video data is first-person viewpoint video data acquired through glassware or the like worn by a user, which is an example of a mobile object.

  In step S110, the sensor data preprocessing unit 38 receives data from the sensor data DB 34 and processes it. Details of the processing will be described later. FIG. 4 shows an example of the data storage format of the sensor data DB 34. The sensor data has elements such as date and time, latitude and longitude, X-axis acceleration, and Y-axis acceleration. Each sensor data has a unique series ID. Further, as described above, it holds a data ID for associating with video data. Each sensor data is data acquired by a wearable sensor attached to the user.

  In step S120, the movement state recognition semi-supervised DNNN model construction unit 42 constructs a DNN model. Details of the processing will be described later.

  In step S130, the movement state recognition DNN model unsupervised learning unit 44 processes the video data processed from the video data preprocessing unit 36, the sensor data processed from the sensor data preprocessing unit 38, and the movement state recognition semi-supervised DNN model. The DNN model is received from the construction unit 42, the parameters of the DNN model are learned, and are output to the movement situation recognition DNN model DB 48.

  In step S140, the moving state recognition DNN model supervised learning unit 46 constructs the video data processed from the video data preprocessing unit 36, the sensor data processed from the sensor data preprocessing unit 38, and the moving state recognition semi-supervised DNN model construction. The DNN model is received from the unit 42, the annotation data is received from the annotation DB 40, and the parameters of the DNN model are received from the movement situation recognition model DB 48. The parameters of the DNN model are learned and output to the movement situation recognition DNN model DB 48. FIG. 5 shows an example of the storage format of the annotation DB 40.

  FIG. 6 is a flowchart showing a subroutine executed by the video data pre-processing unit 36 for realizing step S100. This will be specifically described below.

  In step S200, video data is received from the video data DB 32.

  In step S210, each video data is converted into a time series of image data represented by pixel values of vertical × horizontal × 3 channels. For example, the vertical size is determined to be 100 pixels and the horizontal size is determined to be 200 pixels. FIG. 7 shows an example of time series of image data generated from video data. Each image data holds data ID corresponding to the original image data, each frame number, and time stamp information.

  In step S220, in order to reduce redundant data, N frames are sampled from the time series of image data at regular frame intervals.

  In step S230, each pixel value of the image data in each sampled frame is normalized so that the DNN model can easily handle the image data. For example, each pixel value is divided by the maximum value that the pixel can take so that the range of each pixel value is 0-1.

  In step S240, the video data expressed as time series of image data and the corresponding time stamp information are transferred to the movement situation recognition DNN model unsupervised learning section 44 and the movement situation recognition DNN model supervised learning section 46. .

  FIG. 8 is a flowchart showing a subroutine executed by the sensor data pre-processing unit 38 for realizing the step S110.

  In step S300, sensor data is received from the sensor data DB.

  In step S310, values such as acceleration in each sensor data are normalized so that the sensor data can be easily handled by the DNN model. For example, standardization is performed so that the average value of all sensor data is 0 and the standard deviation is 1.

  In step S320, each value normalized for each sensor data is combined to generate a feature vector.

  In step S330, the sensor feature vector and the corresponding date and time information are transferred to the movement situation recognition DNN model unsupervised learning section 44 and the movement situation recognition DNN model supervised learning section 46.

  FIG. 9 is an example of a network structure of the DNN model constructed by the DNN model construction unit 42 with the movement state recognition semi-supervised in one embodiment of the present invention. As an input, a matrix representing image data of each frame in the video data and a corresponding sensor data vector are received, and a probability of each movement situation is obtained as an output. The network structure of the DNN model is composed of the following units.

  The first unit is a convolution layer that extracts features from a matrix representing image data. Here, for example, the image is convolved with a 3 × 3 filter, or the maximum value in the specific rectangle is extracted (maximum pooling). For the convolution layer, a known network structure such as AlexNet (see Non-Patent Document 3) or a pre-learned parameter can be used.

  The second unit is a fully connected layer A that further abstracts the features obtained from the convolutional layer. Here, the input feature quantity is nonlinearly transformed using, for example, a sigmoid function.

  The third unit is a fully connected layer B that abstracts the sensor data vector to the same level as the image features. Here, like the all coupling layer A, the input is nonlinearly transformed.

  The fourth unit is Long-short-term-memory (LSTM) that further abstracts two abstract features as series data. Specifically, series data is sequentially received, and nonlinear transformation is repeatedly performed while circulating past abstracted information. The LSTM can use a known network structure (Non-patent Document 4) in which a forgetting gate is mounted.

  The fifth unit is an all-connected layer C that calculates the vector representing the probability for each movement situation by dropping the abstracted series feature into a dimension vector of the number of types of the movement situation. Here, nonlinear transformation is performed using a softmax function or the like so that the sum of all the elements of the input feature amount becomes 1.

  The sixth unit is a fully connected layer D that decodes features abstracted as sequence data by LSTM to the same level as the features input to the fully connected layer A. Here, like the all coupling layer A, the input is nonlinearly transformed.

  The seventh unit is a deconvolution layer that decodes the decoded feature values obtained from the fully connected layer D to the same level as the matrix representing the input image data. Here, for example, a 3 × 3 feature matrix is expanded to a large matrix size and then convolved, or a feature amount in the feature matrix is copied into an N × N rectangle (reverse pooling).

  The eighth unit is a fully connected layer E that decodes the features abstracted as sequence data by LSTM to the same level as the sensor data input to the fully connected layer B. Here, like the all coupling layer A, the input is nonlinearly transformed.

  The ninth is an output layer that associates each element of the probability vector with the movement state. For example, the first probability vector is associated with window shopping, and the second is associated with a crosswalk.

[Non-Patent Document 3] Krizhevsky, A., Sutskever, I. and Hinton, G. E .: ImageNet Classification with Deep Convolutional Neural Networks, pp. 1106-1114, 2012.

[Non-Patent Document 4] Felix A. Gers, Nicol N. Schraudolph, and Jurgen Schmidhuber: Learning precise timing with LSTM recurrent networks. Journal of Machine Learning Research, vol. 3, pp. 115-143, 2002.

  FIG. 10 is a flowchart showing a subroutine executed by the movement state recognition DNN model unsupervised learning unit 44 for realizing the step S130. Specifically, the following processing is performed.

  In step S400, the video data and the sensor data are associated with each other based on the time stamp of the received video data and the date / time information of the sensor data.

  In step S410, a DNN model having a network structure as shown in FIG.

  In step S420, model parameters of each unit in the network structure represented by the DNN model are initialized. For example, initialization is performed with a random number from 0 to 1.

  In step S430, model parameters of the DNN model are calculated using the video data and the sensor data. Specifically, a known technique such as back-propagation is used so that the mean square error between the input data, the output decoded image data matrix obtained from the input data and the decoded sensor data vector is minimized. Then, the model parameters of the DNN model are optimized.

  In step S440, the DNN model (network structure and model parameters) is output, and the output result is stored in the movement situation recognition DNN model DB 48. FIG. 12 shows an example of model parameters. Parameters are stored as matrices and vectors in each layer.

  FIG. 11 is a flowchart showing a subroutine executed by the movement state recognition DNN model supervised learning unit 46 for realizing the step S140. Specifically, the following processing is performed.

  In step S500, as in step S400, the video data and the sensor data are associated with each other based on the time stamp of the received video data and the date / time information of the sensor data.

  In step S510, similar to step S410, a DNN model having a network structure as shown in FIG.

  In step S520, the model parameters are received from the movement situation recognition DNN model DB 48 and set as the parameters of the DNN model.

  In step S530, model parameters are calculated using the video data, sensor data, and corresponding annotation data. Specifically, the mean square error between the output decoded image data matrix and the decoded sensor data vector obtained for the input data is minimized, and the output probability vector and the binary vector obtained from the annotation are A model parameter is optimized using a known technique such as backpropagation so that the cross-entropy error is minimized.

  In step S540, the movement situation recognition DNN model (network structure and model parameters) is output, and the output result is stored in the movement situation recognition DNN model DB 48. FIG. 12 shows an example of model parameters. Parameters are stored as matrices and vectors in each layer. In addition, for the output layer, text of the movement status corresponding to each element number of the probability vector is stored.

<Configuration of Movement Situation Recognition Device According to Embodiment of the Present Invention>
Next, the configuration of the movement status recognition apparatus according to the embodiment of the present invention will be described. As illustrated in FIG. 13, the movement status recognition apparatus 100 according to the embodiment of the present invention includes an input unit 120, a calculation unit 130, and an output unit 150.

  The input unit 120 receives a pair of video data and sensor data time series for a user to be recognized.

  The calculation unit 130 includes a video data pre-processing unit 136, a sensor data pre-processing unit 138, a movement situation recognition unit 140, and a movement situation recognition DNN model DB 148, and the video data received by the input unit 120 and The output unit 150 outputs a recognition result for the sensor data.

  In the embodiment of the present invention, the operation of the components shown in FIG. 13 is constructed as a program, and is installed and executed on a computer used as a movement status recognition apparatus.

  The movement state recognition DNN model DB 148 stores the same model parameters of the DNN model as the movement state recognition DNN model DB 48.

  The video data preprocessing unit 136 performs sampling and normalization on the time series of the image data represented by the video data received by the input unit 120 in the same manner as the video data preprocessing unit 36.

  The sensor data pre-processing unit 138 performs normalization and feature vectorization on the time series of sensor data received by the input unit 120 in the same manner as the sensor data pre-processing unit 38.

  The movement status recognition unit 140 stores the time series of image data that is the processing result of the video data preprocessing unit 136, the time series of sensor data that is the processing result of the sensor data preprocessing unit 138, and the movement status recognition DNN model DB 148. Based on the model parameters, the time series of image data and the time series of sensor data are input to the DNN model to recognize the movement status of the user to be recognized.

<Operation of the movement situation recognition apparatus according to the embodiment of the present invention>
FIG. 14 is a flowchart of a movement situation recognition processing routine executed by the movement situation recognition apparatus 100 according to one embodiment of the present invention. This will be specifically described below.

<Movement status recognition processing routine>
First, when the DNN model (network structure and model parameters) output by the movement situation recognition model learning apparatus 10 is input to the movement situation recognition apparatus 100, the inputted DNN model is converted into the movement situation by the movement situation recognition apparatus 100. It is stored in the recognized DNN model DB 148. Then, the movement status recognition apparatus 100 executes the following processes when a pair of video data and time series of sensor data is input.

  In step S150, the video data preprocessing unit 136 receives and processes video data as an input. Step S150 is realized by a flowchart similar to the flowchart of FIG.

  In step S160, the sensor data preprocessing unit 138 receives and processes sensor data as an input. This is realized by a flowchart similar to the flowchart of FIG.

  In step S170, the movement status recognition unit 140 receives processed video data from the video data preprocessing unit 136, sensor data processed from the sensor data preprocessing unit 138, and a learned DNN model from the movement status recognition DNN model DB148. The movement status recognition result is calculated and output by the output unit 150.

  FIG. 15 is a flowchart showing a subroutine executed by the movement status recognition unit 140 for realizing step S170. This will be specifically described below.

  In step S <b> 600, time series of video data and sensor data obtained by pre-processing input data is received from the video data pre-processing unit 136 and the sensor data pre-processing unit 138.

  In step S610, a learned DNN model (network structure and model parameters) is received from the movement situation recognition DNN model DB 148.

  In step S620, the probability for each moving situation is calculated from the time series of video data and sensor data using the DNN model.

  In step S630, the output unit 150 outputs the movement situation with the highest probability as the movement situation recognition result.

  As described above, according to the movement situation recognition model learning apparatus according to the embodiment of the present invention, the movement situation of the user is recognized from data obtained by abstracting each feature of the image data and each feature of the sensor data. And a DNN model for decoding the time series of the decoded image data and the time series of the decoded sensor data from data obtained by abstracting each characteristic of the image data and each characteristic of the sensor data. The time series of the decoded image data and the time series of the decoded sensor data, the time series of the image data, and the time series of the sensor data output from the DNN model when the time series of the image data and the time series of the sensor data are input. When the parameters of the DNN model are learned so that the time series matches, the time series of image data and the time series of sensor data are input. By learning the parameters of the DNN model so that the movement situation recognized by the N model matches the movement situation indicated by the annotation, information can be efficiently extracted and combined from both video data and sensor data, and With a small amount of training data, highly accurate movement situation recognition can be realized.

  In addition, by building and learning a DNN model using video data in addition to sensor data, and using the obtained DNN model for movement situation recognition, it becomes possible to recognize a movement situation of a user that could not be recognized conventionally.

  In addition, a convolution layer that can handle image features effective for user situation recognition, a fully connected layer that can abstract features at an appropriate level of abstraction, and LSTM that can efficiently abstract sequence data Therefore, the user's movement situation can be recognized with high accuracy.

  In addition, by using the sensor data and video data without annotation data to pre-learn the model parameters of the DNN model for movement situation recognition, the movement situation of the user can be recognized with high accuracy even with a small amount of learning data. .

  Also, the video data pre-processing unit can pre-process the video data such as sampling and normalization so that the DNN model can be handled easily. In addition, the sensor data pre-processing unit can pre-process the sensor data such as normalization and feature vectorization so that the DNN model can be easily handled.

  In addition, according to the movement situation recognition apparatus according to the embodiment of the present invention, by using the DNN model learned by the movement situation recognition model learning apparatus, the movement situation can be obtained with high accuracy from both video data and sensor data. It is possible to efficiently extract and combine information to recognize and realize highly accurate movement situation recognition.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, the case where the movement situation recognition model learning device and the movement situation recognition device are configured as separate devices has been described as an example, but the present invention is not limited to this, and the movement situation recognition model learning device and the movement situation recognition device You may make it comprise by one apparatus.

  Moreover, although the case where a user's movement condition is recognized was demonstrated to the example, it is not limited to this, You may make it recognize the movement condition of mobile bodies other than a user.

  Moreover, although the above-described movement situation recognition model learning apparatus and movement situation recognition apparatus have a computer system inside, if the “computer system” uses a WWW system, a homepage provision environment ( Or a display environment).

  Further, in the present specification, the program has been described as an embodiment in which the program is preinstalled. However, the program can be provided by being stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM. is there. It is also possible to distribute the program via a network.

DESCRIPTION OF SYMBOLS 10 Movement situation recognition model learning apparatus 20, 120 Input part 30, 130 Calculation part 36, 136 Image data pre-processing part 38, 138 Sensor data pre-processing part 42 Movement situation recognition semi-supervised DNNN model construction part 44 Movement situation recognition DNN model Unsupervised learning section 46 Movement situation recognition DNN model supervised learning section 50, 150 Output section 100 Movement situation recognition apparatus 140 Movement situation recognition section 150 Output section

Claims (7)

Using the time series of the image data of the camera mounted on the moving body and the time series of the sensor data of the sensor mounted on the moving body as inputs, each feature of the image data and each feature of the sensor data are extracted, and the image A DNN (Deep Neural Network) model for recognizing the movement status of the moving object from data obtained by abstracting each characteristic of data and each characteristic of sensor data, and further, each characteristic of the image data And a movement state recognition semi-supervised DNNN model construction unit for constructing a DNN model for decoding a time series of decoded image data and a time series of decoded sensor data from data obtained by abstracting the characteristics of each of the sensor data;
Based on the time series of the image data and the time series of the sensor data, the time series of the decoded image data output from the DNN model when the time series of the image data and the time series of the sensor data are input. And a moving state recognition DNN model unsupervised learning unit that learns parameters of the DNN model so that the time series of the decoded sensor data matches the time series of the image data and the time series of the sensor data;
The learned DNN model parameters, the time series of the image data and the time series of the sensor data, and the movement status given in advance to the time series of the image data and the time series of the sensor data are shown. Based on the annotation, the DNN model so that the movement situation recognized by the DNN model when the time series of the image data and the time series of the sensor data are input matches the movement situation indicated by the annotation. A movement situation recognition DNN supervised learning unit for learning the parameters of
A moving situation recognition model learning device including: A video data pre-processing unit that performs sampling and normalization on the time series of the image data;
A sensor data pre-processing unit that performs normalization and feature vectorization on the sensor data time series,
The movement situation recognition DNN model unsupervised learning unit learns the parameters of the DNN model using the processing result of the video data preprocessing unit and the processing result of the sensor data preprocessing unit,
The movement situation according to claim 1, wherein the movement situation recognition DNN model supervised learning section learns parameters of the DNN model using a processing result of the video data preprocessing section and a processing result of the sensor data preprocessing section. Recognition model learning device. 3. A time series of image data and a time series of sensor data for a moving object to be recognized are input to the DNN model learned by the moving state recognition model learning apparatus according to claim 1 or 2, and the movement of the moving object is performed. A movement situation recognition device including a movement situation recognition unit for recognizing a situation. A video data pre-processing unit that performs sampling and normalization on the time series of the image data;
A sensor data pre-processing unit that performs normalization and feature vectorization on the sensor data time series,
4. The movement according to claim 3, wherein the movement status recognition unit inputs the processing result of the video data preprocessing unit and the processing result of the sensor data preprocessing unit to the DNN model and recognizes the movement status of the moving body. Situation recognition device. The moving state recognition semi-supervised DNN model construction unit receives the time series of the image data of the camera mounted on the moving body and the time series of the sensor data of the sensor mounted on the moving body, and each feature of the image data In addition, a DNN (Deep Natural Network) model for recognizing the movement state of the moving body from data obtained by extracting each feature of the sensor data and abstracting each feature of the image data and each feature of the sensor data. Further, a DNN model for decoding a time series of decoded image data and a time series of decoded sensor data is constructed from data obtained by abstracting each characteristic of the image data and each characteristic of the sensor data,
When the movement state recognition DNN model unsupervised learning unit inputs the time series of the image data and the time series of the sensor data based on the time series of the image data and the time series of the sensor data, The parameters of the DNN model are learned so that the time series of the decoded image data and the time series of the decoded sensor data that are output match the time series of the image data and the time series of the sensor data,
The movement state recognition DNN supervised learning unit is configured to set the learned DNN model parameters, the time series of the image data and the time series of the sensor data, the time series of the image data, and the time series of the sensor data. The movement indicated by the annotation is recognized by the DNN model when the time series of the image data and the time series of the sensor data are input based on the annotation indicating the movement situation given in advance. A movement situation recognition model learning method for learning parameters of the DNN model so as to match a situation. The movement situation recognition unit inputs a time series of image data and a time series of sensor data about a moving object to be recognized to the DNN model learned by the movement situation recognition model learning method according to claim 5, A method for recognizing the movement status of a moving object   The program for functioning a computer as each part of the movement condition recognition model learning apparatus of Claim 1 or 2, or the movement condition recognition apparatus of Claim 3 or 4.