JP2020166315A - Learning data generation method, training method, prediction model and computer program - Google Patents

Abstract

To properly train a prediction model.SOLUTION: Time-series data indicating time-dependent changes of a parameter measured value which is measured by a sensor attached to a target and is related to movement of the target, and a classification of a state of the target, is acquired. A frequency spectra of the time-dependent changes of the measured value is calculated for the classification of the state. A peak frequency which is a maximum peak frequency for the classification of the state is identified. The lowest peak frequency among the peak frequencies of the all classifications of the state is identified. By using the lowest peak frequency, wider time width can be determined as the lowest peak frequency becomes low. Learning data that is image data of a graph indicating the time-dependent changes of the measured values in a period of a time width and includes image data in association with the classification of the state is generated.SELECTED DRAWING: Figure 4

Description

The present specification relates to techniques for training predictive models.

A technique for estimating the state of an object has been proposed using data acquired by sensors such as an acceleration sensor and an angular velocity sensor. For example, Patent Document 1 proposes the following techniques. That is, the normalized sensor data is converted into an image for each period, and a learning process by a CNN (convolutional neural network) is executed using the generated image group. Then, the time series data is converted into an image in units of a predetermined period, and the state of the target is estimated based on the determination result of determining the image based on the learning model.

Japanese Unexamined Patent Publication No. 2018-124639 JP-A-2017-157213 Japanese Unexamined Patent Publication No. 2012-170740

However, the rate of change of data may vary depending on various causes such as the state of the object. As a result, there are cases where the learning process using images cannot be performed properly.

The present specification discloses techniques for properly training predictive models.

The techniques disclosed herein can be realized as the following application examples.

[Application Example 1] A method of generating training data for training a prediction model, which is a measurement value measured by a sensor attached to the target and the measurement value of a parameter related to the movement of the target. Time-series data showing the classification of the target state and the change with time is acquired, the frequency spectrum of the change with time of the measured value is calculated for each of the classifications of the state, and the maximum is obtained for each classification of the state. By identifying the peak frequency, which is the frequency of the peak of the above, identifying the lowest peak frequency of all the peak frequencies of the above categories in the above state, and using the lowest peak frequency, the lowest peak frequency is low. A moderately wide time width is determined, and training data including the image data associated with the classification of the state, which is image data of a graph showing the time-dependent change of the measured value within the period of the time width, is generated. How to generate training data.

According to this configuration, the time width of the graph of the training data is determined to be wider as the lowest peak frequency is lower, so that the correspondence between the time-dependent change of the measured value and the classification of the target state is appropriately shown. Training data can be generated. Such learning data is suitable for training.

[Application Example 2] The learning data generation method according to Application Example 1, wherein the training data is generated in a plurality of periods having the time width, which are different from each other and have a time difference between two adjacent periods. A method for generating learning data, which comprises generating the learning data corresponding to each of the plurality of periods, wherein is smaller than the time width.

According to this configuration, a large number of training data can be easily generated.

[Application Example 3] The learning data generation method according to Application Example 1 or 2, wherein the time width is ½ or more and 10 times or less of the time of one cycle of the lowest peak frequency. , How to generate training data.

According to this configuration, it is possible to generate learning data that appropriately shows the correspondence between the time-dependent change of the measured value and the classification of the target state.

[Application Example 4] A training method for a prediction model, in which training data is generated according to the generation method according to any one of Application Examples 1 to 3, and the prediction model is trained using the training data.

According to this configuration, training data that appropriately shows the correspondence between the time-dependent change of the measured value and the classification of the state is used for the training of the prediction model, so that appropriate training is possible.

[Application Example 5] The training method according to Application Example 4, wherein the prediction model is a model of a neural network including one or more convolution layers.

According to this configuration, since the prediction model includes one or more convolutional layers suitable for the image data, appropriate training is possible by using the training data including the image data.

[Application Example 6] A predicted model that has been trained by the training method described in Application Example 4 or 5.

[Application Example 7] A computer program for a computer that generates training data for training a prediction model, which is a measured value measured by a sensor attached to the target and a parameter related to the movement of the target. A function of acquiring time-series data indicating the time-dependent change of the measured value and the classification of the target state, and a function of calculating the frequency spectrum of the time-dependent change of the measured value for each of the classification of the state. And the function of specifying the peak frequency which is the maximum peak frequency for each of the classifications of the state, the function of specifying the lowest peak frequency among the peak frequencies of all the classifications of the state, and the lowest. By using the peak frequency, it is a function of determining a wider time width as the lowest peak frequency is lower, and image data of a graph showing the time-dependent change of the measured value within the period of the time width, which is the state. A computer program that allows a computer to realize a function of generating training data including the image data associated with the classification.

The technique disclosed in the present specification can be realized in various aspects, for example, a training data generation method and a generation device, a prediction model training method and a training device, and a function of those methods or devices. It can be realized in the form of a computer program for realizing the above, a recording medium on which the computer program is recorded (for example, a recording medium that is not temporary), and the like.

It is explanatory drawing which shows the system of an Example. It is explanatory drawing of the example of the artificial neural network NN. It is a flowchart which shows the example of the training process. It is a flowchart which shows the example of the generation process of learning data. (A)-(E) are explanatory views showing an example of the state of the worker. It is a schematic diagram of the process using the sensor data. (A) is a graph showing an example of the correspondence between the lowest peak frequency and the extraction time width. (B) is a graph showing an example of the correspondence between the maximum period and the extraction time width. It is the schematic which shows the example of the image data. It is a flowchart which shows the example of the process of estimating a state.

A. First Example:
A1. Device configuration:
FIG. 1 is an explanatory diagram showing a system of an embodiment. The system 1000 is a system for estimating the state of the target. The object may be any object whose movement changes depending on the state, such as a machine (for example, a machine tool, a robot, an unmanned aerial vehicle (also called a drone)), a living thing (for example, an animal such as a dog, or a person). It is decided in advance. In this embodiment, the target is a worker who works in a factory.

The system 1000 includes a data processing device 200 and a sensor 300 connected to the data processing device 200. The data processing device 200 is, for example, a personal computer. The data processing device 200 includes a processor 210, a storage device 215, a display unit 240 for displaying an image, an operation unit 250 for accepting an operation by a user, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

The processor 210 is a device that performs data processing, for example, a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory.

The non-volatile storage device 230 stores the first program 231 and the second program 232, and the trained model 234. The trained model 234 is a predictive model trained to estimate the worker's condition using information from a sensor 300 attached to the worker, which is an example of the target (details will be described later). In this embodiment, the trained model 234 is a program module.

The processor 210 stores various intermediate data used for executing the first program 231 and the second program 232 and the trained model 234 in the storage device 215 (for example, the volatile storage device 220 or the non-volatile storage device 230). Or), temporarily store it.

The display unit 240 is a device for displaying an image, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device that receives operations by the user, such as a touch panel, buttons, and levers that are superposed on the display unit 240. The user can input various instructions to the data processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with another device (for example, a wired interface such as a USB interface or a wired LAN interface, or a wireless interface such as IEEE802.11 or Bluetooth®). In this embodiment, the communication interface 270 is a wireless interface. The data processing device 200 can communicate with the sensor 300 via the communication interface 270.

The sensor 300 is a sensor that measures parameters related to the movement of the target. In this embodiment, the sensor 300 is a motion detection sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. The sensor 300 measures six parameters composed of acceleration in each direction of the three axes orthogonal to each other and angular velocity centered on each of the three axes orthogonal to each other. The sensor 300 is attached to the worker's arm and detects the movement of the worker. The sensor 300 outputs data indicating six measured values of six parameters at a predetermined constant data rate (for example, several hertz). Hereinafter, the data output from the sensor 300 is also referred to as sensor data. The data processing device 200 can acquire time-series data of each measured value by accumulating the sensor data from the sensor 300 in the storage device 215.

FIG. 2 is an explanatory diagram of an example of an artificial neural network NN (hereinafter, also simply referred to as a neural network NN). The neural network NN is a neural network used to generate the trained model 234 (FIG. 1). In this embodiment, the neural network NN is a so-called convolutional neural network. A convolutional neural network has a plurality of layers including a convolutional layer. The details of the neural network NN will be described later.

A2. Neural network NN training:
FIG. 3 is a flowchart showing an example of training processing of the neural network NN. Hereinafter, the processor 210 of the data processing device 200 (FIG. 1) will start the process of FIG. 3 in response to an instruction from the training executor. The processor 210 executes the process of FIG. 3 according to the first program 231 for training.

In S110, the processor 210 generates training data. FIG. 4 is a flowchart showing an example of the learning data generation process. In S210, the processor 210 acquires the sensor data from the sensor 300 and the data indicating the state of the operator.

5 (A) -FIG. 5 (E) are explanatory views showing an example of the state of the operator. In this embodiment, the state of the worker 900 is classified into any of the five states from the first state SA to the fifth state SE. The first state SA (FIG. 5 (A)) is a state in which the worker 900 is walking without carrying luggage. The second state SB (FIG. 5 (B)) is a state in which the worker 900 is loading the luggage 920 on the trolley 910. The third state SC (FIG. 5 (C)) is a state in which the worker 900 is walking while pushing the dolly 910. In the third state SC, the trolley 910 may or may not carry the luggage 920. The fourth state SD (FIG. 5 (D)) is a state in which the worker 900 is unloading the luggage 920 from the trolley 910. The fifth state SE (FIG. 5 (E)) is a state in which the worker 900 is walking with the luggage 920.

FIG. 6 is a schematic diagram of processing using sensor data. The graph Ga is shown at the upper part of the figure. This graph Ga shows the time course of three types of acceleration ACx, ACy, and ACz measured by the sensor 300 and three types of angular velocities AVx, AVy, and AVz. The horizontal axis shows the time T, and the vertical axis shows the measured values of the parameters ACx, ACy, ACz, AVx, AVy, and AVz. The graph Ga also shows the correspondence between the time T and the state of the worker 900. In the example of FIG. 6, the state of the worker 900 changes in the order of SA, SC, SB, SC, SD, and SE. As described in FIGS. 5A-5E, the movements of the worker 900 differ from each other among the plurality of states SA, SB, SC, SD, SE. When comparing the entire pattern of aging of the parameters ACx, ACy, ACz, AVx, AVy, AVz (ie, the shape of the graph) between the states SA, SB, SC, SD, SE, at least some of them are mutually exclusive. Can be different.

In S210 (FIG. 4), the processor 210 acquires a time-series data set showing the time-dependent changes of the six measurements of the six parameters from the sensor 300 and the classification of the states of the worker 900. Such time series data sets can be acquired by various methods. In this embodiment, a time series data set is acquired according to the following procedure. The worker 900 (FIGS. 5A-5E) carries the luggage 920 with the sensor 300 attached to the arm. The data processing device 200 acquires sensor data from the sensor 300. The processor 210 of the data processing device 200 stores data indicating the correspondence between the sensor data and the time at which the sensor data is acquired in the storage device 215 (for example, the non-volatile storage device 230). The sensor 300 transmits sensor data at a constant data rate. Therefore, the processor 210 records the sensor data and the time in the storage device 215 at the data rate. In addition, the training executor shoots the worker 900 with a video camera (not shown). After the work by the worker 900 is completed, the training executor identifies the correspondence between the time and the state of the worker 900 by observing the image taken by the video camera. Then, the training executor inputs the correspondence relationship between the time and the state into the data processing device 200 by operating the operation unit 250. For example, the training executor inputs data indicating the change time, which is the time when the state changes, into the data processing device 200 in the time range from the work start time TS (FIG. 6) to the work end time TE. .. In the example of FIG. 6, five change times T1-T5 are input. Then, the training executor inputs data indicating the state of each period divided by the change time into the data processing device 200. In the example of FIG. 6, data indicating the states SA, SC, SB, SC, SD, and SE of the six periods P1-P6 are input. By using the input data, the processor 210 acquires a time-series data set showing the time-dependent changes of the six measured values of the six parameters and the classification of the state of the worker 900. Hereinafter, the continuous period P1-P6 in which one state continues is also referred to as a state period P1-P6.

In S220 (FIG. 4), the processor 210 generates a time series data set for each state classification. Specifically, the processor 210 sets each data of the parameters ACx, ACy, ACz, AVx, AVy, and AVz included in one continuous attention period in the same state as a time series data set in a state corresponding to the attention period. Get as. In the example of FIG. 6, the processor 210 acquires six time series data sets 811-816 corresponding to the six state periods P1-P6. The second time-series data set 812 corresponding to the second state period P2 and the second time-series data set 814 corresponding to the fourth state period P4 correspond to the same third state SC. However, since the second state period P2 and the fourth state period P4 are not continuous, these time series data sets 812 and 814 are generated as time series data sets different from each other.

In S230 (FIG. 4), the processor 210 acquires the frequency spectrum of each parameter by performing a Fourier transform on each parameter of each classification. The lower part of FIG. 6 shows an overview of the frequency spectrum set 821 obtained from the time series data set 811. In this embodiment, the processor 210 performs a fast Fourier transform of each of the six parameters ACx, ACy, ACz, AVx, AVy, AVz contained in the time series dataset 811. As a result, the six frequency spectra FCx, FCy, FCz, FVx, FVy, and FVz of the six parameters ACx, ACy, ACz, AVx, AVy, and AVz are calculated, respectively. The lower part of FIG. 6 shows an outline of a graph of frequency spectra FCx, FCy, FCz, FVx, FVy, and FVz. The horizontal axis represents the frequency F, and the vertical axis represents the intensity M.

The processor 210 calculates a frequency spectrum for each parameter of each time series data set generated in S220. In the example of FIG. 6, six frequency spectrum sets 821-826 corresponding to the six time series data sets 811-816 are calculated.

In S240 (FIG. 4), the processor 210 identifies the peak frequency, which is the maximum peak frequency of each frequency spectrum. The peak frequency is the frequency of the peak having the highest intensity. From the frequency spectra FCx-FCz and FVx-FVz of the frequency spectrum set 821 of FIG. 6, the peak frequency FP1-FP6 is specified, respectively.

In S250 (FIG. 4), the processor 210 identifies the lowest peak frequency from the peak frequency of each parameter of each classification. The lowest peak frequency is the lowest frequency among the plurality of peak frequencies identified in S240. In the example of FIG. 6, the lowest peak frequency is the lowest of the 36 peak frequencies identified from the 6 frequency spectrum sets 821-826.

In S260 (FIG. 4), processor 210 uses the lowest peak frequency to determine the time width. As will be described later, in order to generate training data, data of a plurality of periods having the same time width and different periods are extracted from the time series data. In S260, the time width of this period is determined. Hereinafter, the time width determined in S260 is referred to as an extraction time width.

FIG. 7A is a graph showing an example of the correspondence between the lowest peak frequency and the extraction time width. The horizontal axis represents the lowest peak frequency Fm, and the vertical axis represents the extraction time width Tw. FIG. 7B is a graph showing an example of the correspondence between the maximum period and the extraction time width. The horizontal axis represents the maximum period Tm, and the vertical axis represents the extraction time width Tw. The maximum period Tm is the time of one cycle of the lowest peak frequency Fm. As shown in FIG. 7A, in the present embodiment, the processor 210 determines the extraction time width Tw so that the higher the lowest peak frequency Fm, the narrower the extraction time width Tw. As shown in FIG. 7B, the extraction time width Tw is proportional to the maximum period Tm. The correspondence between the extraction time width Tw and the lowest peak frequency Fm (that is, the correspondence between the extraction time width Tw and the maximum period Tm) is predetermined. In this embodiment, the extraction time width Tw is the same as the maximum period Tm.

In S270 (FIG. 4), the processor 210 generates image data of each graph for a plurality of periods for each state classification. FIG. 8 is a schematic view showing an example of image data. At the upper part of the figure, the same graph Ga as the graph Ga of FIG. 6 is shown. Below the graph Ga, a plurality of different periods 700 are shown (hereinafter, the period 700 is also referred to as an extraction period 700). The time width of each of the plurality of extraction periods 700 is the same as the extraction time width Tw determined in S260 (FIG. 4). In this embodiment, the processor 210 evenly arranges a plurality of extraction periods 700 in each of the state periods P1-P6. The time difference between two adjacent extraction periods 700 is set to a predetermined time difference Td. In this embodiment, the time difference Td is smaller than the extraction time width Tw (however, the time difference Td may be equal to or larger than the extraction time width Tw). The extraction period 700, which overlaps a plurality of state periods, is not adopted. The reason for this is to enhance the learning effect. For example, the extraction period 700 that overlaps the first state period P1 and the second state period P2 is not adopted.

The processor 210 extracts the data within the extraction period 700 from the time series data set. Then, the processor 210 generates graph image data which is image data of a graph showing changes over time of the six parameters ACx, ACy, ACz, AVx, AVy, and AVz indicated by the extracted data. Hereinafter, the image represented by the graph image data is also referred to as a graph image. FIG. 8 shows a graph image 830t corresponding to the extraction period 700t. The graph image data shows the color values of the plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy perpendicular to the first direction Dx. The processor 210 generates graph image data for each of the plurality of extraction periods 700. Then, the processor 210 generates learning data including graph image data and teacher data indicating a state. The teacher data is data indicating a state corresponding to the graph image data, that is, a state corresponding to a state period including the extraction period 700. For example, the extraction period 700t of the graph image 830t of FIG. 8 is included in the first state period P1, and the state of the first state period P1 is the first state SA. Therefore, the teacher data corresponding to the graph image 830t indicates the first state SA.

In this embodiment, as shown in the graph image 830t of FIG. 8, the graph image is a line graph. The horizontal axis represents the time T, and the vertical axis represents the parameters ACx, ACy, ACz, AVx, AVy, AVz. The scale is omitted. The frame of the graph (including the vertical and horizontal axes) is omitted. However, the graph image may include a frame of the graph. The structure of the graph is common among a plurality of graph images. For example, the scale on the horizontal axis, the position and size of the drawing area of each parameter ACx, ACy, ACz, AVx, AVy, AVz and the scale on the vertical axis are common among a plurality of graph images and are predetermined. ing. The type of graph may be any type showing a change with time. For example, the graph may be a so-called scatter plot. Multiple data points on the scatter plot show a combination of time and measurements. Further, the data format of the graph image data is black-and-white binary bitmap data in this embodiment. The data format of the graph image data may be any other data format (for example, a bitmap of one or more color components such as RGB). The processor 210 stores the generated training data for each extraction period 700 in the storage device 215 (for example, the non-volatile storage device 230). Then, the process of FIG. 4, and eventually the process of S110 of FIG. 3 is completed.

In S120 (FIG. 3), the processor 210 (FIG. 1) trains the neural network NN (FIG. 2) using the plurality of training data.

The neural network NN includes an input layer 505, a first convolution layer 510, a first pooling layer 520, a second convolution layer 530, a second pooling layer 540, a first fully connected layer 550, and a second. It has a fully bonded layer 560 and a third fully bonded layer 570. These layers 505-570 are connected in this order. In this embodiment, the neural network NN is a program module and is included in the first program 231 (FIG. 1). The processor 210 realizes the functions of each layer 505-570 by proceeding with the processing according to the program module which is the neural network NN. Hereinafter, these layers 505-570 will be described in order.

The input layer 505 is a layer for acquiring data from the outside of the neural network NN. In this embodiment, the graph image data 830 included in the learning data is input to the input layer 505. The image data input to the input layer 505 is used as input information by the first convolution layer 510.

The first convolution layer 510 is a layer that performs an image convolution process. The convolution process is a process of calculating a value (also called a feature value) indicating a correlation between an input image, which is an input image, and a filter, while sliding the filter. The filter used in the image convolution process is also called a weight filter. The size of one weight filter is, for example, P × P pixels (P is an integer of 2 or more, for example, P = 5). The stride (ie, the amount of one movement of the filter) is, for example, 1. In this embodiment, the filter is slid over the entire input image so that feature values are calculated at all pixel positions in the input image. In this case, around the input image, the pixels are supplemented by zero padding. Instead, the filter may be slid to calculate feature values at the remaining pixel positions excluding the edges of the input image. In this way, the filter generates bitmap data (also called a feature map) of an image of the same size as the input image (or an image smaller than the input image). Specifically, a list of color values of P × P pixels of the portion corresponding to the position of the filter in the input image data is acquired. The inner product of the acquired list and the list of P × P weights of the filter is calculated. "Inner product + bias" is input to the activation function. Then, the calculation result of the activation function is used as the value of one element of the feature map. In this embodiment, a so-called ReLU (Rectified Linear Unit) is used as the activation function. Further, in this embodiment, Q weight filters are used (Q is an integer of 1 or more). Therefore, the processor 210, which functions as the first convolution layer 510, generates Q feature maps. The bias is prepared for each filter. Then, each element of the Q filters and the Q bias are updated by training.

The color value of each pixel of the input image may be generally represented by U color components (U is an integer of 1 or more, for example, 3 color components of RGB). In this case, one filter has P × P × U weights. Then, the inner product of the list of the color values of the P pixel × P pixel × U color component of the portion corresponding to the position of the filter on the input image and the list of the weights of P × P × U of the filter is calculated. ..

The feature map from the first convolution layer 510 is used as input information by the first pooling layer 520. Pooling is the process of reducing an image (here, a feature map). The first pooling layer 520 performs so-called MaxPooling. Max pooling is a process of reducing the feature map by so-called downsampling, in which the map is reduced by selecting the maximum value in the window while sliding the window. In this embodiment, the size of the window in the first pooling layer 520 is T × T pixels, and the stride is T (T is an integer of 2 or more, for example, T = 2). This will generate a map with 1 / T height and 1 / T width of the original map. The processor 210, which functions as the first pooling layer 520, generates Q reduced feature maps from the Q feature maps.

The feature map from the first pooling layer 520 is used as input information by the second convolution layer 530. The second convolution layer 530 performs the image convolution process according to the same procedure as the process by the first convolution layer 510. For example, one filter includes Q matrices corresponding to Q feature maps. Then, one filter generates one feature map from the Q feature maps. So-called ReLU is used as the activation function. The size of one matrix of one filter (that is, the number of elements in the first direction Dx and the number of elements in the second direction Dy) and the total number of filters may be various values. Also, zero padding around the feature map may be done and may be omitted instead. The stride can be of various values. Each element of each filter and each bias is updated by training.

The feature map from the second convolution layer 530 is used as input information by the second pooling layer 540. The second pooling layer 540 is max pooled according to the same procedure as the treatment by the first pooling layer 520. As a result, the second pooling layer 540 generates a reduced feature map. The window size and stride can be of various values.

The feature map from the second pooling layer 540 is used as input information by the first fully connected layer 550. The first fully connected layer 550 is a layer similar to the fully connected layer used in a general neural network. In the first fully connected layer 550, a list composed of a plurality of elements (also referred to as a first intermediate vector) is generated by using the input feature map. The number of elements of the first intermediate vector is predetermined and may be various values. In the first fully connected layer 550, the inner product of the list of all the elements of the feature map input to the first fully connected layer 550 and the list of the same number of weights as the number of elements of the feature map is calculated. "Inner product + bias" is input to the activation function. Then, the calculation result of the activation function is used as one element of the first intermediate vector. In this embodiment, so-called ReLU is used as the activation function. Also, the list of weights and the bias are prepared separately for each of the plurality of elements of the first intermediate vector. The processor 210, which functions as the first fully connected layer 550, generates the first intermediate vector by performing the above calculation. The weight list and bias are updated by training.

The first intermediate vector from the first fully connected layer 550 is used as input information by the second fully connected layer 560. The second fully connected layer 560 uses the input first intermediate vector to generate a second intermediate vector composed of a plurality of elements according to the same procedure as the processing by the first fully connected layer 550. The number of elements of the second intermediate vector is predetermined and may be various values. In the second fully connected layer 560, the inner product of the first intermediate vector and the list of weights having the same number of elements as the number of elements of the first intermediate vector is calculated. "Inner product + bias" is input to the activation function. Then, the calculation result of the activation function is used as one element of the second intermediate vector. In this embodiment, so-called ReLU is used as the activation function. The list of weights and the bias are prepared separately for each of the plurality of elements of the second intermediate vector. The processor 210, which functions as the second fully coupled layer 560, generates the second intermediate vector by performing the above calculation. The weight list and bias are updated by training.

The second intermediate vector from the second fully connected layer 560 is used as input information by the third fully connected layer 570. The third fully connected layer 570 uses the input second intermediate vector to generate an output vector composed of a plurality of elements according to the same procedure as the processing by the fully connected layers 550 and 560. In this embodiment, the number of elements in the output vector is "5", which is the same as the total number of classes classified by the neural network NN. In the third fully connected layer 570, the inner product of the second intermediate vector and the list of weights having the same number of elements as the number of elements of the second intermediate vector is calculated. "Inner product + bias" is input to the activation function. Then, the calculation result of the activation function is used as one element of the output vector. In this embodiment, a so-called softmax function (SoftMax) is used as the activation function. As is known, the softmax function calculates a value meaning a probability of zero or more and one or less. The five component values PA, PB, PC, PD, and PE of the output vector indicate the probability that the state of the worker 900 is the state SA, SB, SC, SD, and SE, respectively. For example, the first component value PA indicates the probability that the state is the first state SA. An output vector showing such a probability is also called conviction data. Note that the list of weights and the bias are prepared separately for each of the plurality of elements of the output vector. The processor 210, which functions as the third fully coupled layer 570, generates an output vector by performing the above calculations. The third fully connected layer 570 outputs the output vector to the outside of the neural network NN. Such a third fully connected layer 570 is an example of an output layer. The layers 510 to 560 between the input layer 505 and the output layer (here, the third fully connected layer 570) are also referred to as intermediate layers. The weight vector and bias are updated by training.

In S120 of FIG. 3, the neural network NN (FIG. 2) is trained using the plurality of training data described above. In the example of FIG. 8, supervised learning is performed using the learning data of each of the five states SA-SE. Specifically, the processor 210 inputs the graph image data 830 of the training data to the input layer 505. The processor 210 executes the calculations of the plurality of layers 505 to 570 of the neural network NN and calculates the output vector from the third fully connected layer 570. The processor 210 calculates the evaluation value 850 by comparing this output vector with the teacher data 840 of the learning data. The evaluation value 850 indicates a difference between the state estimation result by the neural network NN and the teacher data, that is, an error. The evaluation value 850 is calculated using, for example, a loss function. The loss function is a function that calculates the error obtained from the training data (that is, the error between the output data obtained from the graph image data and the teacher data). The evaluation value 850 is, for example, the sum of the errors of each of the plurality of training data. The processor 210 updates the various parameters (filter, weight vector, etc.) described above of the neural network NN so that the evaluation value 850 becomes smaller. Various functions can be adopted as the loss function. For example, known functions such as sum of squares error, cross entropy, and contrastive loss function may be used. As a training method, various methods can be adopted. For example, a method using a gradient descent method and an error back propagation method may be adopted.

In this way, the neural network NN is trained to classify the states into any of the five states SA-SE.

In S130 (FIG. 3), the trained neural network NN (FIG. 2) is stored in the storage device as the trained model 234. The trained model 234 is configured to proceed with processing using the parameters determined by the above training. In this embodiment, the processor 210 stores the trained model 234 in the non-volatile storage device 230 of the data processing device 200. When the storage is completed, the process of FIG. 3 ends.

A3. State estimation:
FIG. 9 is a flowchart showing an example of the process of estimating the state. Hereinafter, the processor 210 of the data processing device 200 (FIG. 1) will start the process of FIG. 9 in response to an instruction from the user. Processor 210 executes the process of FIG. 9 according to the second program 232 for state estimation.

In S310, the processor 210 (FIG. 1) acquires a time series data set showing the time course of the six measurements of the six parameters from the sensor 300. Such a time series data set is acquired by the same method as the method of S210 of FIG. However, in S310 of FIG. 9, the state of the worker 900 is unknown.

In S320, the processor 210 extracts the data within the extraction time width period from the time series data set acquired in S310. The extraction time width is the same as the extraction time width determined in S260 of FIG. Then, the processor 210 uses the extracted data to generate graph image data. The graph image data is graph image data showing changes over time of the six parameters ACx, ACy, ACz, AVx, AVy, and AVz. The generated graph image data represents a graph image in the same format as the graph image 830t of FIG. The graph image data is generated by the same method as the method of S270 of FIG.

In S330, the processor 210 (FIG. 1) inputs the graph image data generated in S320 into the trained model 234 (FIG. 2). In S340, processor 210 performs calculations for multiple layers 505-570 of the trained model 234. In S350, the processor 210 acquires confidence data from the third fully coupled layer 570. As described above, the five component values PA-PE of the certainty data indicate the probability that the state of the worker 900 is the state SA, SB, SC, SD, and SE, respectively. The conviction data is an example of data showing the estimation result of the state.

In S360, the processor 210 executes the process as the determination unit 580 (FIG. 2). The determination unit 580 is a processing unit that classifies the state of the worker 900 using the conviction data. For example, the processor 210 functioning as the determination unit 580 identifies the largest component value among the five component values PA-PE of the certainty data. Then, the processor 210 determines that the state is the state associated with the largest component value.

In S370 (FIG. 9), the processor 210 (FIG. 1) executes an output process of estimation data indicating the state (that is, the estimation result of the state) specified in S360. In this embodiment, the processor 210 causes the display unit 240 of the data processing device 200 to display an image showing the estimated state. The user can identify the state of the worker 900 by referring to the displayed image.

As a result, the process of FIG. 9 is completed. In S310, long-time time series data may be acquired as shown in the graph Ga of FIG. Then, in S320-S370, each state of the plurality of periods may be estimated as in the plurality of extraction periods 700 of FIG. By referring to such an estimation result, the user can identify the change over time in the state of the worker 900. The user can use the change of state over time for various purposes. For example, the user identifies the duration of the first state SA in which the worker 900 walks without luggage based on the time course of the state. Then, the user may change the work procedure of the worker 900 in the factory so that this time is shortened.

As described above, in this embodiment, the learning data is generated according to the procedure of FIG. Specifically, in S210, the processor 210 is a measured value measured by a sensor 300 attached to the worker 900 and has parameters ACx, ACy, ACz, AVx, AVy, AVz related to the movement of the worker 900. The time-series data showing the change with time of the measured value of the above and the classification of the state of the worker 900 is acquired. In S220 and S230, the processor 210 calculates the frequency spectrum of the time-dependent change of the measured value of each parameter for each state classification. In S240, the processor 210 identifies the peak frequency, which is the maximum peak frequency for each state classification. In S250, processor 210 identifies the lowest peak frequency of all classifications of state. In S260, the processor 210 uses the lowest peak frequency to determine the lower the lowest peak frequency, the wider the extraction time width Tw. In S270, the processor 210 generates learning data including image data of a graph showing a change over time of the measured value within the extraction period 700 of the extraction time width Tw and including image data associated with the classification of the state. The pattern of changes over time in the measured values of the parameters can vary depending on the classification of the state. When the peak frequency of the frequency spectrum of time change is low, the characteristics of the state are indicated by the time change of a long time width as compared with the case where the peak frequency is high. In this embodiment, as described above, the time width Tw of the graph of the training data is determined so as to become wider as the lowest peak frequency is lower. Therefore, the processor 210 determines the time-dependent change of the measured value and the operator 900. It is possible to generate learning data that appropriately shows the correspondence with the classification of states. The processor 210 can appropriately train the neural network NN by using such training data.

Further, as described in S270 (FIG. 4) and FIG. 8, the processor 210 has a plurality of extraction periods 700 having an extraction time width Tw, and learn data corresponding to each of the plurality of extraction periods 700 different from each other. , Generate. Here, the time difference Td between two adjacent extraction periods 700 included in the continuous state period (for example, the first state period P1) corresponding to the same state is smaller than the extraction time width Tw. Therefore, the processor 210 can acquire a large number of training data from the same time series data as compared with the case where the time difference Td is equal to or larger than the extraction time width Tw. The processor 210 can appropriately train the neural network NN by using a large amount of training data.

Further, as described with reference to FIGS. 7 (A) and 7 (B), in this embodiment, the extraction time width Tw is the same as the maximum period Tm. Therefore, the processor 210 can generate learning data that appropriately shows the correspondence between the time-dependent change of the measured value and the classification of the state of the worker 900.

Further, in this embodiment, the neural network NN is trained according to the procedure of FIG. Specifically, in S110, the processor 210 generates learning data according to the procedure of FIG. In S120, the processor 210 trains the neural network NN using the training data. As described above, since the learning data that appropriately shows the correspondence between the time-dependent change of the measured value and the classification of the state is used for the training of the neural network NN, appropriate training is possible.

Further, as shown in FIG. 2, the neural network NN includes convolution layers 510 and 530. The convolutional layer is suitable for processing image data. Since the neural network NN includes one or more convolutional layers suitable for the image data, appropriate training is possible with the training data including the graph image data.

Further, as described with reference to FIG. 3, the trained model 234 is a trained predictive model trained by the above training method. By using such a trained model 234, the processor 210 can appropriately estimate the state of the target (worker 900 in this embodiment).

B. Modification example:
(1) The correspondence between the extraction time width Tw (FIG. 4: S260, FIG. 8) and the lowest peak frequency Fm is different from the correspondence described in FIGS. 7 (A) and 7 (B). There may be various correspondences. For example, the extraction time width Tw may be different from the maximum period Tm. The extraction time width Tw may change stepwise with respect to a change in the lowest peak frequency Fm. In any case, it is preferable that the extraction time width Tw is determined to be a wider value as the lowest peak frequency Fm is lower. In order for the graph image to represent the characteristics of the target state, it is preferable that the extraction time width Tw is wide. For example, the extraction time width Tw is preferably 1/2 or more of the maximum period Tm, more preferably 2/3 or more of the maximum period Tm, and particularly preferably 3/4 or more of the maximum period Tm. Most preferably, the maximum period is Tm or more. Further, in order to acquire a lot of learning data from the same time series data, it is preferable that the extraction time width Tw is narrow. For example, the extraction time width Tw is preferably 10 times or less of the maximum period Tm, more preferably 8 times or less of the maximum period Tm, particularly preferably 6 times or less of the maximum period Tm, and the maximum. Most preferably, it is 4 times or less of the period Tm. Further, when the extraction time width Tw is narrow, the processor 210 can generate training data by using a faster and smaller portion of the same time series data as compared with the case where the extraction time width Tw is wide. For example, when the extraction time width Tw is 60 minutes, the data of the first 60 minutes of the time series data is required to generate the first training data. When the extraction time width Tw is 1 minute, the first training data can be generated with the data of the first 1 minute of the time series data. Further, in the time series data, the same state may continue for a short time. Here, when the extraction time width Tw is narrow, the processor 210 can generate a large amount of training data as compared with the case where the extraction time width Tw is wide. Further, the time difference Td between two adjacent extraction periods 700 may be equal to or larger than the extraction time width Tw.

(2) The parameter used for estimating the state of the target may be any parameter related to the movement of the target. That is, the parameter may be any parameter that changes depending on the movement of the target. For example, the parameters may include one or more parameters arbitrarily selected from a plurality of parameters including velocity, angular velocity, angular acceleration, geomagnetic direction, barometric pressure, and temperature. The geomagnetic direction, atmospheric pressure, and temperature can change according to the movement of the object. The measured values of velocity, angular velocity, and angular acceleration may all contain components of K axes (K is 1 or more and 3 or less) that are different from each other.

(3) The classified states are not limited to the five states SA-SE shown in FIGS. 5 (A) to 5 (E), and may be any plurality of states in which the movements of the objects are different from each other. For example, the state of the target may be estimated from a plurality of states including other states such as "a state in which the target is running" and "a state in which the target is stopped". In either case, the target state may be estimated from a plurality of predetermined states.

(4) The prediction model used for estimating the state of the target may be various other prediction models instead of the neural network NN of FIG. The predictive model may include one or more convolutional layers. The predictive model may also include one or more sets of convolutional layers and pooling layers connected behind the convolutional layers. Further, the prediction model may be composed of a plurality of fully connected layers without including the convolution layer. The prediction model may include various artificial neural networks. Further, the prediction model is not limited to the artificial neural network, and may include various other models. The predictive model may include, for example, one or more models arbitrarily selected from the group of artificial neural networks, hidden Markov models, and inference engines. In general, the prediction model may be various models that output data indicating the estimation result of the target state using graph image data.

(5) The output process of S370 (FIG. 9) may be an arbitrary process of outputting information indicating a state estimation result instead of the image output (specifically, display). For example, the processor 210 may execute a process of outputting a sound (for example, voice or chime) indicating an estimated state from a speaker. Further, the processor 210 may execute a process of outputting (that is, storing) data indicating an estimation result to an external storage device connected to the data processing device 200.

In either case, the state estimation result may be used in various processes, for example, for estimating the behavior pattern of the target. Then, the estimation result of the target behavior pattern may be used for improving the behavior pattern. For example, the target may be a robot that works in a factory. Then, the procedure of the robot's work may be improved based on the estimation result of the robot's behavior pattern. Further, the target may be an operator who operates the multifunction device. Then, the configuration of the multifunction device (for example, the position of the paper cassette, the position of the operation panel, etc.) may be changed based on the estimation result of the behavior pattern of the worker.

(6) In the examples of FIGS. 1, 3, and 4, the same data processing device 200 is an example of a generation device that generates learning data, and is also an example of a training device that trains a prediction model. Alternatively, the training data generation process may be performed by a device different from the device that trains the prediction model. Further, in the examples of FIGS. 1, 3, and 9, the same data processing device 200 is an example of an estimation device that estimates the state of the target using a prediction model. The process of estimating the state of the target may be executed by a device different from the generation device that generates the learning data. Also, the process of estimating the state of the object may be performed by a device different from the device that trains the prediction model.

(7) The generation device that generates the training data may be a device of a type different from that of the personal computer (for example, a multifunction device, a printer, a scanner, a digital camera, a smartphone, a server device connected to a network). Further, even if a plurality of devices (for example, a computer) capable of communicating with each other via a network share the function of the process of generating the learning data part by part and provide the function of generating the learning data as a whole. Good (systems with these devices correspond to learning data generators). The same applies to the training device that trains the prediction model using the training data and the estimation device that estimates the state of the target by inputting graph image data into the trained model.

In each of the above embodiments, a part of the configuration realized by the hardware may be replaced with software, and conversely, a part or all of the configuration realized by the software may be replaced with the hardware. May be good. For example, the trained model 234 of FIG. 1 may be realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) instead of the program module.

In addition, when a part or all of the functions of the present invention are realized by a computer program, the program is provided in a form stored in a computer-readable recording medium (for example, a non-temporary recording medium). be able to. The program may be used while being stored on the same or different recording medium (computer-readable recording medium) as it was provided. The "computer-readable recording medium" is not limited to a portable recording medium such as a memory card or a CD-ROM, but is connected to an internal storage device in the computer such as various ROMs or a computer such as a hard disk drive. It may also include an external storage device.

Although the present invention has been described above based on Examples and Modifications, the above-described embodiments of the invention are for facilitating the understanding of the present invention and do not limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention includes its equivalents.

200 ... data processing device, 210 ... processor, 215 ... storage device, 220 ... volatile storage device, 230 ... non-volatile storage device, 231 ... first program, 232 ... second program, 234 ... trained model, 240 ... display Unit, 250 ... Operation unit, 270 ... Communication interface, 300 ... Sensor, 505 ... Input layer, 510 ... First convolution layer, 520 ... First pooling layer, 530 ... Second convolution layer, 540 ... Second pooling layer 550 ... 1st fully connected layer, 560 ... 2nd fully connected layer, 570 ... 3rd fully connected layer, 580 ... determination unit, 700 ... extraction period, 700t ... extraction period, 811-816 ... time series data set, 821 -826 ... Frequency spectrum set, 830 ... Graph image data, 830t ... Graph image, 840 ... Teacher data, 850 ... Evaluation value, 900 ... Worker, 910 ... Cart, 920 ... Luggage, 1000 ... System, T ... Time, TS ... start time, T1-T5 ... change time, TE ... end time, P1-P6 ... state period, F ... frequency, M ... intensity, FP1-FP6 ... peak frequency, FCx-FCz, FVx-FVz ... frequency spectrum, SA -SE ... state, PA-PE ... component value, NN ... artificial neural network, Ga ... graph, Td ... time difference, Fm ... lowest peak frequency, Tm ... maximum period, Tw ... extraction time width, Dx ... first direction, Dy ... 2nd direction, ACx, ACy, ACz, AVx, AVy, AVz ... Parameters

Claims (7)

A method of generating training data for training predictive models.
Acquire time-series data indicating changes over time between the measured values of the parameters related to the movement of the target, which are the measured values measured by the sensor attached to the target, and the classification of the state of the target.
The frequency spectrum of the change with time of the measured value is calculated for each of the classifications of the state.
The peak frequency, which is the maximum peak frequency, is specified for each of the classifications of the above states.
The lowest peak frequency among the peak frequencies of all the classifications in the above state is identified.
By using the lowest peak frequency, the lower the lowest peak frequency, the wider the time width is determined.
A learning data including the image data of a graph showing the time-dependent change of the measured value within the period of the time width and associated with the classification of the state is generated.
How to generate training data. The method for generating learning data according to claim 1.
The generation of the training data is the learning data corresponding to each of the plurality of periods having the time width, which are different from each other and the time difference between the two adjacent periods is smaller than the time width. Including producing,
How to generate training data. The method for generating learning data according to claim 1 or 2.
The time width is ½ or more and 10 times or less the time of one cycle of the lowest peak frequency.
How to generate training data. It ’s a predictive model training method.
The training data is generated according to the generation method according to any one of claims 1 to 3.
Training the prediction model using the training data,
Training method. The training method according to claim 4.
The prediction model is a model of a neural network containing one or more convolutional layers.
Training method. It ’s a predictive model,
A trained predictive model trained by the training method according to claim 4 or 5. A computer program for computers that produces training data for training predictive models.
A function to acquire time-series data indicating a change over time between the measured value of the parameter related to the movement of the target, which is the measured value measured by the sensor attached to the target, and the classification of the state of the target. ,
A function of calculating the frequency spectrum of the time-dependent change of the measured value for each of the classifications of the state, and
A function to specify the peak frequency, which is the maximum peak frequency for each of the classifications of the state, and
A function to identify the lowest peak frequency among the peak frequencies of all the above categories in the above state, and
By using the lowest peak frequency, the function of determining a wider time width as the lowest peak frequency is lower, and
A function of generating learning data including the image data of a graph showing the time-dependent change of the measured value within the period of the time width and associated with the classification of the state.
A computer program that makes a computer realize.

Images