JP2019220084A - Analysis device, on-vehicle device, and pattern analysis support device - Google Patents

Abstract

To provide an analysis device, an on-vehicle device, and a pattern analysis support device that enable a pattern such as dangerous driving related to the running state of a vehicle to be easily specified.SOLUTION: An analysis unit 31 included in an analysis device 100 has sensor data 33 such as a vehicle speed and a three-axial direction acceleration input as time-series data, and performs GMM-HMM processing to output data classified into a plurality of classes. A processing unit 30 outputs reproduction of moving image data 32 and display of the sensor data 33 classified into classes in synchronization with each other to support an operation of specifying a running pattern performed by a human. A reference database in which reference data representing a model of the specified running pattern has been registered is mounted on the on-vehicle device or the like, and the running pattern of the vehicle is detected in real time with this database while the vehicle is being operated. Various running patterns difficult for a human and the like to become aware of merely with time-series variations in sensor output can be detected and utilized.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an analysis device, a vehicle-mounted device, and a pattern analysis auxiliary device that can be used for analyzing a traveling pattern of a vehicle.

  Conventionally, on-board devices such as drive recorders and digital tachographs installed in automobiles have the function of detecting specific driving conditions to assist crew members in driving safely and provide guidance on safe driving after driving. Is generally provided. For example, such a vehicle-mounted device can detect a state in which the traveling speed of a vehicle exceeds a legal speed or a speed limit, or a sudden steering operation, record the state as operation data, or output an alarm. . In recent years, various technologies have been studied.

  For example, the detection device of Patent Literature 1 discloses a technique for accurately detecting “near-miss”. Specifically, the state value acquiring unit is a value that changes in accordance with the operation amount of the brake while the operation of the brake is continued, and is a value that increases similarly as the operation amount of the brake increases. Get a state value. The determination time detection unit detects a determination time that is a time from when the brake operation is started to when the state value becomes equal to or more than a predetermined state threshold. The determining unit determines that an attention event, which is an event requiring attention in driving the vehicle, has occurred when the determination time is less than a predetermined determination threshold.

  The driving behavior pattern recognition device of Patent Document 2 shows a technology for continuous and real-time recognition processing. Specifically, when a driving behavior pattern is recognized from a driving operation amount by a hidden Markov model, a specific driving operation amount is set for each driving behavior pattern to be recognized, and the driving operation amount satisfies a predetermined condition. In such a case, the recognition start and end are determined.

  Patent Literature 3 discloses a method of generating a turning prediction or a prediction. Specifically, environment layout information of a driving environment in which the first vehicle is running is received, a current position of the first vehicle is received, one or more other vehicles are detected, and an additional environment layout from another vehicle is detected. The information is received and a model is constructed that includes the driving environment, the first vehicle, and one or more other vehicles. The model may be based on the environment layout information and the additional environment layout information, and may indicate a driver's intention of one of these other vehicles. The prediction can be generated based on a model, where the model is based on a hidden Markov model, a support vector machine, a dynamic Bayesian network, or a combination thereof.

JP-A-2017-107483 JP-A-11-99849 JP-A-2017-27599

  However, the “close call” that can be detected by the technique of Patent Literature 1 is only those that have a threshold or the like corresponding to a predetermined condition. That is, since the result of comparing the speed change and the acceleration change with the threshold value while the brake operation is continued is output, the detection condition is determined by the predetermined threshold value. Further, a sensor for detecting the on / off of the brake pedal is required.

  Also, in the technique of Patent Document 2, it is impossible to detect a driving behavior pattern other than a predetermined driving behavior pattern. That is, when the actual driving operation amount satisfies the conditions of the operation amount determined in advance for each driving action pattern, the recognition start and end determinations are performed. The value must be specified and registered.

  Further, the technique of Patent Document 3 indicates that turning prediction is performed based on a vehicle model, but it is unknown whether a realistic turning prediction can be actually performed. Further, it is unknown what kind of feature amount is used to construct a model, and the analysis contents of the prediction and its result are also unknown.

  In the prior art described above, it is possible to detect driving that affects safe driving, such as excessive speed or sudden steering in driving a car, but it is not possible to detect a detailed driving operation pattern or running pattern. is not. In addition, for example, when trying to identify a pattern corresponding to dangerous driving based on operation data such as a moving image obtained by shooting with an in-vehicle camera, human analysis will be involved and data analysis will be performed. There is a problem that the labor of such work becomes enormous.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an analysis device, an on-board device, and a pattern that are useful for facilitating identification of a pattern such as a dangerous driving related to a traveling state of a vehicle. An analysis assisting device is provided.

In order to achieve the above-described object, an analysis device, an in-vehicle device, and a pattern analysis auxiliary device according to the present invention are characterized by the following (1) to (5).
(1) a data acquisition unit that acquires a plurality of types of time-series sensor data detected on an arbitrary vehicle;
For a plurality of types of time-series sensor data acquired by the data acquisition unit, a mixed Gaussian distribution, and a data analysis unit that applies a hidden Markov model and outputs classification data classified into a plurality of classes as a result of the analysis.
A classification data storage unit that stores the classification data output by the data analysis unit;
An analysis device, comprising:

  According to the analyzer having the configuration (1), it is possible to acquire classification data classified into a plurality of classes from a plurality of types of time-series sensor data. That is, by applying a Gaussian mixture model (GMM) to the time-series sensor data, it is possible to express the observed value change in the sensor output in various time-series packets. Furthermore, by applying a Hidden Markov Model (HMM), time-series data can be regarded as a set of patterns in which various packets are arranged in a specific combination. By using a model created by unsupervised learning based on GMM-HMM, a packet can be classified as a class, and a time change of the class can be output in a time series. By using the results of the classified classes, it becomes easy to identify patterns such as dangerous driving related to the running state of the vehicle.

(2) The time-series sensor data acquired by the data acquisition unit includes an image captured by a vehicle-mounted camera,
An image recognition unit that recognizes the content of the image,
The data analysis unit is based on the recognition result of the image recognition unit, and outputs data of a plurality of classified classes from a time-series image as a result of the analysis,
The analyzer according to (1), wherein:

  According to the analyzer having the above configuration (2), it is possible to reduce the burden on an operator or the like who specifies the traveling pattern of the vehicle. For example, since other vehicles, pedestrians, bicycles, signs, road markings, and the like included in the image can be specified by image recognition and reflected in the class classification, the traveling pattern of the vehicle can be easily specified from the classification data.

(3) The plurality of types of time-series sensor data acquired by the data acquisition unit include at least a traveling speed of the vehicle and a plurality of accelerations applied to the vehicle in different directions,
The data analysis unit outputs classification data classified into a plurality of classes of a number larger than the type of the time-series sensor data,
The analyzer according to the above (1) or (2), characterized in that:

  According to the analyzing device having the configuration of the above (3), a database that can be used for estimating various running patterns of the vehicle can be easily constructed. That is, various behaviors of the vehicle can be detected based on a combination of the traveling speed of the vehicle and the acceleration in a plurality of directions. Further, by increasing the number of classes to be set in the output classification data beyond the number of sensors, the classification data can be output in a state closer to a specific traveling pattern of the vehicle. This facilitates the specification of the traveling pattern.

(4) An in-vehicle device including a plurality of sensors for detecting a state related to a traveling state of the own vehicle, and processing a signal output from the plurality of sensors to estimate a current traveling pattern,
For a plurality of types of time-series sensor data detected on an arbitrary vehicle, mixed Gaussian distribution, and classification data classified into a plurality of classes as a result of analysis applying a hidden Markov model, or included in the classification data A reference data holding unit that holds in advance analysis pattern data that specifies a traveling pattern of the vehicle,
Signals output by a plurality of sensors mounted on the vehicle, a pattern estimating unit that estimates the current traveling pattern based on the data held by the reference data holding unit,
An on-vehicle device comprising:

  According to the in-vehicle device having the above configuration (4), the analysis pattern data held by the reference data holding unit is obtained by analyzing a plurality of types of time-series sensor data by applying a mixed Gaussian distribution and a hidden Markov model. Reflects the results of Therefore, even if the pattern is difficult to find from general time-series changes, the pattern estimating unit can accurately estimate the current traveling pattern of the own vehicle.

(5) A pattern analysis assisting device that assists in specifying a traveling pattern in an arbitrary vehicle,
A data input unit that inputs a result of analyzing a plurality of types of time-series sensor data including the time-series image captured by the on-board camera and the traveling speed of the own vehicle as processing target data,
A data display unit that simultaneously displays the processing target data input by the data input unit for each type,
For a plurality of the processing target data displayed by the data display unit, associating a display range with each other, and a display control unit that synchronizes the display time with each other,
An operation unit that receives a change of a display range or a display time in the data display unit, and an input operation of information for specifying a traveling pattern;
A pattern analysis assisting device comprising:

  According to the pattern analysis assisting device having the configuration of the above (5), it is possible to support the operation of a human (operator) for specifying the traveling patterns of various vehicles. In particular, the display range and the time of the time-series sensor data such as the time-series image and the traveling speed are automatically displayed in a synchronized state, so that the worker associates the state of the image with the state of the traveling speed and the like. The work of comparison and judgment can be easily performed.

  ADVANTAGE OF THE INVENTION The analysis apparatus of this invention, an in-vehicle device, and a pattern analysis assistance apparatus can make it easy to identify the pattern of dangerous driving etc. related to the running state of a vehicle.

  The present invention has been briefly described above. Further, details of the present invention will be further clarified by reading through the modes for carrying out the invention described below (hereinafter, referred to as “embodiments”) with reference to the accompanying drawings. .

FIG. 1 is a block diagram illustrating a configuration example of an in-vehicle device that generates operation data. 2 (a), 2 (b), 2 (c), and 2 (d) are graphs showing specific examples of the time-series sensor data. FIG. 2 (a) shows the vehicle speed and FIG. b) shows the x-axis acceleration, FIG. 2 (c) shows the y-axis acceleration, and FIG. 2 (d) shows the time-series change of the z-axis acceleration. FIG. 3 is a block diagram illustrating a configuration example of the analysis device. FIG. 4 is a flowchart illustrating a processing procedure in the analysis device. FIG. 5 is a flowchart showing an outline of a processing procedure by the GMM-HMM shown in FIG. FIG. 6 is a diagram illustrating an outline of the GMM-HMM. FIG. 7 is a graph showing an example of a Gaussian distribution curve in a GMM-HMM analysis result. FIG. 8 is a graph showing an example of the result of applying the GMM-HMM analysis to the time-series speed data. FIG. 9 is a front view showing a data display example regarding data after the GMM-HMM processing. FIG. 9A is a data display example regarding vehicle speed data, and FIG. 9B is a data table regarding x-axis acceleration data. FIG. 9C is a front view illustrating a data display example regarding y-axis acceleration data, and FIG. 9D is a front view illustrating a data display example regarding z-axis acceleration data. FIG. 10 is a front view showing a screen display example of an application operating on the analysis device. FIG. 11 is a front view showing an example of the characteristics of the right / left turn pattern in the vehicle speed display window of the application. FIG. 12 is a flowchart illustrating an example of characteristic processing in the analysis device. FIG. 13 is a block diagram illustrating a configuration example of an on-vehicle device having a traveling pattern detection function. FIG. 14 is a block diagram illustrating a configuration example of the analysis device. FIG. 15 is a schematic diagram illustrating a configuration example of data processed by the analysis device illustrated in FIG.

  Specific embodiments of the present invention will be described below with reference to the drawings.

<Configuration example of on-board unit that generates operation data>
FIG. 1 shows a configuration example of the vehicle-mounted device 10 that generates operation data. The vehicle-mounted device 10 shown in FIG. 1 is used to collect operation data required for analyzing a traveling pattern related to the traveling state of the automobile. The on-vehicle device 10 is used while being mounted on, for example, a truck, a bus, a taxi vehicle, or the like. Instead of the vehicle-mounted device 10, a general vehicle-mounted device such as an existing digital tachograph or drive recorder can be used as it is or can be used with additional functions.

  The vehicle-mounted device 10 shown in FIG. 1 includes a CPU 11, interfaces (I / F) 12 to 15, a communication interface 16, a card interface 17, and an acceleration sensor 18. Further, a vehicle-mounted camera 21 is connected to an input of the interface 12, a GPS receiver 22 is connected to an input of the interface 13, and a vehicle speed sensor 23 is connected to an input of the interface 14. The acceleration sensor 18 is connected to an input of the interface 15.

  The CPU 11 is a semiconductor device having a built-in microcomputer function, and realizes functions required for the vehicle-mounted device 10 by executing a software program incorporated in advance. That is, the CPU 11 can collect, record, and transmit various information related to the operation of the own vehicle during the operation of the automobile.

  The in-vehicle camera 21 is mounted on the vehicle body in a state capable of continuously photographing a scene including a road, a sign, another vehicle, a pedestrian, a bicycle, and the like in front of the own vehicle in the same manner as a forward view of a driver of the own vehicle. Fixed. Alternatively, it is installed in the vehicle interior so that the driving operation and behavior of the driver in the vehicle interior can be continuously photographed. Further, a plurality of vehicle-mounted cameras 21 may be connected to the vehicle-mounted device 10 at the same time.

  The GPS receiver 22 can calculate the current position (latitude / longitude) of the vehicle substantially in real time based on the reception time of radio waves received from a plurality of GPS (Global Positioning System) satellites.

  The vehicle speed sensor 23 is a standard sensor mounted on the vehicle body side of the host vehicle, and can output a fixed number of pulse signals as a vehicle speed pulse, for example, every time the transmission output shaft or the wheel makes one revolution. it can. Therefore, it is possible to calculate the vehicle speed [km / h] and the traveling distance [km] based on the vehicle speed pulse.

The acceleration sensor 18 has a built-in sensor that independently detects accelerations [m / s ² ] in three directions of the x-axis, y-axis, and z-axis of the vehicle. For example, the x-axis is set so as to coincide with the traveling direction of the host vehicle (front-back direction), the y-axis is set in the vertical direction of the host vehicle, and the z-axis is set so as to match the horizontal direction of the host vehicle.

  The communication interface 16 can realize wide-area wireless communication by performing wireless communication with a public wireless base station that provides a communication function such as a mobile phone. In the present embodiment, it is assumed that the cloud 24 that manages information collected by the vehicle-mounted device of each vehicle exists on a server of a predetermined data center. That is, the vehicle-mounted device 10 can be constantly connected to the cloud 24 by wireless communication via the communication interface 16 during the operation of the vehicle, and can transmit the collected operation data to the cloud 24 and upload it.

  The card interface 17 has a slot capable of detachably holding a recording medium 25 such as an SD card. The CPU 11 can access the recording medium 25 via the card interface 17 to read and write data.

  The CPU 11 shown in FIG. 1 includes: moving image data corresponding to the image captured by the on-board camera 21; the current position calculated by the GPS receiver 22; the vehicle speed calculated based on the output of the vehicle speed sensor 23; Can be collected as operation data. The operation data collected by the CPU 11 is transmitted to the cloud 24 and recorded on the recording medium 25.

<Specific examples of time-series sensor data>
2A to 2D show specific examples of the time-series sensor data handled by the analysis device. 2A shows the vehicle speed, FIG. 2B shows the x-axis acceleration, FIG. 2C shows the y-axis acceleration, and FIG. 2D shows the time-series change of the z-axis acceleration. The horizontal axes in FIGS. 2A to 2D represent the same time. That is, these four time-series data are measured in the same time zone.

  For example, the operation data stored in the cloud 24 or the recording medium 25 shown in FIG. 1 includes time-series data indicating a change in vehicle speed, and time-series data indicating a change in acceleration in each of x, y, and z directions. Since sequence data is included, time-series sensor data as shown in FIGS. 2A to 2D can be obtained. Further, since the time-series sensor data recorded on the cloud 24 and the recording medium 25 includes time information, the vehicle speed and the axial acceleration at the same time as shown in FIGS. Four types of information can be obtained simultaneously.

<Example of configuration of analyzer>
FIG. 3 shows a configuration example of an analysis device 100 used to analyze a traveling pattern based on the operation data of a vehicle. The analysis device 100 is assumed to be installed on a personal computer (PC) installed in a management section of a company that manages a large number of vehicles. It is also possible to configure.

  The analysis device 100 illustrated in FIG. 3 includes a processing unit 30, a display unit 34, a data storage unit 38, and an operation unit 39. The processing unit 30 includes an analysis unit 31. The processing unit 30 is configured by, for example, a main body of a personal computer, and the function of the analysis unit 31 is configured as software executable by the personal computer. The display unit 34 has a two-dimensional screen that can display various kinds of visible information. The data storage unit 38 is configured by a storage device such as a hard disk. The operation unit 39 is configured as an input device capable of accepting a user's input operation, such as a mouse, a keyboard, and a touch panel.

  The operation data of the vehicle to be analyzed by the analysis device 100 illustrated in FIG. 3 includes moving image data 32 and sensor data 33. For example, the operation data acquired by the cloud 24 from the vehicle-mounted device 10 shown in FIG. 1 or the whole or a part of the operation data recorded on the recording medium 25 is input to the analysis device 100 as the moving image data 32 and the sensor data 33. Can be. In this case, the input moving image data 32 and sensor data 33 are operation data collected in the same time zone and include time information. The sensor data 33 includes four types of time-series data of vehicle speed data and acceleration data in the x, y, and z axes. In addition, the operation data recorded during the operation of an arbitrary vehicle, not limited to a specific vehicle, can be an analysis target of the analysis device 100.

<Processing procedure of analyzer>
FIG. 4 shows an outline of a processing procedure in the analyzer 100 shown in FIG. The processing procedure of FIG. 4 will be described below.

  The analysis device 100 inputs moving image data 32 and sensor data 33 from, for example, the recording medium 25 or the cloud 24 shown in FIG. 1 as data to be analyzed (S01).

  The analysis device 100 converts the sensor data 33 input in S01 into time-series data in S02. That is, the input data is rearranged such that it is arranged in the order of the time at which it was collected or the time at which it was recorded. Also, the time of each data is managed so that the time of the vehicle speed data and the time of the acceleration data in each of the x, y, and z axes are aligned.

  The analysis unit 31 of the analysis device 100 applies the GMM-HMM analysis to the four types of time-series data of the sensor data 33 in S03.

  The analysis device 100 displays the analysis result of S03 on the four types of time-series data of the sensor data 33 on the screen of the display unit 34 in the next S04, and also displays the moving image data 32 at the same time. The data of the analysis result of S03 is stored in the data storage unit 38.

  The analysis device 100 performs a process for specifying the traveling pattern of the vehicle in S05. In the present embodiment, an operator, that is, a person performs comparison and determination for specifying a traveling pattern, but the analysis device 100 performs control for supporting comparison and determination of a person. This control will be described later in detail. Further, data to be converted from the classified classes into the respective driving patterns specified in S05 is stored in the data storage unit 38.

<Description of GMM-HMM analysis>
In the present embodiment, in order to express models representing various running patterns of the vehicle, the time series data of the sensor data 33 is analyzed as a set of patterns in which various packets are arranged in a specific arrangement. I have. Then, the analysis is performed so that the time-series packets are represented by the Gaussian mixture model (GMM), and the arrangement of the packets is represented by the hidden Markov model (HMM).

  FIG. 5 shows an outline of a processing procedure by the GMM-HMM shown in FIG. FIG. 6 shows an outline of the GMM-HMM. The processing procedure of FIG. 5 will be described below.

<S31> The number of hidden states of the HMM is set. The analysis target data input to the GMM-HMM analysis is four-dimensional × time t-dimensional data generated from four types of time-series velocity data and time-series acceleration data in each of x, y, and z directions. Further, when the number of elements of the Gaussian distribution is set to 3 in the GMM-HMM analysis as in an example described later, empirically, the maximum likelihood estimation is performed up to the number of m × n states where the number of dimensions of the data is m × the number of elements of the Gaussian distribution. Are easily converged, and in this embodiment, the state variables in the GMM-HMM analysis are set to 12 states. That is, in the present embodiment, since the state variables are set to 12 states as described above, the analysis result of the GMM-HMM is output in a state of being classified into 12 classes. Note that the number of states can be increased or decreased as necessary.

<S32> Each GMM parameter is set.
The Gaussian mixture distribution p (x) is expressed by the following equation (1).
Where π _k : mixing coefficient (probability of selecting the kth Gaussian distribution)
μ _k : mean Σ _k : variance k: parameter representing each element of Gaussian distribution Here, the initial value of the GMM parameter N _k constituting each HMM state that is hidden and cannot be observed by the HMM is set. Also, an initial value of a series of HMM states that cannot be observed is set.

<S33> An unobservable state of the HMM constituted by the GMM parameters is input, and an HMM sequence is obtained. In the Hidden Markov Model (HMM), it is assumed that time-series data (observed variables) that are actually observed are generated according to the state of an unobservable state (latent variable). The latent variable Z is set to “Z = [z ₁ , z ₂ ,..., Z _n ]”, and a set of parameters constituting the latent variable is set to θ. When the observation variable X of “X = [x ₁ , x ₂ ,..., X _n ]” is observed, a likelihood function represented by Expression (2) is obtained. A parameter for increasing the likelihood is obtained by the EM algorithm. Given a set X of observed values, a set Z of latent variables, and a set θ of parameters, a likelihood function p (X | θ) is expressed by the following equation (2).
FIG. 7 shows an example of a Gaussian distribution curve obtained from the GMM-HMM analysis result. In the example shown in FIG. 7, it is assumed that the number of elements (the maximum value of k) of a one-dimensional Gaussian distribution is three. That is, in the example of FIG. 7, a complex curve corresponding to the observed value Xn is represented by the first Gaussian distribution (normal distribution) represented by the mixing coefficient π ₁ , the average μ ₁ , and the variance Σ ₁ , the mixing coefficient π ₂ , A second Gaussian distribution represented by mean μ ₂ and variance Σ ₂ and a third Gaussian distribution represented by mixing coefficient π ₃ , mean μ ₃ and variance _{３ 3} are represented as a model of a mixed curve. . FIG. 8 shows a more specific example of the GMM-HMM analysis result. FIG. 8 illustrates a result of applying the GMM-HMM analysis to the time-series speed data included in the sensor data 33. In the example of FIG. 8, it is assumed that the number of elements (the maximum value of k) of the Gaussian distribution is 3, and the width (window length) is the data measurement time. When the window length is small, the energy resolution decreases and the analysis accuracy decreases. When the window length is wide, the time resolution decreases and the analysis time response decreases.Therefore, it is necessary to select an optimal value according to the purpose. There is. As shown in FIG. 8, a curve of a complicated change in the time-series speed data can be represented as a model in which a plurality of Gaussian distributions are mixed. Although not shown, the GMM-HMM analysis is simultaneously applied to the time-series data representing the acceleration in each of the x, y, and z directions, and is expressed as a model in which a plurality of Gaussian distributions are mixed as in FIG. Therefore, n-dimensional data is actually represented by an n-dimensional Gaussian mixture distribution.

<S34> The value of the likelihood is obtained from the likelihood function represented by the equation (2), and if the value is equal to or smaller than the threshold, S33 is repeated. If the value is equal to or larger than the threshold, it is determined as an optimized GMM parameter (S35). .

From the determined GMM parameters and sensor time-series data, it is possible to estimate an unobservable state of the HMM, which is a latent variable of the sensor time-series data, using the Viterbi algorithm. An unobservable state of a different HMM is called a class.
FIGS. 9A to 9D show display examples of states in which the estimated HMM cannot be observed. 9A shows a display example of vehicle speed data, FIG. 9B shows a display example of x-axis acceleration data, FIG. 9C shows a display example of y-axis acceleration data, and FIG. 9D shows z-axis acceleration data. Are shown below.

  In the example shown in FIG. 9A, the class of the HMM state of each of the time-series data of the vehicle speed is represented so as to be visually distinguishable to humans by using marks having different shapes, and the time-series change is represented by: Each of the marks is displayed in the classification data display area 41. The correspondence between the type of mark assigned to each time-series data and the class is displayed on the class display section 41a. In the example of FIG. 9A, twelve classes x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, and x12 of the data classified in a state where the HMM cannot be observed are classes. It is displayed on the display unit 41a. In the classification data display area 41, the horizontal axis represents time, and the vertical axis represents vehicle speed [km / h]. Therefore, the human can visually read the class corresponding to each time-series data representing the vehicle speed change from the display content of the classification data display area 41.

Similarly, in the example shown in FIG. 9B, the class of the time-series data of the x-axis acceleration is represented so as to be visually distinguishable by a human using marks having different shapes, and the time-series change is represented. Are displayed in the classification data display area 42 of each class. Further, the correspondence between the type of the mark assigned to each time-series data and the class is displayed on the class display section 42a. In the example of FIG. 9B as well, twelve classes x1 to x12 of data classified by the HMM are displayed on the class display section 42a. The horizontal axis in the classification data display area 42 indicates time, and the vertical axis indicates acceleration [m / s ² ].

  In the example shown in FIG. 9C, the class arrangement of the time-series data of the y-axis acceleration is displayed in the classification data display area 43 in the same display format as that of FIG. 9B. The correspondence between the type of mark assigned to each time-series data and the class is displayed on the class display section 43a. Further, in the example shown in FIG. 9D, the arrangement of the classes of the time-series data of the z-axis acceleration is displayed in the classification data display area 44 in the same display format as that of FIG. 9B. The correspondence between the type of mark assigned to each time-series data and the class is displayed on the class display section 44a.

<Example of application screen display>
FIG. 10 shows a screen display example of the application software operating on the analysis device 100 shown in FIG.

  Since this GMM-HMM analysis is unsupervised learning, it is necessary for a human to make comparisons and judgments for the class obtained in S05 shown in FIG. 4 to actually specify the driving pattern of the car. Therefore, the application software operating on the analysis device 100 outputs visible information useful for comparison and determination by a human in S04, and provides a screen display as shown in FIG. 10 to the human who is the operator.

  In the example shown in FIG. 10, various pieces of visible information are simultaneously displayed in the analysis application area 50 on the screen. That is, a moving image display window 51, a vehicle speed display window 52, and acceleration display windows 53 to 55 are displayed, respectively.

  The moving image display window 51 is an area for reproducing and displaying the moving image data 32 as a two-dimensional moving image. The vehicle speed display window 52 is an area for displaying the time series change of the vehicle speed data for each class obtained by the GMM-HMM process, and is similar to the classification data display area 41 shown in FIG. The contents are displayed as visible information. A class display section 52a is displayed on the right end of the vehicle speed display window 52.

  The acceleration display windows 53, 54, and 55 are regions for displaying the time-series changes of acceleration data (after GMM-HMM processing) in the x-axis, y-axis, and z-axis directions for each class. In other words, the same content as the classification data display area 42 in FIG. 9B is displayed in the acceleration display window 53, and the same content as the classification data display area 43 in FIG. 9C is displayed in the acceleration display window 54. The same contents as the classification data display area 44 in FIG. 9D are displayed in the acceleration display window 55. In the operation area 56 in the analysis application area 50, a plurality of buttons that can be operated by the operator are displayed.

  In other words, the operator using this application software can use the screen display as shown in FIG. 10 to display the moving image captured by the on-vehicle camera, the time-series change of the vehicle speed, and the time of the x, y, and z axial acceleration. A series change can be visually recognized at the same time. In addition, the display contents of the vehicle speed display window 52 and the acceleration display windows 53 to 55 are in a state where the class classified by the GMM-HMM process can be determined for each time-series data.

  Further, time axis cursors C1, C2, C3, and C4 are displayed in the vehicle speed display window 52 and the acceleration display windows 53, 54, and 55, respectively. The time axis cursors C1 to C4 are vertical straight lines, and represent the times at which the data of each window is currently referred to. Further, the time of the time axis cursors C1 to C4 and the time of the moving image frame being reproduced in the moving image display window 51 are synchronized with each other. The display in the class display section 52a may be controlled so that the class at the time indicated by the current position of the time axis cursor C1 can be visually recognized by highlighting or the like.

<Examples of features of right / left turn patterns>
FIG. 11 shows an example of the characteristics of the right / left turn pattern in the vehicle speed display window of the application. That is, FIG. 11 shows a state in which the position of the vehicle speed display window 52 included in the screen shown in FIG. 10 is enlarged. That is, FIG. 11 shows the same contents as the classification data display area 41 of FIG.

  In the example illustrated in FIG. 11, the classes of the vehicle speed change appear as characteristic curves representing the “vehicle right / left turn pattern” in three regions at different times in the vehicle speed display window 52. In addition, humans can visually recognize that the curves of these three areas are classified into the same class from the types of marks of the respective time-series data. Therefore, a human can specify the traveling pattern so as to assign the corresponding class to the “right / left turn pattern of the car”. In addition, actually, not only the change in the vehicle speed but also the contents of the moving image of the on-board camera at the same time and the change in the three-axis acceleration can be confirmed as shown in FIG. 10, so that the human can easily specify the running pattern. it can. The information on the running pattern specified by the human is registered in the data storage unit 38 as pattern data on the database.

<Characteristic operation of analyzer>
FIG. 12 shows an example of a characteristic process in the analysis device 100. That is, when the processing unit 30 of the analysis apparatus 100 executes the predetermined application software, the processing of FIG. 12 is performed as the detailed contents of S04 and S05 shown in FIG. I do. The processing of FIG. 12 will be described below.

  The analysis device 100 inputs the sensor data and the moving image data 32 in S11 and displays them on the moving image display window 51, the vehicle speed display window 52, and the acceleration display windows 53 to 55, for example, as shown in FIG. Further, the class classified by the GMM-HMM process is reflected on the display contents of the vehicle speed display window 52 and the acceleration display windows 53 to 55, and the class of each data is displayed so that a human can visually recognize it.

The reproduction of the moving image data 32 in the moving image display window 51 can be started by, for example, operating the button of the operation area 56 on the analysis application area 50 by the person who operates the analysis apparatus 100. During the reproduction of the moving image data 32, the image frame to be displayed is automatically updated at regular intervals.
If the moving image data 32 is being reproduced, the analysis apparatus 100 proceeds from S12 to S13, and compares the time of the currently reproduced frame of the moving image with other time-series data (vehicle speed data, three-axis acceleration data). ) To synchronize the display time.

  When the change of the display time is designated by a human input operation on the operation area 56 or the like, the analysis device 100 proceeds from S14 to S15, and the moving image and other time-series data (vehicle speed) for the designated time. Data and three-axis acceleration data).

  When the display time of each data is changed in S13 or S15, the analysis device 100 sets the display positions of the time axis cursors C1 to C4 of the vehicle speed display window 52 and the acceleration display windows 53 to 55 to the display time in S16. Change it accordingly.

  The analyzer 100 displays the moving image display window 51, the vehicle speed display window 52, and the acceleration display window 53 so as to reflect the display time of S13 or S15 and the change of the display position of the time axis cursor C1 to C4 updated in S16. Update the display contents of .about.55.

  Therefore, by executing S17, the playback frame time of the moving image played back in the moving image display window 51, the time of the time axis cursor C1 in the vehicle speed display window 52, and the time of the time axis cursor C2 in the acceleration display window 53 , And the time of the time axis cursor C3 in the acceleration display window 54 and the time of the time axis cursor C4 in the acceleration display window 55 are displayed in synchronization with each other. Further, the class display section 52a displays the class of the vehicle speed data at the time indicated by the time axis cursor C1, and the class of the time-series data of the acceleration data of the three axes.

  Therefore, the operator who is visually recognizing the analysis application area 50 shown in FIG. 10 visually recognizes a characteristic curve of the traveling pattern as shown in FIG. The content of the reproduced frame of the displayed moving image and each acceleration in the acceleration display windows 53 to 55 can be visually recognized in association with the same time and the same class. That is, the operator can make a determination for specifying the traveling pattern of the car in this case while comparing the moving image at the same time, the vehicle speed, and the triaxial acceleration on the same screen. In addition, the data displayed in the vehicle speed display window 52 and the data displayed in the acceleration display windows 53 to 55 can be distinguished from data of the same class and data of other classes. And judgment becomes easy.

  When the operator specifies a travel pattern and performs an input operation for registering the travel pattern as one pattern data, the analysis device 100 proceeds from S18 to S19 and stores reference data representing the corresponding travel pattern in data. Register in the section 38. In the present embodiment, since the analysis device 100 handles data classified into 12 classes, it is possible to easily specify each of the 12 types of traveling patterns and create a database of reference data for converting those classes into pattern data. .

<Configuration of vehicle-mounted device with running pattern detection function>
FIG. 13 shows a configuration example of the vehicle-mounted device 10A having the traveling pattern detection function.
The vehicle-mounted device 10A shown in FIG. 13 includes a nonvolatile memory 62 in which a database 63 of reference data for converting a class into a pattern is registered in advance. Further, the CPU 11A has a function of the pattern detection unit 61 built therein. Other configurations are the same as those of the vehicle-mounted device 10 shown in FIG. Note that, for example, the pattern detection unit 61 and the reference database 63 may be mounted on the cloud 24 side, and the cloud 24 side may be configured to detect a traveling pattern of each vehicle.

  The reference database 63 held by the non-volatile memory 62 is the same data as the reference data for converting a class registered in the data storage unit 38 into a pattern in S19 of FIG. 12, for example. That is, the reference database 63 can be created using the analyzer 100 shown in FIG. The reference data representing each traveling pattern is data for converting a model represented by a GMM-HMM classified into 12 classes into a pattern.

  The pattern detecting section 61 grasps the traveling speed and the three-axis direction acceleration of the own vehicle in real time based on the output of the vehicle speed sensor 23 and the output of the acceleration sensor 18. Further, GMM-HMM processing is performed using the Viterbi algorithm from the time-series changes in the running speed and the three-axis direction acceleration, the class of the time-series data is estimated, and the running pattern is detected from the estimated class and the data in the reference database 63. .

  The CPU 11A transmits information on the traveling pattern detected by the pattern detection unit 61 to the cloud 24 as a part of the operation data of the own vehicle. The operation data is stored in the recording medium 25 together with the time information. Furthermore, when a driving pattern that requires attention is detected for the driver, an alarm sound is output, a voice message is output, and the like.

<Description of Modification>
− <Configuration of analyzer>
FIG. 14 shows a configuration example of an analyzer 100A according to a modification. The analysis device 100A shown in FIG. 14 receives the moving image data 32 and the sensor data 33 described above and outputs an analysis result 37.

  The processing unit 30A illustrated in FIG. 14 includes an image recognition unit 35. The image recognition unit 35 performs an image recognition process on the moving image data 32. That is, the image recognition unit 35 recognizes a pattern using a local model such as a Haal-Like feature, a SIFT feature, or a SURF feature, or recognizes an object or the like using a deep learning model such as a CNN (Convolutional Neural Network). I do. Representative examples of the object to be recognized include traffic-related objects such as traffic lights, signs, signs, pedestrians, and vehicles ahead on the road.

  The analysis unit 31 performs the GMM-HMM process on the time-series data of the sensor data 33 as described above. The analysis unit 36 provided in the processing unit 30A receives the recognition result of the image recognition unit 35 and the processing result of the analysis unit 31, and performs data analysis by GMM-HMM processing.

-<Configuration example of data to be processed>
FIG. 15 shows a configuration example of data processed by the analysis unit 36 at the subsequent stage in the analysis device 100A shown in FIG.

  The sensor data “X1, X1, X1, X2, X1, X2,...” Shown in FIG. 15 is an output of the analysis unit 31 and is a time series classified by the GMM-HMM process as described above. This is time-series data having a class of sensor data as a value. Note that the original time-series data may be input instead of the result of classifying the time-series data of the sensor data. The moving image data “NN, NN, NN, NN, TL, TL, NN, TL,...” Shown in FIG. 15 is a value classified based on the result recognized by the image recognition unit 35 for each image frame. This is time-series data of the included moving image. For example, moving image data of an image frame in which the image recognizing unit 35 has recognized a traffic light is represented by “TL”, and moving image data of an image frame in which the image recognizing unit 35 has not recognized a target such as a traffic light is represented by “NN”. Therefore, the moving image data input to the analysis unit 36 is time-series data including two of “NN” and “TL”.

  When the analysis unit 36 performs the GMM-HMM analysis of the unsupervised learning, it is possible to construct a model for automatically classifying a traveling pattern affected by a traffic light, a sign, a sign, and the like while the vehicle is traveling.

<Advantages of each device in the embodiment>
The analysis device 100 shown in FIG. 3 outputs data obtained by classifying the class by applying the GMM-HMM process to the time series data of the sensor data 33 by the analysis unit 31, and thus can be used for detecting various driving patterns. Models can be easily constructed. For example, it is possible to detect even a running pattern that has not been noticed until now only by looking at a graph of the time series change in the sensor output.

  For example, when the operation data of vehicles collected from various vehicles is analyzed by the analysis device 100 using the cloud 24 shown in FIG. 1 or the like, a highly versatile traveling pattern that does not depend on the characteristics of a specific individual is detected. Can build models.

  Further, by increasing the number of sensors (for example, 4) in the sensor data 33, even if the number of states (for example, 12) handled in the GMM-HMM process of the analysis unit 31 is increased, it is easy to converge the calculation of the GMM-HMM process. This makes it possible to increase the number of classes to be classified and to increase the types of traveling patterns that can be detected.

  In addition, by inputting the vehicle speed and accelerations in a plurality of directions as the sensor data 33 to the analysis unit 31, it is possible to easily construct a model for specifying a traveling pattern to be detected in a general operation state of the automobile. Note that, as the sensor data 33, sensor data indicating, for example, a distance from a vehicle ahead may be added in addition to the vehicle speed and the acceleration. Further, the travel pattern may be detected using sensor data including a combination of the current position and acceleration detected by GPS, for example.

  Further, when the analysis device 100 performs the control as shown in FIG. 12 when the human specifies the running pattern in S05 shown in FIG. Efficiency can be improved. That is, the moving image, the vehicle speed, and the time of each axial acceleration are displayed in synchronization with each other, and the classified class is reflected on the display, so that the running pattern can be easily specified.

  When the pattern detection unit 61 uses the reference database 63 reflecting the result of the GMM-HMM analysis as in the vehicle-mounted device 10A shown in FIG. Patterns can be detected correctly. In particular, since the model obtained by classifying the sensor data by the GMM-HMM process is used, it is possible to detect a traveling pattern such as a right / left turn which is difficult to recognize only by a change in the time-series data.

  Also, by mounting the image recognition unit 35 in the processing unit 30A as in the analysis device 100A shown in FIG. 14, the analysis unit 36 can also automatically specify the traveling pattern, and the analysis pattern can be used for human judgment. Work can be reduced or eliminated.

Here, the features of the analysis device, the vehicle-mounted device, and the pattern analysis auxiliary device according to the above-described embodiment of the present invention will be briefly summarized and listed in the following [1] to [5].
[1] a data acquisition unit (processing unit 30, S01, S02) for acquiring a plurality of types of time-series sensor data (33) detected on an arbitrary vehicle;
A Gaussian mixture distribution and a Hidden Markov Model (GMM-HMM) are applied to the plurality of types of time-series sensor data acquired by the data acquisition unit, and classification data classified into a plurality of classes is output as a result of the analysis. A data analysis unit (analysis unit 31);
A classification data storage unit (data storage unit 38) that stores the classification data output by the data analysis unit;
An analysis device (100), comprising:

[2] The time-series sensor data acquired by the data acquisition unit includes an image (moving image data 32) captured by the vehicle-mounted camera (21),
An image recognition unit (35) for recognizing the content of the image,
The data analysis unit (analysis unit 36) outputs data of a plurality of classified classes from a time-series image as a result of the analysis based on the recognition result of the image recognition unit.
The analysis device according to the above [1], wherein:

[3] The plurality of types of time-series sensor data acquired by the data acquisition unit include at least a traveling speed of the vehicle and a plurality of accelerations applied to the vehicle in different directions,
The data analysis unit outputs classification data classified into a plurality of classes of a number larger than the type of the time-series sensor data,
The analyzer according to the above [1] or [2], wherein:

[4] A plurality of sensors (vehicle speed sensor 23, acceleration sensor 18) for detecting a state related to the running state of the vehicle are included, and signals output from the plurality of sensors are processed to estimate a current running pattern. An on-vehicle device (10A),
For a plurality of types of time-series sensor data detected on an arbitrary vehicle, a Gaussian mixture distribution, and classification data classified into a plurality of classes as a result of analysis applying a hidden Markov model, or included in the classification data A reference data holding unit (non-volatile memory 62) for holding class-pattern conversion data (reference database 63) for specifying a traveling pattern of the vehicle in advance;
A pattern estimating unit (pattern detecting unit 61) for estimating a current traveling pattern based on signals output by a plurality of sensors mounted on the own vehicle and data held by the reference data holding unit;
An on-vehicle device comprising:

[5] A pattern analysis assisting device (analyzing device 100) that assists in specifying a traveling pattern in an arbitrary vehicle,
A data input unit (S01, S02) for inputting, as processing target data, results of analyzing a plurality of types of time-series sensor data including a time-series image captured by an on-board camera and a traveling speed of the own vehicle;
A data display unit (display unit 34, moving image display window 51, vehicle speed display window 52, acceleration display windows 53 to 55) for simultaneously displaying the processing target data input by the data input unit for each type;
A display control unit (S13, S15, S16) for associating display ranges with each other and synchronizing display times of the plurality of processing target data displayed by the data display unit;
An operation unit (operation regions 56, S05, S14, and S18) for receiving a change of a display range or a display time on the data display unit and an input operation of information for specifying a traveling pattern;
A pattern analysis assisting device comprising:

10,10A On-board unit 11,11A CPU
12, 13, 14, 15 interface 16 communication interface 17 card interface 18 acceleration sensor 21 vehicle-mounted camera 22 GPS receiver 23 vehicle speed sensor 24 cloud 25 recording medium 30, 30A processing unit 31, 36 analysis unit 32 moving image data 33 sensor data 34 Display unit 35 Image recognition unit 37 Analysis result 38 Data storage unit 39 Operation unit 41, 42, 43, 44 Classification data display area 41a, 42a, 43a, 44a, 52a Class display unit 50 Analysis application area 51 Video display window 52 Vehicle speed display Windows 53, 54, 55 Acceleration display window 56 Operation area 61 Pattern detection unit 62 Non-volatile memory 63 Reference database 100, 100A Analysis device C1, C2, C3, C4 Time axis cursor

Claims (5)

A data acquisition unit that acquires a plurality of types of time-series sensor data detected on any vehicle;
For a plurality of types of time-series sensor data acquired by the data acquisition unit, a mixed Gaussian distribution, and a data analysis unit that applies a hidden Markov model and outputs classification data classified into a plurality of classes as a result of the analysis.
A classification data storage unit that stores the classification data output by the data analysis unit;
An analysis device, comprising: The time-series sensor data obtained by the data obtaining unit includes an image captured by a vehicle-mounted camera,
An image recognition unit that recognizes the content of the image,
The data analysis unit is based on the recognition result of the image recognition unit, and outputs data of a plurality of classified classes from a time-series image as a result of the analysis,
The analyzer according to claim 1, wherein: The plurality of types of time-series sensor data obtained by the data obtaining unit includes at least a running speed of the vehicle and a plurality of accelerations in different directions applied to the vehicle,
The data analysis unit outputs classification data classified into a plurality of classes of a number larger than the type of the time-series sensor data,
The analysis device according to claim 1 or 2, wherein: An in-vehicle device that includes a plurality of sensors that detect a state related to a traveling state of the own vehicle, processes a signal output by the plurality of sensors to estimate a current traveling pattern,
For a plurality of types of time-series sensor data detected on an arbitrary vehicle, a Gaussian mixture distribution, and classification data classified into a plurality of classes as a result of analysis applying a hidden Markov model, or included in the classification data A reference data holding unit that holds class-pattern conversion data for specifying a traveling pattern of the vehicle in advance;
Signals output by a plurality of sensors mounted on the vehicle, a pattern estimating unit that estimates the current traveling pattern based on the data held by the reference data holding unit,
An on-vehicle device comprising: A pattern analysis assisting device that supports a running pattern identification in an arbitrary vehicle,
A data input unit that inputs a result of analyzing a plurality of types of time-series sensor data including the time-series image captured by the on-board camera and the traveling speed of the own vehicle as processing target data,
A data display unit that simultaneously displays the processing target data input by the data input unit for each type,
For a plurality of the processing target data displayed by the data display unit, associating a display range with each other, and a display control unit that synchronizes display times with each other,
An operation unit that receives a change of a display range or a display time in the data display unit, and an input operation of information for specifying a traveling pattern;
A pattern analysis assisting device comprising: