JP7269694B2 - LEARNING DATA GENERATION METHOD/PROGRAM, LEARNING MODEL AND EVENT OCCURRENCE ESTIMATING DEVICE FOR EVENT OCCURRENCE ESTIMATION - Google Patents

Description

The present invention relates to a technology for generating learning data for generating a learning model for image recognition.

Currently, for the purpose of surveillance and marketing, and also as a "visual system" for self-driving cars and autonomous robots, technology is being developed to analyze the image data generated by cameras and identify the photographed object. It is progressing vigorously.

Here, for example, in an automatic driving vehicle, it is very important to accurately recognize the situation around the vehicle based on the captured image data, in order to enable safe and reliable driving.

As a technique for recognizing such a surrounding situation, for example, Patent Document 1 discloses a vehicle surroundings monitoring technique intended to avoid a dangerous situation while reducing the burden on the driver. Here, the obstacle monitoring device according to this technology displays an image captured by the on-board camera to the driver when the vehicle speed condition for the vehicle equipped with the on-board camera is satisfied. Furthermore, if an obstacle is recognized around the vehicle from the image captured by the on-board camera, the presence of the obstacle is displayed along with the image to inform the driver even if the speed is outside the vehicle speed conditions. Notify

Further, Patent Literature 2 discloses an accident image acquisition system for appropriately acquiring an accident image captured through a vehicle. In this accident image acquisition system, a vehicle transmits to a center a captured image recognized as an accident image involving another accident vehicle other than the own vehicle. Next, when the center judges through image processing of the received captured image that the accident vehicle and a relative object close to the accident vehicle exist in the image, the center takes images based on the degree of proximity between the accident vehicle and the relative object. It is determined whether or not the image is an accident image.

Furthermore, Patent Document 3 discloses an in-vehicle device that transfers peripheral situation information representing the situation around the vehicle to an external server device when a predetermined transfer condition is satisfied. Here, the surrounding situation information includes at least one of traffic accident occurrence information indicating that a traffic accident has occurred in the vicinity of the vehicle and parking lot information related to parking lots existing in the vicinity of the vehicle.

JP 2012-069154 A JP 2015-230579 A JP 2015-210775 A

It is true that the prior art as described in Patent Documents 1 to 3 described above recognizes the existence of an accident site, an accident vehicle, an obstacle, etc. as surrounding conditions based on image data, and the recognition results are used as driving It is possible to present it as useful information for

Here, for example, in the field of automatic driving, it is very important to acquire information on whether or not the situation around the vehicle is likely to cause an accident, not just the presence of obstacles. . For example, when driving on a narrow road with a lot of people, it is extremely important for the autonomous driving body to recognize that "accidents are likely to occur" from the captured image data and to drive safely. And of course, this perception is just as important for drivers of non-autonomous vehicles.

However, with the prior arts such as those described in Patent Documents 1 to 3, it is difficult to determine the likelihood of such an accident from the captured image data of the surroundings. It is generally impossible to estimate the degree of risk of an accident taking into consideration not only the existence of a vehicle but also various surrounding conditions.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to enable execution of processing for estimating the likelihood of occurrence of a predetermined event based on image data relating to the surroundings or the state of a predetermined environment.

According to the present invention, a learning data generation method in a computer for generating learning data for generating a learning model capable of estimating the likelihood of occurrence of a given event based on input image data relating to surroundings or given environment conditions. and
Perform processing to assign a class related to the situation to each pixel or unit area for image data related to the situation of the surroundings or a predetermined environment, and correspond to the class related to the situation to each pixel area as the formed class area. generating segmented image data, which is image data labeled with
determining context information, which is information that may be related to the occurrence of the event, based on the pixel region and the label of the segmented image data;
generating learning data including a data set in which a value indicating that the event is likely to occur is associated as correct data with respect to image data in which the situation information satisfying a predetermined condition is determined. A method is provided.

As an embodiment of the learning data generating method according to the present invention, in the step of determining the situation information, for each item related to the situation information set in advance, an occurrence factor score indicating the degree to which the event occurs is determined, and the occurrence factor score determined for each item is used as the situation information,
In the step of generating learning data, learning includes a data set in which a value indicating that the event is likely to occur is associated with image data whose occurrence factor score satisfies a predetermined condition as correct data. It is also preferred to generate data.

Further, in the learning data generation method according to the present invention, the image data is generated by photographing with a moving body moving in a traffic area, the predetermined event is an accident or an attempted accident event related to the moving body,
The situation information includes information on items related to the width of the traffic area, information on items related to the existence of people and/or moving objects in the traffic area, and information on items related to the existence of intersections where multiple traffic areas intersect. , information on items related to the presence of visual obstacles, information on items related to the number of people and/or moving objects, and information on items related to changes in the positions of surrounding moving objects and/or people beyond a predetermined level. It is also preferable that the information includes at least one of

Furthermore, as another embodiment of the learning data generation method according to the present invention, the image data is generated at or near the point in time when the moving object performs a sudden deceleration or braking operation that satisfies a predetermined condition, or a sudden steering operation. It is also preferable that the predetermined event is an accident or an attempted accident related to the mobile object generated by photographing with the mobile object in .

Further, in the above-described embodiment, the CAN (Controller) mounted on the moving object is used at the point in time when the moving object performs a sudden deceleration or braking operation, or a sudden steering operation that satisfies a predetermined condition, or at a point close thereto. At least one of velocity or braking information obtained from a network), the vector amount of optical flow in the image data, and the motion vector determined during encoding and compression processing for the image data group including the image data. It is also preferred that the time point is determined based on

Furthermore, the segmented image data in the learning data generation method according to the present invention is preferably generated by subjecting the image data to semantic segmentation processing.

According to the present invention, a learning model generated based on a machine learning algorithm using learning data generated by the learning data generation method described above ,
means for inputting image data relating to the situation of the surroundings or a predetermined environment and outputting feature information, which is information relating to features of the image data;
means for inputting the feature information and outputting information relating to the likelihood of occurrence of the event;
A learning model is provided that features a computer functioning by
Further, according to the present invention, there is provided event occurrence estimating means for estimating the likelihood of occurrence of a given event with respect to input image data relating to the situation of the surroundings or given environment, using this learning model, and outputting the estimated result of the estimation . is provided.

As an embodiment of the event occurrence estimation device according to the present invention, the event occurrence estimation device comprises:
A process for assigning a class related to the situation of the surrounding area or a predetermined environment to each pixel or unit area of the input image data is performed, and each pixel area as a formed class area is assigned to the situation of the surrounding area or the predetermined environment. a region-divided image generation means for generating region-divided image data, which is image data to which a label corresponding to the class is added ;
situation information determination means for determining situation information, which is information that may be related to the occurrence of the event, based on the pixel area and the label of the area-divided image data;
and an occurrence factor information generating means for generating and outputting occurrence factor information, which is information supporting the output estimation result based on the situation information and is information related to the cause of the occurrence of the event, based on the situation information. preferable.

Further, according to the present invention, a computer that generates learning data for generating a learning model capable of estimating the likelihood of occurrence of a predetermined event is caused to function based on input image data relating to the situation of the surroundings or the predetermined environment. a program,
Perform processing to assign a class related to the situation to each pixel or unit area for image data related to the situation of the surroundings or a predetermined environment, and correspond to the class related to the situation to each pixel area as the formed class area. a region- divided image generation means for generating region-divided image data, which is image data labeled with a
situation information determination means for determining situation information, which is information that may be related to the occurrence of the event, based on the pixel area and the label of the area-divided image data;
A computer as learning data generating means for generating learning data including a data set in which a value indicating that the event is likely to occur is associated as correct data with respect to the image data in which the situation information satisfying a predetermined condition is determined. A functioning learning data generator is provided.

According to the learning data generating method/program, learning model, and event occurrence estimating device of the present invention, it is possible to perform a process of estimating the likelihood of occurrence of a given event based on image data relating to the surroundings or the situation of the given environment. .

1 is a schematic diagram and a functional block diagram for explaining an embodiment of an event occurrence estimation system including a learning model generation device and an event occurrence estimation device according to the present invention; FIG. FIG. 10 is a schematic diagram for explaining an embodiment of region-divided image generation processing and peripheral situation information generation processing according to the present invention; FIG. 10 is a schematic diagram for explaining an embodiment of region-divided image generation processing and peripheral situation information generation processing according to the present invention; FIG. 4 is a schematic diagram for explaining an example of presentation information generation processing according to the present invention; 1 is a sequence diagram showing an outline of an embodiment of a learning data/learning model generation method and an event occurrence estimation method according to the present invention; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Event occurrence estimation system]
FIG. 1 is a schematic diagram and a functional block diagram for explaining an embodiment of an event occurrence estimation system having a learning model generation device and an event occurrence estimation device according to the present invention.

The event occurrence estimation system of this embodiment shown in FIG.
(a) one or more terminals 20 that are event occurrence estimation devices (movable in this embodiment);
(b) a cloud server 1 which is a learning model generation device capable of generating a learning model based on image data acquired from the terminal 20 (or image database); "Learning data" is generated using image data selected in consideration of the "surrounding situation information" described in 1, furthermore, a "learning model" is generated by this, and this "learning model" is supplied to the terminal 20 be done.

Here, this "learning model" is based on input image data (for example, image data showing the situation around the automobile 2 captured and generated by the camera 203) relating to the situation of the surroundings or a given environment, and a given event (for example, traffic It is a model that can estimate the likelihood of accidents or attempted traffic accidents). The terminal 20 supplied with such a "learning model" uses this "learning model" to perform event occurrence estimation processing for estimating the likelihood of occurrence of a predetermined event.

Incidentally, in this embodiment, the terminal 20 is a drive recorder having a communication function, and is installed at a position (for example, above the dashboard) of the automobile 2 where, for example, the front of the vehicle can be photographed. In addition, each terminal 20 can be connected to the cloud server 1 for wireless communication via, for example, a mobile phone communication network or the Internet. It can be sent to the server 1.

Furthermore, the cloud server 1 generates "learning data" for generating the "learning model" described above,
(A) Performing a process of assigning a value related to the situation to each pixel or unit area for image data related to the situation of the surroundings or a predetermined environment, and labeling the formed pixel area (image partial area) related to the situation a region-divided image generation unit 112 that generates “region-divided image data”, which is image data to which
(B) Peripheral situation information determination unit 113 that determines "peripheral situation information" (situation information), which is information that may be related to the occurrence of a predetermined event, based on the pixel region and the label of the "region-divided image data"; ,
(C) Generating "learning data" including a data set in which, as correct data, a value indicating that a predetermined event is likely to occur is associated with image data for which "surrounding situation information" that satisfies a predetermined condition has been determined. It is characterized by having a learning data generation unit 114 .

In this manner, the cloud server 1 generates the "region-divided image data" and determines the "peripheral situation information". This "surrounding information", which will be described in detail later, can be, for example, road width information or information on the number of people near the road. This is important information that you can relate to. Based on such "peripheral situation information", the cloud server 1 can determine, select or extract image data suitable for "learning data". It is possible to generate "learning data" suitable for generating a "learning model" that enables

Incidentally, the "region-divided image data" generated in (A) above can be generated by applying semantic segmentation processing to image data relating to the situation of the surroundings or a predetermined environment.

This semantic segmentation process classifies each pixel in the frame image into a class such as "person", "car", "road", "sky", etc., and then classifies the class area such as "person" into the frame image. It is a known process for determining regions, such as "car" regions, "road" regions, and "sky" regions. As a result, it becomes possible to give meaning to what the object in the image is, and as a result, various "surrounding situation information" is determined from the information of the pixel area (image partial area) that has the meaning. It becomes possible.

For example, when generating "learning data", it is conceivable to use image data of the scene where a predetermined event (for example, a traffic accident or attempted traffic accident) occurred. A learning model to be estimated may not be effectively generated. On the other hand, the cloud server 1 is not limited to the image data of the occurrence site, but uses image data determined, selected or extracted in consideration of the "surrounding situation information" with meaning. This makes it possible to more effectively generate a "learning model" that estimates the ease of doing so.

Incidentally, the terminal 20 is of course not limited to an in-vehicle device (drive recorder) installed in the automobile 2. ). It may also be a wearable terminal such as an HMD (Head Mounted Display) or a glass-type terminal. Furthermore, it may be an information processing device such as a non-movable (non-mobile) personal computer (PC).

Further, for example, a monitoring robot in a factory may be equipped with a terminal (event occurrence estimation device) according to the present invention, and the occurrence of various troubles (events) that may occur in the factory can be predicted from the images captured by the robot's camera. It becomes possible.

1, the cloud server 1 also has an event occurrence estimation function, and the terminal 20 uploads image data captured by the camera 203 to the cloud server 1, It is also possible to adopt an embodiment in which the event occurrence estimation result is acquired from 1.

Here, the functional configuration of the apparatus of this embodiment will be described in detail below. Assume that it is an attempted event.

[Functional Configuration of Learning Model Generating Device]
Similarly, according to the functional block diagram shown in FIG. 1, the cloud server 1 has a communication interface 101 and a processor/memory. Here, the processor memory stores an embodiment of the learning model generation program including the learning data generation program according to the present invention, and further has a computer function to execute the learning model generation program. By doing so, the learning model generation process is executed.

In addition, from this, as a learning model generation device according to the present invention, instead of the cloud server 1, a non-cloud server device, such as a personal computer (PC), which is equipped with the learning model generation program according to the present invention. , a notebook or tablet computer, or a smart phone or the like can be used.

For example, the learning model generation program according to the present invention can be installed in the terminal 20, and the terminal 20 can be used as the learning model generation device according to the present invention. Further, an embodiment in which the learning model generation device according to the present invention is installed in the car 2 together with the terminal 20 is also possible.

Further, the processor memory has an image acquisition unit 111 , a segmented image generation unit 112 , a surrounding situation information determination unit 113 , a learning data generation unit 114 , and a learning model generation unit 115 . It should be noted that these functional configuration units can be regarded as the functions of the learning model generation program, including the learning data generation program according to the present invention, stored in the processor memory. Further, the flow of processing in which the functional components of the cloud server 1 are connected by arrows in FIG. 1 can also be understood as an embodiment of the learning model generation method according to the present invention.

Also in the functional block diagram of FIG. 1, an image acquisition unit 111 collects and stores image data included in "learning data" for generating a "learning model", and stores the image data for generating learning data. image data management means for appropriately outputting to . For example, the image acquisition unit 111 can acquire a large amount of image data from each terminal 20 via the communication interface 101 . Also, a large number of image data may be acquired from an external image database.

Here, in this embodiment, the image data acquired from each terminal 20 by the image acquisition unit 111 will be described in detail later, but the vehicle 2 in which the terminal 20 is mounted suddenly decelerates or brakes only to satisfy a predetermined condition. It is also preferable to include image data generated by taking pictures of the motor vehicle 2 at or near the time of the action (braking action) or sharp steering (steering action). Furthermore, it is preferable that a "rapid deceleration" tag, a "rapid braking" tag, and a "rapid steering" tag are added to the image data related to such sudden deceleration, sudden braking, and sudden steering.

Such tagged image data is image data suitable for generating a "learning model" for estimating the likelihood of a traffic accident or attempted traffic accident involving the automobile 2 as a predetermined event. There is.

In this embodiment, the segmented image generation unit 112 performs the above-described Semantic Segmentation processing on the image data output from the image acquisition unit 111 to generate segmented image data. The specific contents of the segmented image data will be described in detail later with reference to FIGS. 2 and 3. FIG.

The surrounding situation information determining unit 113 determines whether a traffic accident or Surroundings information is determined which is information that may be relevant to the occurrence of an attempted traffic accident event. Here, in this embodiment, in particular, surrounding situation information can be determined for image data to which a "sudden deceleration" tag, a "sudden braking" tag, or a "sudden steering" tag has been assigned (region-divided image data generated from the image data). important. The specific content of the peripheral situation information will be described later in detail with reference to FIGS.

2 and 3 are schematic diagrams for explaining an embodiment of the region-divided image generation processing and the peripheral situation information generation processing according to the present invention.

In the embodiment shown in FIG. 2, the segmented image generation unit 112 performs semantic segmentation processing on RGB image data including scenery in the traveling direction of the automobile 2 to generate segmented images. As shown in FIG. 2, the segmented image generated here includes pixel regions Ra to which the class label "road" is assigned, while pixel regions Ra to which the class labels "automobile" and "human" are assigned. , does not contain pixel regions with the class label "visual obstruction".

Next, the peripheral situation information determining unit 113 determines peripheral situation information based on the information on the pixel regions in the generated segmented image and the information on the class labels (given or not given). Specifically, in this embodiment, as items of the surrounding situation information,
(a) road width, (b) visibility obstruction, (c) number of people, (d) number of cars, ...
is set in advance, and for each item, a "risk score (occurrence factor score)" that indicates the degree to which a traffic accident or attempted traffic accident event occurs is determined. Score” is used as peripheral situation information.

Incidentally, in the case where the image data is generated by shooting with a moving body moving in a traffic area, and the predetermined event is an accident or an attempted accident involving this moving body, the peripheral situation information includes (a) this moving body (b) item information related to the presence of visual obstructions; (c) item information related to the presence of people and/or moving objects (including animals) in the passage area; (d) item information related to the existence of an intersection where multiple traffic areas intersect; (e) item information related to the number of people and/or moving bodies; (f) surrounding moving bodies and/or At least one of (g) item information related to a change in position of a person more than a predetermined amount (e.g. behavior such as sudden braking or sudden interruption of a vehicle in front), and (g) information related to brightness in the original image data. can be information including

In the example shown in FIG. 2, the calculated "road width" danger score, "visual obstruction" danger score, and "number of people" danger score are all relatively small values. Here, the "road width" risk score is determined in advance by a coordinate conversion formula between the image coordinate system and the real space coordinate system (with the camera position as the origin), and is used to calculate the actual pixel area Ra. The width of the space (distance in the horizontal direction) is calculated as the road width, and the road width can be determined by referring to a table in which the smaller the road width, the higher the score. Here, it is also preferable that the score of this table is determined taking into consideration the width of the automobile 2 (ratio to the road width).

In addition, the "visual field obstruction" risk score is defined by taking into account the pixel area Ra in the segmented image and defining the "visual field area". It is possible to determine the ratio of the region to be occupied, and refer to a table in which the higher the ratio, the higher the score is associated in advance. Further, a "visual obstruction" danger score may be determined based on the position occupied by the "visual obstruction" assigned pixel area within the pixel area Ra. For example, a higher score may be given as the position of the pixel region given the “visual obstruction” is closer to the center of the pixel region Ra.

Furthermore, the "number of people" risk score is determined by taking into account the pixel area Ra in the segmented image and defining the "human presence attention area", and then assigning the class label "human" to the pixel area. The "number" of pixel areas whose bottom edge (foot) is within this human presence attention area is determined, and from this "number", a table is determined in which the larger the "number" is, the higher the score is associated in advance. be able to. Also, in this case, the closer the position of the lower end (foot) is within the "human presence attention area" and closer to the center of the pixel area Ra, the more weight is given when counting the "number". may also account for a larger number of counts.

Next, in the embodiment shown in FIG. 3, the segmented image generation unit 112 performs semantic segmentation processing on RGB image data including scenery in the traveling direction of the automobile 2, as in the embodiment in FIG. Generating segmented images. As shown in FIG. 3, the segmented image generated here includes a pixel region Ra assigned with the class label "road", a pixel region Rb assigned with the class label "columnar body (visual obstruction)", and a pixel region Rb assigned with the class label It includes a pixel region Rc assigned the "wall-like body (visual obstruction)" and a pixel region Rd assigned the class label "human", while it includes a pixel area assigned the class label "automobile". not

2, the surrounding situation information determination unit 113 determines the surrounding area based on the pixel area information and the (assigned or not assigned) class label information in the generated segmented image. Determining status information. Specifically, in the embodiment of FIG. 3, the calculated "road width" risk score and "visual field obstruction" risk score are relatively large values, and the "number of people" risk score is relatively medium. It is a value of degree.

Returning to the functional block diagram of FIG. 1, the learning data generation unit 114
(a) Values (labels) indicating the likelihood of traffic accidents (or attempted traffic accidents) are associated as correct data with respect to the original image data for which surrounding situation information (danger score group) that satisfies a predetermined condition has been determined. generate a dataset with
(b) A value (label ) as correct data,
Generate "learning data" containing these datasets.

Here, as the (original) image data, it is also preferable to use only the image data to which the "sudden deceleration" tag, the "sudden braking" tag, or the "sudden steering" tag is attached. In this case, the learning data generation unit 114 can associate more accurate correct labels in consideration of the surrounding situation information for the tagged image data that may be related to traffic accidents or attempted traffic accidents. It becomes possible. That is, it is possible to generate more appropriate "learning data" that does not simply depend on the presence or absence of "sudden deceleration" tags, "sudden braking" tags, and "sudden steering" tags. However, it is of course possible to assign appropriate labels to image data that are not tagged with a "sudden deceleration" tag, a "sudden braking" tag, or a "sudden steering" tag, and then use them as learning data.

As the predetermined condition (a), for example, the value of the summation of weighted risk scores for the risk score group as the surrounding situation information, that is, the "total risk score" is equal to or greater than a predetermined threshold. It is also possible to be In this case, for example, a data set in which 1 (dangerous) is associated with image data whose "total risk score" is greater than or equal to a predetermined threshold, and 0 (safe) is associated with image data whose "total risk score" is less than the predetermined threshold. can be generated as "learning data". For example, 0 (safe) is associated with the image data shown in FIG. 2, and 1 (dangerous) is associated with the image data shown in FIG. 3 to generate "learning data."

Furthermore, as a modification mode, the 1st to Nth ranges (for example, 1st (small), 2nd (middle), and 3rd (large) ranges are defined in advance for the above-mentioned "total risk score"), For image data whose "total risk score" is in the p-th range (1 ≤ p ≤ N), p is associated as a correct label (for example, 1 ( It is also preferable to generate a data set associated with safety) and use it as “learning data”. Alternatively, the "total risk score" can be converted to the degree of risk (0.0: safe to 1.0: dangerous), and the degree of risk can be used as the correct label to generate a data set associated with the image data, which can be used as "learning data." good.

In any case, the learning data generation unit 114 is image data generated, for example, during sudden braking, sudden deceleration, or sudden steering, and the surrounding situation information (risk score group) is a traffic accident (or attempted traffic accident event). For the image data indicating that the situation is likely to occur, a data set is generated in which a value (label) indicating that a traffic accident (or attempted traffic accident event) is likely to occur is associated as correct data. It is possible to generate "learning data".

The learning model generation unit 115 uses the “learning data” generated by the learning data generation unit 114 to generate a “learning model” based on a predetermined machine learning algorithm. In this embodiment, this "learning model" is a model capable of outputting the likelihood (degree of risk) of traffic accidents (or attempted traffic accident events) from the input image data.

Here, as machine learning algorithms, there are various algorithms such as deep neural networks (DNN, Deep Neural Network), which are widely used for image recognition, support vector machines (SVM), and random forests. Applicable. In any case, various algorithms can be employed as long as they constitute a discriminator that inputs image data and outputs a discrimination result.

Specifically, as one embodiment, the learning model generation unit 115
(a) Convolutional Layers as a first NN that inputs image data and outputs feature information related to these features;
(b) Configure a classifier including fully-connected layers as a second NN that inputs feature information output from the convolutional layer and outputs information related to classes (information related to labels) However, a "learning model" may be generated by performing a learning process on this using "learning data".

Here, the convolution layer unit (a) above executes convolution processing for generating a feature map by sliding a kernel (weighting matrix filter) on image data. By this convolution process, basic features such as edges and gradients can be extracted while the resolution of the image is gradually reduced, and information on local correlation patterns can be obtained. For example, it is possible to use the well-known AlexNet using a plurality of convolution layers as this convolution layer section.

In this AlexNet, each convolutional layer is paired with a pooling layer, and convolutional processing and pooling processing are repeated. The pooling process here means that the feature map output from the convolution layer (reaction of the convolution filter in a certain area) is summarized by the maximum value, average value, etc., and local translation invariance is secured while reducing the adjustment parameters. It is a process to

As another embodiment, the learning model generation unit 115 includes a support vector machine (SVM ), and perform learning processing using learning data to generate a “learning model”.

In any case, in this embodiment, the learning model generation unit 115 can transmit the generated “learning model” to the terminal 20, which is an event occurrence estimation device, for example, via the communication interface 101. FIG.

[Functional Configuration of Event Occurrence Estimating Device]
Also according to the functional block diagram shown in FIG. 1, the terminal 20 has a communication interface 201, a communication interface 202, a camera 203, a display (DP) 204, and a processor memory. Here, the processor memory stores an event occurrence estimating program according to an embodiment of the present invention, and further has a computer function. Perform estimation processing.

For this reason, as the event occurrence estimation device according to the present invention, instead of the present terminal 20, which is a drive recorder, another in-vehicle information processing device equipped with the event occurrence estimation program according to the present invention, or a camera is provided. Alternatively, a smart phone, notebook or tablet computer, or personal computer (PC) connected to a camera can be used. A terminal, such as a smartphone, which is connected to the drive recorder via Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like, may be used as the event occurrence estimation device.

Further, the processor memory includes a video generation unit 211, an image selection unit 212, an accident occurrence estimation unit 213, a presentation information generation unit 214, a segmented image generation unit 215, a peripheral situation information determination unit 216, an occurrence and a factor information generator 217 . It should be noted that these functional components can be regarded as functions of the event occurrence estimation program stored in the processor memory. Further, the flow of processing in which the functional components of the terminal 20 are connected by arrows in FIG. 1 can also be understood as an embodiment of the event occurrence estimation method according to the present invention.

Similarly, in the functional block diagram of FIG. In the present embodiment, the terminal 20 is a drive recorder, and the image generation unit 211 normally acquires image data of the situation outside the vehicle from the camera 203 at least when the vehicle 2 is running as a default setting, and outputs the image data ( image data group).

In this embodiment, the image selection unit 212 performs a sudden deceleration or braking operation (brake operation) or a sudden steering operation (handle operation) that satisfies a predetermined condition from the image data generated by the image generation unit 211. Select image data at a point in time or a point in the vicinity thereof (for example, a point in time within a predetermined time range with the point in time as the middle point), and select the corresponding "sudden deceleration" tag, "sudden braking" tag, or "sudden steering" tag , and the image data with these tags is transmitted to the cloud server 1 via the communication interface 201 connected to a communication network such as a mobile phone communication network or the Internet.

Here, the terminal 20 is connected to the CAN (Controller Area Network) of the automobile 2 via a communication interface 202 via a short-range wireless communication network such as Wi-Fi (registered trademark) or Bluetooth (registered trademark) or by wire. The image selection unit 212 may acquire speed information, braking information, and steering information from this CAN via the communication interface 202 .

In this case, the image selection unit 212 calculates the acceleration of the automobile from the acquired speed information, and satisfies a predetermined condition, for example, that the acceleration is a negative value and the absolute value exceeds a predetermined threshold for a predetermined time or more. , a "rapid deceleration" tag can be attached to the corresponding image data. Also, a value indicating the strength of the braking (brake) operation of the vehicle is calculated from the acquired braking information, and the value indicating the strength indicates a predetermined condition, for example, a condition that the value exceeds a predetermined threshold for a predetermined time or more. If so, it is also preferable to add a "sudden braking" tag to the corresponding image data. Furthermore, a value indicating the strength of the steering (steering wheel) operation of the automobile is calculated from the acquired steering information, and the value indicating the strength indicates a predetermined condition, for example, a condition that the value exceeds a predetermined threshold for a predetermined period of time or longer. If so, it is also preferable to tag the corresponding image data with a "quick handle" tag.

As a modification mode, the image selection unit 212 performs optical flow analysis on the time-series image data acquired from the image generation unit 211, and when a large optical flow vector amount that satisfies a predetermined condition is calculated, the corresponding image data. It is also preferable to attach a “sudden change” tag to the data and transmit it to the cloud server 1 . Furthermore, it is also possible to perform image difference analysis on the time-series image data, and attach a "sudden change in situation" tag when an image difference amount equal to or greater than a predetermined amount is detected within a predetermined period of time.

In addition, when image data is transmitted from the terminal 20 to the cloud server 1, the image data is usually encoded by MPEG (Moving Picture Experts Group) or the like to generate and transmit compressed image data. However, if the motion vector (as an encoding parameter) determined during the encoding process has a size that satisfies a predetermined condition, the corresponding compressed image data is given a "sudden change" tag. It is also preferable to transmit to the cloud server 1 by

As a modification, the cloud server 1 decompresses (decodes) the compressed image data received from the terminal 20 to extract a motion vector, and if the motion vector has a size that satisfies a predetermined condition, the decompressed image data. may be given a "sudden change in situation" tag. Furthermore, when the CAN information corresponding to the compressed image data is always transmitted from the terminal 20 to the cloud server 1, or when the compressed image data to which the CAN information is added is transmitted, the cloud server 1 acquires Based on the received CAN information, a "sudden deceleration" tag, a "sudden braking" tag, a "sudden steering" tag, and a "sudden change in situation" tag may be added to the received and decompressed image data.

In any case, the cloud server 1 treats the image data to which the "sudden change in situation" tag described above is attached as the image data to which the "sudden deceleration" tag, the "sudden braking" tag, or the "sudden steering" tag is attached. will be treated in the same way.

It goes without saying that the image selection unit 212 transfers image data to which the "sudden deceleration" tag, the "sudden braking" tag, the "sudden steering" tag, and the "sudden change in situation" tag are not attached to the cloud server 1 for generating learning data. You may send. For example, such image data can be added to learning data after being associated with a value (correct label) indicating that a traffic accident (or attempted traffic accident event) is unlikely to occur.

Similarly, in the functional block diagram of FIG. 1, an accident occurrence estimation unit 213 as event occurrence estimation means uses a "learning model" acquired from the cloud server 1 via the communication interface 201 to carry out traffic accident occurrence estimation processing. receives input image data generated by the image generation unit 211 or selected by the image selection unit 212 (given a predetermined tag), and detects a traffic accident in the input image data related to the surrounding situation. (or attempted traffic accident event) is estimated.

Here, as the estimation result, one of two values, for example, one of the class labels "danger" and "safe", according to the setting of the correct class label in the "learning data" used when generating the "learning model" You can output either Alternatively, one of multiple values, for example, one of the class labels "danger", "caution", and "enforce safe driving" may be output.

The region-divided image generation unit 215 assigns a value related to the situation of the surroundings or a predetermined environment to each pixel or unit region to the input image data (image data input to the accident occurrence estimation unit 213), which is the event occurrence estimation target. Processing is performed to generate region-divided image data, which is image data in which each formed pixel region (image partial region) is labeled according to the surroundings or given environmental conditions. Here, the processing contents for generating the segmented image data can be the same as the processing contents in the segmented image generation unit 112 of the cloud server 1 described above, and specifically, the semantic segmentation processing is performed. It's okay.

The peripheral situation information determination unit 216 determines peripheral situation information, which is information that may be related to the occurrence of a traffic accident or attempted traffic accident, based on the pixel regions and labels of the region-divided image data generated by the region-divided image generation unit 215. decide. Here, too, the processing content for determining the peripheral situation information can be the same as the processing content in the peripheral situation information determining unit 113 of the cloud server 1 described above.

Based on the surrounding situation information determined by the surrounding situation information determining section 216, the occurrence factor information generating section 217 generates information that supports the estimation result output from the accident occurrence estimating section 213 and that the traffic accident or attempted traffic accident event occurred. It generates and outputs "occurrence factor information" which is information related to the cause of occurrence. For example, if the "road width" risk score (surrounding situation information) is equal to or greater than a predetermined threshold, this "occurrence factor information" may be "road narrow" or "number of people" risk score (surrounding situation information). is equal to or greater than a predetermined threshold, it may be "a lot of pedestrians".

The presentation information generation unit 214 generates “presentation information” including (a) the output estimation result and (b) the generated cause information for the input image data input to the accident occurrence estimation unit 213, and generates the “presentation information”. The “presentation information” is displayed on the display 204 together with the input image data. Here, for example, "presentation information" including "occurrence factor information" may be generated and displayed only when the estimation result of (a) above is content corresponding to "danger" (or a predetermined degree of danger or higher). is also preferred.

FIG. 4 is a schematic diagram for explaining an embodiment of presentation information generation processing according to the present invention.

According to FIG. 4, for the image data of the traveling direction of the automobile 2 captured and generated by the camera 203,
(a) The accident occurrence estimating unit 213 outputs an estimation result of "dangerous",
(b) The surrounding situation information determination unit 216 determines a “number of people” risk score equal to or greater than a predetermined threshold, and a “obstruction in sight” risk score equal to or greater than a predetermined threshold. generates and outputs the occurrence factor information "many pedestrians" and "poor visibility",
The presentation information generation unit 214 generates the presentation information “Danger! Too many pedestrians! Poor visibility!” there is

Here, in this embodiment, a series of processes including the event occurrence estimation process are performed immediately after the input image data is generated, and the image data and presentation information are displayed on the display 204 substantially in real time.

In this way, by presenting the reason for the danger (matter that may cause an accident) along with the estimated result corresponding to "danger", the user is more accurately alerted, and the user can avoid the accident. It is also possible to immediately and surely understand the points to be noted in order to avoid accidents, and to practice more appropriate and safer driving.

Also, for example, in the case of self-driving cars, it is possible to more accurately recognize the degree of danger (the likelihood of a traffic accident occurring) in unknown road conditions that are not registered as so-called near-miss points. Therefore, it becomes possible to carry out a driving operation that matches the situation of the degree of danger.

[Learning data/learning model generation method, event occurrence estimation method]
FIG. 5 is a sequence diagram showing an outline of an embodiment of the learning data/learning model generation method and the event occurrence estimation method according to the present invention. Here, in this embodiment, each of the terminals 20A to 20C always captures the situation in the traveling direction of the automobile 2 with the camera 203 to generate an image (image data) (step S101). Although the number of terminals 20 is three in this embodiment, the present invention is of course not limited to this.

(S102) The terminal 20C acquires sudden braking information of the automobile 2 in which it is mounted from the CAN.
(S103) The terminal 20C selects the image data generated when the sudden braking information was acquired.
(S104) The terminal 20C transmits the selected image data at the time of sudden braking (image data with the sudden braking tag attached) to the cloud server 1 .
Here, the processes corresponding to steps S102 to 104 are performed on the other terminals 20A and 20B, and the processes corresponding to steps S102 to 104 are repeated in each of the terminals. Sufficient amount of tagged image data shall be obtained.

(S111) The cloud server 1 performs semantic segmentation processing on the acquired image data to generate a segmented image.
(S112) The cloud server 1 determines a risk score group as surrounding situation information from the generated segmented image.
(S113) The cloud server 1 generates "learning data" including a data set in which the correct label "danger" is associated with image data having a risk score group that satisfies a predetermined condition.

(S114) When the "learning data" reaches a predetermined amount, the cloud server 1 generates a "learning model" using the "learning data".
(S115) The cloud server 1 supplies the generated "learning model" to each terminal 20A-20C.

(S121) Each of the terminals 20A to 20C uses the obtained "learning model" to estimate the degree of risk of a traffic accident (likelihood of a traffic accident) for each generated image data.
(S122) Each of the terminals 20A to 20C displays presentation information related to the estimated degree of risk on the display 204 in accordance with the image data.

As described in detail above, according to the present invention, when generating "learning data" including image data, "region-divided image data" is generated from the image data, and the data can be related to the occurrence of a predetermined event. Determine the "surrounding information" that is important information. Then, image data suitable for the "learning data" can be determined, selected or extracted based on such "peripheral situation information". In addition, this makes it possible to generate "learning data" suitable for generating a "learning model" that enables execution of the process of estimating the likelihood of occurrence of a predetermined event.

In addition, for example, when generating "learning data", it is conceivable to use image data of the site where the predetermined event occurred. may not be generated in On the other hand, the present invention is not limited to the image data of the occurrence site, but uses image data determined, selected, or extracted in consideration of the "surrounding situation information" with meaning. It is possible to more effectively generate a “learning model” that estimates the degree of

By the way, the "learning model" generated using the present invention can be used to recognize the "meaning" of surrounding situations in autonomous mobile objects such as self-driving cars, autonomous mobile robots, and even autonomous mobile drones, which are expected to spread further in the future. It is thought that it will contribute greatly to strengthening power. It can also be very useful in predicting and coping with the occurrence of various events in a given environment or area using a monitoring camera system or monitoring robot.

A person skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and aspect of the present invention for the various embodiments of the present invention described above. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and the equivalents thereof.

1 Cloud server (learning model generation device)
101 communication interface 111 image acquisition unit 112 segmented image generation unit 113 peripheral situation information determination unit 114 learning data generation unit 115 learning model generation unit 2 automobiles 20, 20A, 20B, 20C terminal (event occurrence assumption device)
201, 202 communication interface 203 camera 204 display (DP)
211 Video generation unit 212 Image selection unit 213 Accident occurrence estimation unit 214 Presentation information generation unit 215 Segmented image generation unit 216 Surrounding situation information determination unit 217 Occurrence factor information generation unit

Claims (10)

A learning data generation method in a computer for generating learning data for generating a learning model capable of estimating the likelihood of occurrence of a predetermined event based on input image data relating to surroundings or a predetermined environment, comprising:
Perform processing to assign a class related to the situation to each pixel or unit area for image data related to the situation of the surroundings or a predetermined environment, and correspond to the class related to the situation to each pixel area as the formed class area. generating segmented image data, which is image data labeled with
determining context information, which is information that may be related to the occurrence of the event, based on the pixel region and the label of the segmented image data;
and generating learning data including a data set in which a value indicating that the event is likely to occur is associated as correct data with respect to the image data in which the situation information satisfying a predetermined condition is determined. A training data generation method. In the step of determining the situation information, for each item related to the situation information set in advance, an occurrence factor score indicating the degree to which the event occurs is determined, and the occurrence factor score determined for each item is determined. as the relevant situation information,
In the step of generating learning data, learning data including a data set in which a value indicating that the phenomenon is likely to occur is associated with image data whose occurrence factor score satisfies a predetermined condition as correct data. 2. The learning data generating method according to claim 1, wherein: The image data is generated by photographing with a mobile object moving in a traffic area, and the predetermined event is an accident or an attempted accident event related to the mobile object,
The situation information includes information on items related to the width of the traffic area, information on items related to the existence of people and/or moving objects in the traffic area, and information on items related to the existence of intersections where multiple traffic areas intersect. , information on items related to the presence of visual obstacles, information on items related to the number of people and/or moving objects, and information on items related to changes in the positions of surrounding moving objects and/or people beyond a predetermined level. 3. The learning data generation method according to claim 1, wherein the information includes at least one of: The image data is generated by photographing the moving object at or near the time when the moving object performs a sudden deceleration or braking operation or a sudden steering operation that satisfies a predetermined condition, and the predetermined event is 4. The learning data generating method according to any one of claims 1 to 3, wherein the learning data generation method is an accident or an attempted accident involving the moving object. The speed obtained from the CAN (Controller Area Network) mounted on the moving object at the time when the moving object performs a sudden deceleration or braking operation or a sudden steering operation that satisfies a predetermined condition, or at a time near it. or the time point determined based on at least one of the braking information, the vector amount of the optical flow in the image data, and the motion vector determined during the encoding compression process for the image data group including the image data. 5. The learning data generation method according to claim 4, wherein: 6. The learning data generation method according to any one of claims 1 to 5, wherein the region-divided image data is generated by subjecting the image data to a semantic segmentation process. A learning model generated based on a machine learning algorithm using learning data generated by the learning data generation method according to any one of claims 1 to 6,
means for inputting image data relating to the situation of the surroundings or a predetermined environment and outputting feature information, which is information relating to features of the image data;
means for inputting the feature information and outputting information relating to the likelihood of occurrence of the event;
A learning model characterized by making a computer function by An event occurrence estimating means for estimating the likelihood of occurrence of a predetermined event with respect to the input image data relating to the situation of the surroundings or the predetermined environment, using the learning model recited in claim 7, and outputting the estimation result of the estimation . An event occurrence estimation device characterized by: A process for assigning a class related to the situation of the surrounding area or a predetermined environment to each pixel or unit area of the input image data is performed, and each pixel area as a formed class area is assigned to the situation of the surrounding area or the predetermined environment. a region-divided image generation means for generating region-divided image data, which is image data to which a label corresponding to the class is added ;
situation information determination means for determining situation information, which is information that may be related to the occurrence of the event, based on the pixel area and the label of the area-divided image data;
further comprising an occurrence factor information generation means for generating and outputting occurrence factor information, which is information supporting the output estimation result and related to factors causing the event, based on the situation information. 9. The event occurrence estimation device according to claim 8. A program for causing a computer to generate learning data for generating a learning model capable of estimating the likelihood of occurrence of a given event based on input image data relating to the situation of the surroundings or given environment, comprising:
Perform processing to assign a class related to the situation to each pixel or unit area for image data related to the situation of the surroundings or a predetermined environment, and correspond to the class related to the situation to each pixel area as the formed class area. a region- divided image generation means for generating region-divided image data, which is image data labeled with a
situation information determination means for determining situation information, which is information that may be related to the occurrence of the event, based on the pixel area and the label of the area-divided image data;
A computer as learning data generating means for generating learning data including a data set in which a value indicating that the event is likely to occur is associated as correct data with respect to the image data in which the situation information satisfying a predetermined condition is determined. A learning data generation program characterized by functioning.

Images