JP7001795B1 - Data generators, methods and computer programs for simulation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a simulation data generation device capable of reducing a calculation cost required for a simulation of driving support or automatic driving. A data generation device includes an information search unit that performs a search process for searching for a plurality of conceptual elements having a time stamp of the same time within a specified time and an allowable range from at least object information and vehicle information. Multi-class classification that classifies multiple found conceptual elements into multiple classes and generates class data showing the relationship between each class of the plurality of classes and the conceptual elements classified into each class. It includes a unit and a structured data generation unit that generates a structured data set having a time stamp by relating a plurality of class data generated for each of the plurality of classes by the multi-class classification unit to each other. .. [Selection diagram] Fig. 1

Description

The present disclosure relates to techniques for generating data used for driving assistance or simulation of autonomous driving of vehicles such as automobiles.

In recent years, research and development of driving support technology and automatic driving technology for vehicles such as automobiles and railroad vehicles have been actively promoted. The driving support system is a target vehicle (a vehicle driven by the driver himself) by automatically operating the brakes or automatically controlling the steering, for example, in order to realize the safety and comfort of the driver. It is a system that controls the behavior of. For example, a driving support system called an advanced driver assistance system (ADAS) is known.

On the other hand, in general, an automatic driving system is realized by a cognitive function, a judgment function, and an operation function using various sensors. For example, the cognitive function estimates the position of the target vehicle and the periphery of the target vehicle by using the results detected by various sensors mounted on the target vehicle and the map data showing the current position of the target vehicle and its surrounding area. Performs cognitive processing such as detection of peripheral objects such as other vehicles and pedestrians in the area. The judgment function generates a driving action plan of the target vehicle and a track candidate of the target vehicle based on the result of the cognitive processing. The operation function can control the behavior of the target vehicle based on the driving action plan and the track candidate generated by the judgment function. In recent years, some driving support and automatic driving systems have been developed that employ learning models such as artificial neural networks (ANN).

In order to verify the reliability and safety of such a driving support or automatic driving system, an actual driving test in which a vehicle equipped with the system is actually driven is required. However, it takes a lot of time to commercialize the system only by the actual driving test. Therefore, a test is also conducted in which driving support or automatic driving is simulated using a computer in a virtual simulation environment that imitates a real environment. Such a simulation technique is disclosed in, for example, Patent Document 1 (Re-Table 2019/021898).

Re-table 2019/021898 Gazette

In order to fully verify the reliability and safety of a system for driving assistance or autonomous driving by computer simulation, a large amount of data showing a wide variety of simulation environments is required. Such data can be generated, for example, by performing analysis processing of sensing data centering on video data acquired from a sensor mounted on an actual vehicle (for example, detection of peripheral objects such as other vehicles).

However, since the calculation cost required for analysis of a large amount of sensing data is very high, there is a problem that this leads to an increase in the development cost of the system or a lengthening of the development period.

In view of the above, an object of the present disclosure is to provide a data generation device, a method and a computer program for generating simulation data, which enables reduction of calculation cost required for simulation of driving assistance or automatic driving.

The data generation device according to the first aspect of the present disclosure includes a vehicle information storage unit that stores vehicle information with a time stamp indicating the state of the vehicle in real space, and a time stamped object relating to a peripheral object in the peripheral region of the vehicle. It is a data generation device that generates simulation data by referring to the object information storage unit that stores information, and is the same time within a specified time and allowable range from at least the object information and the vehicle information. An information search unit that performs search processing to search for multiple conceptual elements with the time stamp of, and the multiple conceptual elements found are classified into multiple conceptual classes, and each of the plurality of conceptual classes A multi-class classification unit that generates class data showing the relationship between the conceptual elements classified into each concept class, and a plurality of class data generated by the multi-class classification unit for each of the plurality of concept classes. It includes a structured data generation unit that generates a structured data set having the time stamp by relating to each other.

In the method according to the second aspect of the present disclosure, a vehicle information storage unit in which time-stamped vehicle information indicating the state of the vehicle in the real space is stored, and time-stamped object information regarding peripheral objects in the peripheral region of the vehicle are stored. It is a method of generating simulation data by referring to the stored object information storage unit, and a time stamp of the same time within a specified time and an allowable range is obtained from at least the object information and the vehicle information. A step of performing a search process for finding a plurality of conceptual elements having the same, a step of classifying the plurality of found conceptual elements into a plurality of conceptual classes, and each conceptual class of the plurality of conceptual classes and each of the conceptual classes. A structure having the time stamp by correlating the step of generating class data showing the relationship between the conceptual elements classified in the above and the plurality of class data generated for each of the plurality of conceptual classes. It includes a step to generate a conversion data set.

The computer program according to the third aspect of the present disclosure is a computer program read from the non-volatile memory and to be executed by one or more processors, and the method according to the second aspect is applied to the one or more processors. It is configured to run.

According to the first to third aspects of the present disclosure, it is possible to generate a structured data set having a time stamp based on the time stamped vehicle information and the time stamped object information. Since the amount of data in this structured data set is very small compared to that of sensing data, the data storage cost can be reduced. Further, since the structured data set has a data structure that can be used for driving support or automatic driving simulation with almost no preprocessing, the calculation load of the simulation can be reduced. Therefore, by executing the simulation using the structured data set, it is possible to easily reduce the development cost or the development period of the driving support or automatic driving system.

It is a figure which shows the schematic structure of the data generation system which concerns on one Embodiment of this disclosure. It is a schematic diagram which shows the moving image data which consists of a series of image frames, and the example of an image frame at a certain time. It is a figure which shows the relation example between one concept class, and the attribute information (concept element) which embodies the concept shown by this concept class. It is a figure which shows the relationship example between the vehicle class which shows the concept of "vehicle", and the attribute information classified into a vehicle class. It is a figure which shows roughly the target vehicle traveling on the ramp to the expressway, and the surrounding situation. It is a figure which conceptually represents an example of the structured data set generated in the case of the situation shown in FIG. It is a figure which conceptually shows the structured data set which has each time stamp of a series of time. It is a flowchart which shows typically an example of the data generation method which concerns on one Embodiment of this disclosure. It is a figure which shows the configuration example of the simulator which executes the simulation of driving support or automatic driving using the structured data set group. It is a flowchart which shows an example of the operation procedure of a simulator. It is a block diagram which shows the schematic structure of the data generation system which concerns on other embodiment of this disclosure. It is a flowchart which shows typically an example of the data generation method which concerns on other embodiment of this disclosure. It is a block diagram which shows the example of the hardware composition which realizes the function of a data generation system.

Next, various embodiments and modifications thereof will be described in detail with reference to the drawings. The components with the same reference numerals in the entire drawing shall have the same configuration and the same function.

FIG. 1 is a block diagram showing a schematic configuration of a data generation system 1 according to an embodiment. This data generation system 1 generates a structured data set used in a computer simulation of driving assistance or automatic driving. As shown in FIG. 1, the data generation system 1 includes a data storage unit 10 in which information actually collected using a vehicle existing in a real space (hereinafter referred to as a “target vehicle”) is stored. It includes a data generation device 20 that generates a structured data set for computer simulation with reference to a data storage unit 10, and a structured information database (structured information DB) 30 that stores the structured data set. A time stamp is added to each structured data set.

As the data storage unit 10, a large-capacity storage medium such as a hard disk drive (Hard Disk Drive, HDD) or a solid state drive (Solid State Drive, SSD) can be used.

The data storage unit 10 includes an object information storage unit 11 in which object information with a time stamp regarding peripheral objects in the peripheral area of the target vehicle is stored in advance, and a time stamp indicating the position state, behavior state, and control state of the target vehicle. The vehicle information storage unit 12 in which vehicle information is stored in advance and the sensing data storage unit 13 in which sensing data generated by using one or a plurality of active sensors for observing the peripheral area of the target vehicle are stored. It is composed of including.

The object information stored in the object information storage unit 11 includes information recognized by analyzing the sensing data of the external world sensor mounted on the target vehicle. Examples of the external sensor include passive sensors such as an image pickup sensor and an infrared temperature sensor, and active sensors such as a lidar (Laser imaging Detection And Ringing) device, a radar (RADAR: RAdio Detection And Ringing) device, and an ultrasonic sensor. Will be. The radar device irradiates the peripheral area of the target vehicle with radio waves such as millimeter waves, receives the radio waves reflected or scattered by the peripheral objects, and processes the received signals to determine the relative distance to the peripheral objects and the relative distance to the peripheral objects. Sensing data indicating the speed and the relative orientation of surrounding objects can be generated. In addition, the rider device receives the light reflected or scattered by the peripheral object while scanning the peripheral area of the target vehicle with the laser beam, and processes the received signal to determine the relative distance to the peripheral object, the relative speed of the peripheral object, and the relative speed of the peripheral object. Sensing data indicating the relative orientation of peripheral objects can be generated.

By analyzing the sensing data of the external sensor that observes the peripheral area of the target vehicle, for example, the type, size, and various states of the peripheral object can be recognized as object information. Types of peripheral objects include, for example, peripheral vehicles (eg, four-wheeled vehicles, motorcycles and bicycles), pedestrians, and signs that exist in the peripheral areas of the target vehicle (eg, front areas, side areas, and rear areas). , Buildings, obstacles (eg, pebbles on the road) and lane areas.

Specifically, the target vehicle can be equipped with an image recognition sensor capable of recognizing object information from moving image data which is sensing data of the image pickup sensor. By analyzing the moving image data, the image recognition sensor can detect peripheral objects appearing in the moving image data and recognize the object information. FIG. 2 is a schematic diagram showing an example of a moving image data Md composed of a series of image frames arranged along an elapsed time t and an image frame Fd at a certain time. In the image frame Fd, the front image acquired by the image pickup sensor is shown, and the result of detecting a plurality of peripheral vehicles appearing in the front image by the image recognition sensor is shown by a square frame.

In addition, the target vehicle has an active sensor such as a rider device that can generate a high-definition three-dimensional image of the peripheral area of the target vehicle, and by analyzing the three-dimensional image, the peripheral object is detected and the information is recognized. It can also be equipped with a possible 3D image recognition sensor. By analyzing the sensing data, the 3D image recognition sensor can recognize object information such as road shape and road surface condition (for example, road surface unevenness, road surface wetness or road surface freezing) in the peripheral region of the target vehicle. ..

The sensing data storage unit 13 can store time-stamped sensing data of active sensors such as radar devices and rider devices.

The vehicle information storage unit 12 stores vehicle information with a time stamp indicating a position state, a behavior state, and a control state of the target vehicle. For example, the position state of the target vehicle can be detected by using a GNSS (Global Navigation Satellite System) sensor mounted on the target vehicle and highly accurate map data. The behavior state of the target vehicle is the in-vehicle sensors such as the acceleration sensor, yaw rate sensor (sensor that detects the angular speed generated when the target vehicle turns), steering angle sensor, shift position sensor and brake sensor, and the sensing data of these in-vehicle sensors. It can be detected by using an electronic control unit (ECU: Electronic Control Unit) that analyzes the above. As the information indicating the control state of the target vehicle, for example, vehicle control information transmitted through an in-vehicle communication network compliant with a communication protocol such as CAN (Control Area Network) can be used.

The above-mentioned object information, vehicle information, and sensing data can be aggregated in an external server device via a mobile communication network by an electronic control unit having a telematics function mounted on the target vehicle. The data generation system 1 can use the object information, the vehicle information, and the sensing data aggregated in the external server device.

Next, the configuration of the data generation device 20 will be described in detail. As shown in FIG. 1, the data generation device 20 includes an information search unit 23, an information estimation unit 26, a multi-class classification unit 24, a structured data generation unit 25, and a data output unit 27.

The information search unit 23 executes the search process with reference to the object information storage unit 11 and the vehicle information storage unit 12. That is, the information search unit 23 searches for a plurality of data elements called conceptual elements having the same time stamp at the designated time t and the allowable range ΔT from the object information and the vehicle information. Since a time stamp is added to each conceptual element, the information search unit 23 refers to the data storage unit 10 to search for a plurality of conceptual elements having the same time stamp at the same time T, and the searched concept. The element can be acquired from the data storage unit 10.

Here, in the present specification, the "conceptual element" refers to a data element (for example, the size and speed of a "pedestrian") for embodying the concept of a target vehicle or a peripheral object.

On the other hand, the information estimation unit 26 refers to the sensing data storage unit 13 to search for sensing data having a time stamp at the same time as or near the designated time t. Further, the information estimation unit 26 estimates the conceptual element of the object information regarding the peripheral object in the peripheral region of the target vehicle based on the searched sensing data. For example, when the image sensor mounted on the target vehicle is backlit at the specified time t, or when the target vehicle is in a poor visibility environment at the specified time t, the image recognition sensor uses the moving image at the specified time t. It is difficult to recognize peripheral objects from image data with high accuracy. Even in such a case, the information estimation unit 26 analyzes the sensing data stored in the sensing data storage unit 13 to analyze the size of the conceptual element (for example, the “forward traveling vehicle”) related to the peripheral object (for example, the “forward traveling vehicle”). ) Can be estimated.

Further, the information estimation unit 26 can also estimate new object information based on the sensing data stored in the sensing data storage unit 13 and the vehicle information stored in the vehicle information storage unit 12. For example, the information estimation unit 26 determines the absolute speed of the peripheral vehicle based on the relative speed of the peripheral vehicle (for example, the vehicle traveling ahead or the vehicle traveling behind) obtained from the sensing data and the absolute speed of the target vehicle obtained from the vehicle information. The speed can be estimated.

The multi-class classification unit 24 and the structured data generation unit 25 are hierarchically called ontology based on the conceptual elements found by the information search unit 23 and the conceptual elements estimated by the information estimation unit 26. It is possible to generate a structured data set (ontology data set) that represents a conceptual system. As used herein, the term "ontology" refers to a conceptual system composed of a plurality of conceptual classes appearing in knowledge and models and semantic links relating between these conceptual classes.

Specifically, the multi-class classification unit 24 classifies the conceptual elements found by the information search unit 23 and the conceptual elements estimated by the information estimation unit 26 into a plurality of predetermined conceptual classes, and each concept. Generate class data showing the relationship between the class and the conceptual elements classified in each conceptual class. The structured data generation unit 25 generates a structured data set having a time stamp at time T by associating a plurality of class data generated for each of the plurality of conceptual classes with each other. The data output unit 27 outputs the structured data set to the structured information DB 30.

FIG. 3 is a diagram showing an example of a relationship between one concept class and attribute information (concept element) that embodies the concept shown in this concept class. The attribute information as a conceptual element shown in FIG. 3 includes numerical information, state information, physical property information, and functional information. FIG. 4 is a diagram showing an example of a relationship between a vehicle class showing the concept of “vehicle” and attribute information classified into the vehicle class. The attribute information shown in FIG. 4 includes numerical information, state information, physical property information, and functional information. As shown in FIG. 4, the "numerical information" includes attribute values indicating the vehicle speed, acceleration, steering angle, direction, total length of the vehicle, and width of the vehicle, and the "state information" includes ACC (following cruise control). (Adaptive Cruise Control)) Includes attribute values indicating the state, automatic operation state, and communication state, respectively, and "physical property information" includes attribute values indicating vehicle body information and material information, respectively, and a sensor in "functional information". The information includes attribute values indicating temperature sensors, millimeter-wave radars, LiDARs, and image recognition sensors, respectively, and the lighting information in the "functional information" includes attribute values indicating outdoor lights and indoor lights, respectively.

FIG. 5 is a diagram schematically showing a target vehicle H1 traveling on a ramp HR to a highway HW and its surrounding conditions at a certain time. As shown in FIG. 5, the target vehicle H1 is about to proceed to the confluence area A1 between the linear highway HW and the curved ramp HR in the automatic driving state. The highway HW is composed of a first lane L1 and a second lane L1. In the first lane L1, a motorcycle vehicle H2 is traveling in a manual driving state in a region behind the target vehicle H1. The merging area A1 is provided with a sign Rs for displaying speed limit information (for example, a speed limit of 100 km / h). Further, there is a pebble St on the road surface of the confluence area A1. The electronic control unit in the target vehicle H1 uses the above sensor group mounted on the target vehicle H1 and high-precision map data, and uses the sign Rs, the merging area A1, the pebble St, the motorcycle H2, the first lane L1 and the like. The second lane L2 can be detected as peripheral objects, and conceptual elements related to these peripheral objects can be generated as attribute information.

FIG. 6 is a diagram conceptually representing an example of a structured data set with a time stamp at a given time, which is generated in the case of the situation shown in FIG. The structured dataset shown in FIG. 6 is a semantic link that correlates various conceptual classes 51-53, 61-63, 71-72 and these conceptual classes 51-53, 61-63, 71-72. It is composed of C1 to C8. Specifically, as shown in FIG. 6, the target vehicle class 50 includes attribute information such as "four-wheeled vehicle", "automatic running", and "current position coordinates" that embody the concept of "target vehicle". Class data is composed of class data together with attribute information such as "ramp" and "regional coordinates" that embody the concept of "driving road", and highway class 52 is ". Class data is composed of attribute information such as "regional coordinates" and "road shape" that embody the concept of "highway", and the confluence area class 53 includes "regional coordinates" that embody the concept of "confluence area" and Class data is configured together with attribute information such as "relative distance", and the peripheral vehicle class 61 is attribute information such as "motorcycle vehicle", "manual driving" and "regional coordinates" that embody the concept of "peripheral vehicle". In the first lane class 62, class data is provided together with attribute information such as "front area coordinates", "rear area coordinates" and "lane shape" that embody the concept of "first lane". The second lane class 63 constitutes class data together with attribute information such as "front area coordinates", "rear area coordinates" and "lane shape" that embodies the concept of "second lane", and is a marker. Class 71 constitutes class data together with attribute information such as "speed limit information" that embodies the concept of "sign", and obstacle class 72 embodies the concept of "obstacle", "stone" and " Class data is composed of attribute information such as "relative distance" and "relative orientation".

Further, as shown in FIG. 6, the semantic link C1 correlates the target vehicle class 50 and the traveling road class 51, and the semantic link C2 correlates the target vehicle class 50 and the merging area class 53. Attached, the semantic link C3 correlates the merging area class 53 and the sign class 71, the semantic link C4 correlates the merging area class 53 and the obstacle class 72, and the semantic link C5 is the target vehicle. Class 50 and highway class 52 are interrelated, semantic link C6 correlates highway class 52 and first lane class 62, and semantic link C7 is highway class 52 and second lane class. 63 is related to each other, and the semantic link C8 is related to the peripheral vehicle class 61 and the first lane class 62.

The information search unit 23, the information estimation unit 26, the multi-class classification unit 24, and the structured data generation unit 25 are used for search processing, conceptual element estimation, class data generation, and time-stamped structuring each time a specified time is given. Perform each dataset generation. As a result, as shown in FIG. 7, the structured data sets X (T ₁ ), X (T ₂ ), X (T ₃ ) having a series of time stamps T ₁ , T ₂ , T ₃ , ... ), ... can be generated as time-series data.

Next, the operation procedure of the data generation device 20 will be described with reference to FIG. FIG. 8 is a flowchart schematically showing an example of a data generation method according to an embodiment. Hereinafter, it is assumed that the resolution (time accuracy) of the time stamp of the vehicle information is lower than the resolution (time accuracy) of the time stamp of the object information.

Referring to FIG. 8, first, the time t (k) is specified (step S10). k is a number that specifies the designated time t (k). The designated time t (k) may be given by a controller (not shown) outside the data generation device 20 or by the inside of the data generation device 20 (for example, the information search unit 23).

The information search unit 23 searches for a single or a plurality of conceptual elements having a time stamp of a designated time t (k) and a time T (k) corresponding to the designated time t (k) from the object information stored in the object information storage unit 11. (Step S11). When the finding of the conceptual element is successful (YES in step S12), the process shifts to step S13, and when the finding of the conceptual element fails (NO in step S12), the process shifts to step S32. do.

In step S13, the information search unit 23 is within the permissible range with the time T (k) indicated by the time stamp of the conceptual element found in step S11 from the vehicle information stored in the vehicle information storage unit 12. Attempts to find a single or multiple conceptual element with the same time stamp. When the finding of the conceptual element is successful (YES in step S14), the process shifts to step S17, and when the finding of the conceptual element fails (NO in step S14), the process shifts to step S15. do.

In step S15, the information search unit 23 uses the vehicle information stored in the vehicle information storage unit 12 as a time stamp at a time T (m) (m ≠ k) different from the same time T (k). Gets a single or multiple conceptual elements with. Subsequently, the information search unit 23 interpolates a new conceptual element based on the single or a plurality of conceptual elements acquired in step S15 (step S16). Here, it is desirable that the different time T (m) is as close as possible to the time T (k). For example, one or both of the times T (k-1) and T (k + 1), which are one unit time before and after the time T (k), may be selected as the time T (m).

For example, in step S16, the information search unit 23 has concepts such as the current position coordinates (latitude, longitude and altitude) of the target vehicle, speed, steering angle, acceleration, brake command value, shift position command value, and access opening degree. Elements can be interpolated.

In step S17, the singular or a plurality of conceptual elements found in step S13 are acquired from the vehicle information storage unit 12.

Next, the information estimation unit 26 determines whether or not the sensing data stored in the sensing data storage unit 13 is available (step S18). That is, the information estimation unit 26 refers to the sensing data storage unit 13 and determines whether or not there is sensing data having a time stamp at the same time as or near the designated time t (k). When it is determined that the sensing data is not available (NO in step S18), the process shifts to step S20.

When it is determined that the sensing data is available (YES in step S18), the information estimation unit 26 acquires the available sensing data from the sensing data storage unit 13, and another object based on the sensing data. The conceptual elements of information can be estimated (step S19). In this case, for example, the information estimation unit 26 may analyze available sensing data to newly estimate conceptual elements related to peripheral vehicles (for example, the size and operating state of "peripheral vehicles"). The information estimation unit 26 outputs the estimated conceptual element to the information search unit 23.

Here, the information estimation unit 26 may estimate new object information based on both the available sensing data and the vehicle information stored in the vehicle information storage unit 12. For example, the information estimation unit 26 may use the relative speed of a peripheral vehicle (for example, a vehicle traveling ahead or a vehicle traveling behind) obtained from available sensing data and the absolute speed of the target vehicle obtained from the vehicle information. The absolute velocity of can be estimated as a new conceptual element.

In the next step S20, the information search unit 23 acquires the conceptual element found in step S11 from the object information storage unit 11. At this time, the information search unit 23 integrates the conceptual elements interpolated in step S16, the conceptual elements acquired in step S17, the conceptual elements estimated in step S19, and the conceptual elements acquired in step S20. Supply to the classification unit 24.

After that, the multi-class classification unit 24 classifies the conceptual elements supplied from the information search unit 23 into a plurality of predetermined conceptual classes (step S21). Subsequently, the multi-class classification unit 24 generates class data showing the relationship between each concept class and the conceptual elements classified into each concept class (step S22).

Next, the structured data generation unit 25 generates a structured data set having a time stamp of time T (k) by relating a plurality of class data generated for each of the plurality of conceptual classes to each other. (Step S23). The data output unit 27 outputs the structured data set generated in step S23 to the structured information DB 30.

Next, it is determined whether or not to continue the process (step S32). When it is determined to continue the process (YES in step S32), the information search unit 23 increments the designated time t (k) (step S33). That is, the number k that specifies the designated time t (k) is incremented by 1. After that, steps S11 to S32 are executed for the incremented designated time t (k). When it is determined in step S32 that the process is not continued (NO in step S32), the process ends.

As described above, the data generation device 20 creates a structured data set having a time stamp based on the object information with a time stamp, the vehicle information with a time stamp, and the sensing data with a time stamp each time a specified time is given. Can be generated. Since the amount of data in the structured data set is very small compared to that in the sensing data, the data storage cost can be reduced. Further, since the structured data set has a data structure that can be used for driving support or automatic driving simulation with almost no preprocessing, the calculation load of the computer simulation can be reduced. Therefore, it is possible to easily reduce the development cost or the development period of the driving support or automatic driving system by executing the computer simulation using the structured data set.

FIG. 9 is a diagram schematically showing a configuration example of a simulator 40 that executes a simulation of driving support or automatic driving using a structured data set group stored in the structured information DB 30. As shown in FIG. 9, the simulator 40 has an input interface (input I / F) 41 that acquires a structured data set with a time stamp from the structured information DB 30, and a structured data set acquired by the input I / F 41. Judgment unit 42 that executes judgment processing for driving support or automatic driving under a virtual simulation environment based on the above, and operation processing that operates the virtual vehicle based on the result of the judgment processing is executed and the result is output. The operation unit 43 is provided. It is possible to verify the reliability and safety of the driving support or automatic driving system based on the output of the operation unit 43.

FIG. 10 is a flowchart showing an example of the operation procedure of the simulator 40. As shown in FIG. 10, when the time T (k) is specified (step S40), the input I / F 41 transfers a structured data set having a time stamp of the specified time T (k) from the structured information DB 30. Attempt to acquire (step S41). When the acquisition of the structured data set is successful (YES in step S42), the determination unit 42 executes the determination process based on the structured data set (step S43), and then the operation unit 43 performs the determination process. The operation process is executed based on the result of (step S44). After that, when the designated time T (k) is incremented (step S45), steps S41 and S42 are executed. If the acquisition of the structured data set fails in step S42, the simulation ends. By supplying the structured data set to the determination unit 42 at each designated time T (k) in this way, it is possible to verify the certainty of the logic of the driving support or automatic driving system. For example, it is possible to verify the hiyari hat phenomenon.

Next, the data generation system 2 according to another embodiment will be described with reference to FIGS. 11 and 12.

FIG. 11 is a block diagram showing a schematic configuration of the data generation system 2 according to another embodiment. As shown in FIG. 11, the data generation system 2 includes a data storage unit 10, a data generation device 20, a data generation device 21, and a structured information DB 30. The data generation device 21 includes an information search unit 23, an information estimation unit 26, a multi-class classification unit 24, a structured data generation unit 25, a data output unit 28, and a buffer memory 29. The configuration of the data generation system 2 is the same as the configuration of the data generation system 2 of FIG. 1 except that the data output unit 28 and the buffer memory 29 are provided in place of the data output unit 27 of FIG.

In the data generation device 21 of the present embodiment, as described above, the structured data generation unit 25 generates a structured data set X (T) having a time stamp of the time T corresponding to the designated time t. The buffer memory 29 stores a structured data set X (T—α) generated before the structured data set. The data output unit 28 includes a structured data set X (T) generated by the structured data generation unit 25 and a structured data set X (T) stored in the buffer memory 29 and having a time stamp of time T—α. Compared with T-α), it is determined between the attribute value of the conceptual element constituting the structured data set X (T) and the attribute value of the conceptual element constituting the structured data set X (T-α). Determine if there is a difference greater than or equal to the threshold of. When it is determined that there is no difference of a predetermined threshold value or more, the data output unit 28 does not output the structured data set X (T) to the structured information DB 30. As a result, it is possible to avoid storing the structured data set X (T) in the structured information DB 30 when it can be evaluated that there is almost no change in the environment between the time T—α and the time T. For example, it is possible to avoid storing redundant data in the structured information DB 30 for a scene in which the target vehicle is in a stationary state.

Next, the operation procedure of the data generation device 21 will be described with reference to FIG. FIG. 12 is a flowchart schematically showing an example of a data generation method according to another embodiment. The flowchart of FIG. 12 is the same as the flowchart of FIG. 8 except that it has steps S24, S25, and S31.

Referring to FIG. 12, when the structured data set having the time stamp of time T (k) is generated in step S23, the data output unit 28 uses the currently generated structured data set X (T (k)). ) And the structured data set X (T (k-1)) having a time stamp of time T (k-1) obtained from the buffer memory 29 (step S24). That is, a predetermined value is set between the attribute value of the conceptual element constituting the structured data set X (T (k)) and the attribute value of the conceptual element constituting the structured data set X (T (k-1)). Determine if there is a difference greater than or equal to the threshold.

When there is a difference of a predetermined threshold value or more (YES in step S25), the data output unit 28 outputs the structured data set X (T (k)) to the structured information DB 30 (step S30), and the structured data Set X (T (k)) is stored in the buffer memory 29 (step S31). On the other hand, when there is no difference of more than a predetermined threshold value (NO in step S25), the data output unit 28 does not output the structured data set X (T (k)) to the structured information DB 30, but the structured data. Set X (T (k)) is stored in the buffer memory 29 (step S31). After that, the steps after step S32 are executed.

All or part of each function of the above-mentioned data generators 20 and 21 is a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) that executes a software or firmware program code read from a non-volatile memory. It may be realized by a single or a plurality of processors including an arithmetic unit such as. Alternatively, all or part of each function of the data generators 20 and 21 is a single integrated circuit having a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). Or it can be realized by multiple processors. Alternatively, all or part of each function of the data generators 20 and 21 is realized by a single or multiple processors including a combination of a semiconductor integrated circuit such as a DSP, ASIC or FPGA and an arithmetic unit such as a CPU or GPU. It is also possible.

FIG. 13 is a block diagram schematically showing the configuration of the information processing apparatus 90, which is an example of the hardware configuration that realizes the functions of the data generation systems 1 and 2. The information processing apparatus 90 includes a processor 91, a random access memory (RAM: Random Access Memory) 92, a non-volatile memory 93, a large capacity memory 94, an input / output interface 95, and a signal path 96. The signal path 96 is a bus for connecting the processor 91, the RAM 92, the non-volatile memory 93, the large capacity memory 94, and the input / output interface 95 to each other. The RAM 92 is a data storage area used when the processor 91 executes digital signal processing. When the processor 91 incorporates an arithmetic unit such as a CPU, the non-volatile memory 93 has a data storage area for storing software or firmware program codes executed by the processor 91. The large-capacity memory 94 may be provided with the data storage unit 10 and the structured information DB 30 of FIG. 1 or 11.

Although various embodiments and variations thereof have been described above, the above-described embodiments and variations thereof are merely examples and do not limit the scope of the present invention. For example, the above-mentioned target vehicle is not limited to a passenger car, and the above-described embodiment and its modification can also be applied to a railway vehicle.

It should also be understood that the embodiments can be modified, added and improved as appropriate without departing from the spirit and scope of the present invention. The scope of the invention should be construed on the basis of the statements in the claims and should be understood to include its equivalents.

The data generation device, method and computer program for simulation according to the present disclosure enable the provision of a structured data set suitable for vehicle driving assistance and autonomous driving simulation, and thus vehicle driving assistance technology and autonomous driving technology. It can be used for research and development of.

1: Data generation system, 2: Data generation system, 10: Data storage unit, 11: Object information storage unit, 12: Vehicle information storage unit, 13: Sensing data storage unit, 20, 21: Data generation device, 23: Information Search unit, 24: Multi-class classification unit, 25: Structured data generation unit, 26: Information estimation unit, 27, 28: Data output unit, 29: Buffer memory, 30: Structured information database, 40: Simulator, 41: Input interface (input I / F), 42: judgment unit, 43: operation unit, 90: information processing device, 91: processor, 92: random access memory, 93: non-volatile memory, 94: large capacity memory, 95: input Output interface, 96: Signal path.

Claims (17)

Refer to the vehicle information storage unit that stores the vehicle information with a time stamp indicating the state of the vehicle in the real space, and the object information storage unit that stores the object information with the time stamp regarding the peripheral objects in the peripheral area of the vehicle. It is a data generation device that generates data for simulation.
An information search unit that performs a search process for searching for a plurality of conceptual elements having the same time stamp within a specified time and an allowable range from at least the object information and the vehicle information.
The plurality of found conceptual elements are classified into a plurality of conceptual classes, and class data showing the relationship between each conceptual class among the plurality of conceptual classes and the conceptual elements classified into the respective conceptual classes is obtained. Multi-class classification unit to generate and
It is provided with a structured data generation unit that generates a structured data set having the time stamp by relating a plurality of class data generated for each of the plurality of conceptual classes by the multi-class classification unit to each other. A data generator characterized by. The data generation device according to claim 1, wherein the information search unit, the multi-class classification unit, and the structured data generation unit perform the search process and generate the class data each time the designated time is given. , And a data generator, characterized in that it performs the generation of the structured data set. The data generation device according to claim 1 or 2, wherein the information search unit searches for a conceptual element having a time stamp of a time corresponding to the designated time from the object information, and the vehicle information. A data generation device for finding a conceptual element having a time stamp of the same time as the time indicated by the time stamp of the found conceptual element of the object information. The data generation device according to claim 3.
When the information search unit fails to find a conceptual element having a time stamp of the same time as the time indicated by the time stamp of the found conceptual element of the object information from the vehicle information. , Acquire a single or a plurality of conceptual elements having a time stamp different from the same time from the vehicle information, and interpolate a new conceptual element based on the acquired single or a plurality of conceptual elements. death,
The multi-class classification unit classifies the found conceptual element and the new conceptual element into the plurality of conceptual classes.
A data generator characterized by that. The data generation device according to claim 3 or 4, wherein the resolution of the time stamp of the vehicle information is lower than the resolution of the time stamp of the object information. The data generator according to any one of claims 1 to 5.
Comparing the structured data set with the structured data set generated before the structured data set, the attribute values of the conceptual elements constituting the structured data set and the previously generated structured data set. It also has a data output unit that determines whether there is a difference of a predetermined threshold or more with the attribute values of the conceptual elements that make up the data set.
The data output unit outputs the structured data set only when it is determined that there is no difference of the predetermined threshold value or more.
A data generator characterized by that. The data generator according to any one of claims 1 to 6.
Refer to the sensing data storage unit that stores the time-stamped sensing data generated by using one or more active sensors that observe the peripheral area of the vehicle, and in the peripheral area of the vehicle based on the sensing data. It also has an information estimation unit that estimates the conceptual elements of object information related to peripheral objects.
The multi-class classification unit is a data generation device characterized by classifying the found plurality of conceptual elements and the estimated conceptual elements into the plurality of conceptual classes. The data generation device according to claim 7, wherein the single or plurality of active sensors includes at least one of a radar device, a rider device, and an ultrasonic sensor. The data generation device according to any one of claims 1 to 8, wherein the vehicle information indicates information indicating at least one of a position state, a behavior state, and a control state of the vehicle. A data generator characterized by including. The data generation device according to any one of claims 1 to 9, wherein the object information stored in the object information storage unit is sensing by a single sensor or a plurality of sensors including an image recognition sensor. A data generator characterized in that it is information about peripheral objects recognized by analyzing the data. Refer to the vehicle information storage unit that stores the vehicle information with a time stamp indicating the state of the vehicle in the real space, and the object information storage unit that stores the object information with the time stamp regarding the peripheral objects in the peripheral area of the vehicle. It is a method to generate data for simulation.
A step of searching for a plurality of conceptual elements having the same time stamp within a specified time and an allowable range from at least the object information and the vehicle information, and a step of performing a search process.
Steps to classify the multiple conceptual elements found into multiple conceptual classes, and
A step of generating class data showing the relationship between each concept class among the plurality of concept classes and the conceptual elements classified into each concept class, and
A method comprising: a step of generating a structured data set having the time stamp by relating a plurality of class data generated for each of the plurality of conceptual classes to each other. The method according to claim 11, wherein the step of performing the search process, the step of generating the class data, and the step of generating the structured data set are each given the designated time. A method characterized by being performed on. The method according to claim 11 or 12.
The step of performing the search process is
A step of searching for a conceptual element having a time stamp of a time corresponding to the specified time from the object information, and
A method comprising: finding a conceptual element having a time stamp of the same time as the time indicated by the time stamp of the found conceptual element of the object information from the vehicle information. The method according to claim 13.
When it fails to find a conceptual element having a time stamp of the same time as the time indicated by the time stamp of the found conceptual element in the object information from the vehicle information, the inside of the vehicle information From the step of getting a single or multiple conceptual elements with time stamps different from the same time,
A step of interpolating a new conceptual element based on the acquired singular or plural conceptual elements,
A method further comprising the step of classifying the found conceptual element and the new conceptual element into the plurality of conceptual classes. The method according to any one of claims 11 to 14.
Comparing the structured data set with the structured data set generated before the structured data set, the attribute values of the conceptual elements constituting the structured data set and the previously generated structured data set. A step to determine whether there is a difference of a predetermined threshold or more from the attribute value of the conceptual element constituting the data set, and
A method further comprising a step of outputting the structured data set only when it is determined that there is no difference of the predetermined threshold value or more. The method according to any one of claims 11 to 15.
Refer to the sensing data storage unit that stores the time-stamped sensing data generated by using one or more active sensors that observe the peripheral area of the vehicle, and in the peripheral area of the vehicle based on the sensing data. Steps to estimate conceptual elements of object information about peripheral objects,
A method further comprising a step of classifying the estimated conceptual element into the plurality of conceptual classes. A computer program that is read from a non-volatile memory and is to be executed by one or more processors, wherein the method according to any one of claims 11 to 16 is applied to the one or more processors. A computer program characterized by being configured to run.

Images