CN112181149B - Driving environment recognition method and device and simulated driver - Google Patents

Abstract

The invention provides a method and a device for identifying a driving environment and a simulated driver, wherein the method comprises the following steps: acquiring original eye movement information of a target object and scene information of a virtual driving scene, and preprocessing the original eye movement information to obtain target eye movement information; determining the importance level of an object to be viewed in the virtual driving scene based on the target eye movement information; determining a visual recognition sequence of objects to be viewed in the virtual driving scene based on the target eye movement information and the scene information; and identifying the driving environment of the virtual driving scene based on the importance level and the visual recognition sequence. The invention can better improve the accuracy and efficiency of environment sensing.

Description

Driving environment recognition method, device and simulator driver

technical field

The invention relates to the technical field of visual recognition, in particular to a driving environment recognition method, device and simulated driver.

Background technique

Sensing the environment of unmanned vehicles is an important part of the development of unmanned driving technology. The existing environmental perception methods mainly involve radar-based environmental perception methods. Therefore, related technologies propose an environment perception method based on machine learning to improve the accuracy of environment perception. However, the inventors have found through research that the environment perception method based on machine learning usually increases the number of fully connected layers or global average pooling layers to realize the recognition of various types of visual recognition objects in the environment, but because this method will expand The overall size of the machine learning models used to perceive the environment, resulting in reduced accuracy and efficiency of environment perception.

Contents of the invention

In view of this, the object of the present invention is to provide a driving environment recognition method, device and simulated driver, which can better improve the accuracy and efficiency of environment perception.

In the first aspect, the embodiment of the present invention provides a driving environment recognition method, including: collecting the original eye movement information of the target object and the scene information of the virtual driving scene, and preprocessing the original eye movement information to obtain the target eye movement information. Based on the target eye movement information, determine the importance level of the object to be visually recognized in the virtual driving scene; based on the target eye movement information and the scene information, determine the object to be visually recognized in the virtual driving scene The visual recognition order of the objects: based on the importance level and the visual recognition order, the driving environment of the virtual driving scene is recognized.

In one embodiment, the target eye movement information includes pupil diameter data and blink reflex data; the step of determining the importance level of the object to be recognized in the virtual driving scene based on the target eye movement information, It includes: performing multi-scale geometric analysis on the pupil diameter data to select an object of interest from objects to be recognized in the virtual driving scene to obtain a set of objects of interest; performing harmonic analysis on the blink reflection data, To select a focused object from the objects to be visually recognized in the virtual driving scene to obtain a set of focused objects; according to preset interest weights, preset focus weights, the set of interested objects and the set of focused objects, determine the The importance level of the objects to be recognized in the virtual driving scene.

In one embodiment, the step of performing multi-scale geometric analysis on the pupil diameter data to select an object of interest from the objects to be recognized in the virtual driving scene includes: performing a multi-scale geometric analysis on the pupil diameter data Wavelet transform to obtain the first data set; determine the first interval containing the peak point in the first data set, and identify the target pupil diameter range corresponding to the first interval; based on the sensory sensitivity corresponding to the target pupil diameter range The degree of interest selects an object of interest from objects to be recognized in the virtual driving scene.

In one embodiment, the step of performing harmonic analysis on the blink reflection data to select an object to focus on from the objects to be recognized in the virtual driving scene includes: performing a harmonic analysis on the blink reflection data Fourier transform to obtain a second data set; determine a second interval containing amplitude points in the second data set, and identify the target blink reflection range corresponding to the second interval; based on the target blink The degree of concentration corresponding to the reflection range selects the focus object from the objects to be recognized in the virtual driving scene.

其中，C_i表示第i个瞳孔直径数据，S_1j表示第j个重要等级的瞳孔直径标准值，n表示瞳孔直径数据的总个数；所述预设专注权重w₂的设定方式为：

其中，D_p表示第p个瞬目反射数据，S_2q表示第q个重要等级的瞬目反射标准值，m表示瞬目反射数据的总个数。In an implementation manner, the setting method of the preset interest weight w ₁ is:

Wherein, C _i represents the i pupil diameter data, S _j represents the pupil diameter standard value of the j important level, and n represents the total number of pupil diameter data; the setting method of the preset concentration weight w ₂ is:

Among them, _Dp represents the pth blink reflection data, S _2q represents the blink reflection standard value of the qth important level, and m represents the total number of blink reflection data.

In one embodiment, the step of determining the visual recognition sequence of objects to be recognized in the virtual driving scene based on the eye movement information and the scene information includes: based on the target eye movement information and Described scene information, obtain line-of-sight point data; Determine the line-of-sight point data that satisfies the preset condition as target visual recognition data; Wherein, the preset condition includes gaze time condition and gaze angle condition; According to the visual recognition time and the target The association relationship between the visual recognition data determines the visual recognition order of the target visual recognition data corresponding to the objects to be recognized.

In one embodiment, the step of preprocessing the original eye movement information to obtain target eye movement information includes: using Lagrangian interpolation algorithm and empirical mode decomposition method to process the original eye movement information Perform preprocessing to obtain target eye movement information.

In the second aspect, the embodiment of the present invention also provides a driving environment recognition device, including: a data collection module, used to collect the original eye movement information of the target object and the scene information of the virtual driving scene, and analyze the original eye movement information Preprocessing is performed to obtain target eye movement information; a level determination module is used to determine the importance level of objects to be recognized in the virtual driving scene based on the original eye movement information; an order determination module is used to determine the order based on the target eye movement information and the scene information to determine the visual recognition sequence of the objects to be visually recognized in the virtual driving scene; the environment recognition module is used for driving in the virtual driving scene based on the importance level and the visual recognition sequence The environment is identified.

In a third aspect, an embodiment of the present invention also provides a simulated driver, including a simulator screen, a processor, and a memory; the simulator screen is used to display the virtual driving scene, and a computer program is stored on the memory, so The computer program executes any one of the methods provided in the first aspect when executed by the processor.

In a fourth aspect, an embodiment of the present invention further provides a computer storage medium for storing computer software instructions used in any one of the methods provided in the first aspect.

The driving environment recognition method, device and simulated driver provided by the embodiments of the present invention first collect the original eye movement information of the target object and the scene information of the virtual driving scene, and preprocess the original eye movement information to obtain the target eye movement Information, based on the target eye movement information to determine the importance level of the objects to be recognized in the virtual driving scene, and based on the target eye movement information and scene information, determine the visual recognition order of the objects to be recognized in the virtual driving scene, and then based on the importance level and visual recognition sequence to recognize the driving environment of the virtual driving scene. The above method proposes a new method of perceiving the environment. The embodiment of the present invention determines the importance level of the objects to be recognized based on the eye movement information of the target, and determines the recognition sequence of the objects to be recognized based on the eye movement information of the target and the scene information. Therefore, recognizing the driving environment based on the order of visual recognition and the important classification can effectively improve the recognition accuracy and recognition efficiency of the driving environment.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative work.

FIG. 1 is a schematic flowchart of a method for identifying a driving environment provided by an embodiment of the present invention;

Fig. 2 is a schematic structural diagram of a simulated driver provided by an embodiment of the present invention;

Fig. 3 is a process schematic diagram of a method for identifying a driving environment provided by an embodiment of the present invention;

FIG. 4 is a structural framework diagram of a visual search strategy model provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an identification device for a driving environment provided by an embodiment of the present invention;

Fig. 6 is a schematic structural diagram of another simulated driver provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. the embodiment. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

At present, the existing environment perception methods have the problems of low perception accuracy and low perception efficiency. For example, a vehicle radar multi-target recognition method based on FMCW (Frequency Modulated Continuous Wave, frequency modulated continuous wave) is proposed in the related art. Transform) operation to extract and match the main frequency, and calculate the target speed and distance, which reduces the misjudgment rate of false targets; the related technology proposes an intelligent multi-target comprehensive recognition method based on data mining, and applies data mining technology to In the field of target recognition, two target recognition ideas based on target feature knowledge recognition and target related knowledge recognition are proposed, which provide automatic and intelligent means for multi-target recognition; related technologies propose a vehicle-based surrounding environment perception system and its control method, The main control unit is connected to the millimeter-wave radar unit and the visual perception module through the CAN bus; the related technology proposes a multi-lidar fusion recognition method for intelligent vehicles based on target characteristics, and performs target matching according to similarity, and performs target tracking to obtain target motion characteristics and perform Corrected to enhance the ability to identify targets. However, the above technologies all have the problem of low sensing accuracy or low sensing efficiency. In order to improve this problem, embodiments of the present invention provide a driving environment recognition method, device, and simulated driver, which can better improve the accuracy and efficiency of environment perception.

In order to facilitate the understanding of this embodiment, a method for identifying a driving environment disclosed in an embodiment of the present invention is firstly introduced in detail, referring to a schematic flowchart of a method for identifying a driving environment shown in FIG. 1 , the method mainly includes the following steps S102 to step S108:

Step S102, collecting the original eye movement information of the target object and the scene information of the virtual driving scene, and performing preprocessing on the original eye movement information to obtain the target eye movement information. Among them, the target object is the driver, and the original eye movement information is the unpreprocessed eye movement information, such as pupil diameter data, blink reflex data, etc. There may be missing points or abnormal points in these original eye movement information, and the scene information It can be the image data of the virtual driving scene, and the eye movement information of the target is the eye movement information obtained after preprocessing, and can also include pupil diameter data, blink reflex data, etc. In one embodiment, a camera for collecting raw eye movement information of a target object and a camera for collecting scene information of a virtual driving scene may be respectively set up, and the data collected by each camera may be respectively obtained.

Step S104, determining the importance level of the object to be recognized in the virtual driving scene based on the eye movement information of the target. Wherein, multiple objects to be recognized are set in the virtual driving scene, and the objects to be recognized may include, for example, pedestrians, motor vehicles, bicycles, and marking lines. The importance level is used to represent the importance of the object to be recognized. For example, the importance level can be divided into five levels: key goals, important goals, moderately important goals, general goals, and unimportant goals. In an implementation manner, multiple importance levels may be divided in advance, and the importance level to which each eye movement information belongs is determined based on the cognitive neurological reflection and cognitive psychological reflection represented by the eye movement information.

Step S106, based on the target eye movement information and the scene information, determine the visual recognition order of the objects to be visually recognized in the virtual driving scene. The visual recognition order can be used to represent the order in which the target object observes the objects to be visually recognized in the virtual driving scene. In one embodiment, the visual recognition order is jointly determined according to the target eye movement information, scene information and visual recognition time. Optionally, the target visual recognition data can be determined based on the target eye movement information and scene information. The association between the recognition time and the target visual recognition data can be combined with the sequence of the visual recognition time to determine the visual recognition order of the objects to be visually recognized corresponding to the target visual recognition data.

In step S108, the driving environment of the virtual driving scene is recognized based on the importance level and the recognition order. In one embodiment, the driving environment of the virtual driving scene can be represented by the importance level and the visual recognition sequence.

The above-mentioned driving environment recognition method provided by the embodiment of the present invention proposes a new method of perceiving the environment. The embodiment of the present invention determines the importance level of the object to be recognized based on the target eye movement information, and based on the target eye movement information and the scene The information determines the visual recognition sequence of the objects to be visually recognized, so that the driving environment can be recognized based on the visual recognition sequence and importance classification, which can effectively improve the recognition accuracy and recognition efficiency of the driving environment.

In one embodiment, the above-mentioned identification method of the driving environment can be performed by a simulated driver, which is an experimental device for a virtual driving scene, and a certain number of drivers with rich driving experience are selected as target objects, and the target objects wear head-mounted The eye tracker sits in the cockpit of the driving simulator. Referring to the schematic structural diagram of a simulated driver shown in Figure 2, the simulated driver includes a driving simulation cabin and a simulator screen, and the simulator screen can display a virtual driving scene of multi-view recognition targets according to the setting of experimental requirements (also called driving environment), the driving simulation cabin can simulate the driving operation environment, and the simulator screen is set at a preset distance (such as 5-8 meters) directly in front of the driving simulation cabin. In practical applications, the head-mounted eye tracker includes a scene camera and an eye movement camera. The scene camera is installed to collect scene information, and the eye movement camera is used to collect raw eye movement information. The target object is the subject of original eye movement information generation. In the embodiment of the present invention, when selecting a driver, an experienced driver with a driving mileage of 50,000 km can be selected as the standard for the experiment. The number of drivers is N>= 10. The ratio of male to female is about 3:1, and they are both physically and mentally healthy.

Optionally, the simulated driver can be set in advance, so that the simulated driver provides a virtual driving scene, and simultaneously collects scene information of the virtual driving scene and original eye movement information of the target object. Among them, the virtual driving scene refers to the driving environment with a single visual recognition target and the multi-visual recognition target driving environment under different driving tasks.

Considering that there may be abnormal data such as missing points or abnormal points in the collected original eye movement information and scene information, the embodiment of the present invention can preprocess the original eye movement information and scene information. Optionally, Lagrang can be used to The daily interpolation algorithm and the empirical mode decomposition method perform preprocessing on the original eye movement information such as elimination, compensation, noise reduction, etc. to obtain the target eye movement information, and then use the preprocessed target eye movement information for analysis. in. The original eye movement information is the sight point position data, pupil diameter data and blink reflex data of the target object at different times, where the sight point position data (X0,Y0)={(X1,Y1),(X2,Y2)... (Xi, Yi),...(Xn, Yn)}, pupil diameter data D0=(D1, D2...Di,...Dn), blink reflex data B0=(B1, B2...Bi,...Bn).

获得缺失值或异常值对应点处的近似值，其中，n表示视线点数据的总个数，x_a表示第a个视线点数据，x_b表示第b个视线点数据。(3)使用EMD(Empirical Mode Decomposition，经验模态分解)找出瞳孔直径数据D0和瞬目反射数据B0中的极大值和极小值，计算两者均值，然后基于该均值对数据进行降噪。其中，x(t)为原始信号，在本发明实施例中，原始信号可以为视线点数据。经过上述预处理后，目标眼动信息为视线点位置数据(X1,Y1)、瞳孔直径数据D1、瞬目反射数据B1。For the scene information and original eye movement information collected above, perform the following (1) to (3): (1) Determine missing values and outliers in the data. (2) Use the Lagrange interpolation formula at missing points and outliers

Obtain the approximate value at the corresponding point of the missing value or outlier, where n represents the total number of sight point data, x _a represents the a-th sight point data, and x _b represents the b-th sight point data. (3) Use EMD (Empirical Mode Decomposition, Empirical Mode Decomposition) to find the maximum value and minimum value in the pupil diameter data D0 and the blink reflection data B0, calculate the mean value of the two, and then reduce the data based on the mean value noise. Wherein, x(t) is an original signal, and in the embodiment of the present invention, the original signal may be line-of-sight data. After the above preprocessing, the eye movement information of the target is the gaze point position data (X1, Y1), pupil diameter data D1, and blink reflex data B1.

In practical applications, the above target eye movement information includes pupil diameter data and blink reflex data. Based on this, an embodiment of the present invention provides a method for determining the importance level of objects to be recognized in a virtual driving scene based on the target eye movement information. For specific implementation, see the following steps 1 to 3:

Step 1: Perform multi-scale geometric analysis on the pupil diameter data to select objects of interest from objects to be recognized in the virtual driving scene to obtain a set of objects of interest. In one embodiment, the object of interest can be determined according to the method shown in the following steps 1.1 to 1.3:

α为控制尺度，τ为控制位置，α和τ为任意实数，ψ为母小波，上述第一数据集合也可称之为初始待匹配瞳孔直径数据。Step 1.1, performing wavelet transformation on the pupil diameter data to obtain the first data set. In the specific implementation, the unified serial number of the pupil diameter contained in the pupil diameter data D1 is used as the label information of each sampling point, and the set U=(1,2...i,...n) of all sampling points is obtained, and then the pupil The diameter data D1 is subjected to wavelet transformation to obtain the first data set Q=(Q1, Q2...Qi,...Qn), and the data serial number of the sampling point in the first data set Q is L(L1, L2...Li,...Ln). Among them, the formula of wavelet transform is:

α is the control scale, τ is the control position, α and τ are arbitrary real numbers, ψ is the mother wavelet, and the first data set above can also be called the initial pupil diameter data to be matched.

Step 1.2, determining the first interval including the peak point in the first data set, and identifying the target pupil diameter range corresponding to the first interval. In practical applications, the first interval including the peak point may be determined from the first data set, so that the pupil diameter range corresponding to the first interval is determined as the target pupil diameter range.

Step 1.3, selecting an object of interest from the objects to be recognized in the virtual driving scene based on the degree of interest corresponding to the target pupil diameter range. In one embodiment, the peak point of each peak in the pupil diameter in the first data set Q is identified, and the set of the sampling point data numbers of the target pupil diameter corresponding to the peak point in the first data set Q is I(I1 , I2...Ii,...In), the embodiment of the present invention obtains the variation range of the pupil diameter through the set I, and takes the corresponding object to be recognized when the pupil diameter is large as the object of interest. In an optional implementation, the range of the pupil diameter data in the target eye movement information can be determined, and the range of the pupil diameter data is divided into multiple intervals (such as five intervals), and it is determined that each interval corresponds to a different The degree of interest, so that the object of interest is selected from the objects to be recognized in the virtual driving scene in descending order of the degree of interest.

Step 2: Harmonic analysis is performed on the blink reflection data to select focused objects from the objects to be recognized in the virtual driving scene to obtain a set of focused objects. In one embodiment, the focus object can be determined according to the method shown in the following steps 2.1 to 2.3:

a_n和b_n为实频率分量的振幅，T为函数周期，

n为整数且n>0，上述第二数据集合也可称之为初始待匹配瞬目反射数据集合。Step 2.1, perform Fourier transform on the blink reflection data to obtain the second data set. During concrete realization, the unified serial number of blink reflection in the blink reflection data B1 is used as the mark information of each sampling point, obtains the collection F=(1,2...i,...n) of all sampling points; Then to blink The reflection data B1 is subjected to Fourier transform to obtain the second data set E=(E1, E2...Ei,...En), and the data serial numbers of the sampling points in the second data set E are P=(P1, P2...Pi,...Pn ). Among them, the Fourier transform formula is:

a _n and b _n are the amplitudes of real frequency components, T is the function period,

n is an integer and n>0, the above second data set may also be referred to as an initial to-be-matched blink reflection data set.

Step 2.2, determining the second interval containing the amplitude points in the second data set, and identifying the target blink reflection range corresponding to the second interval. In practical applications, the second interval including the amplitude point may be determined from the second data set, so that the blink reflection range corresponding to the second interval is determined as the target blink reflection range.

Step 2.3, based on the degree of concentration corresponding to the target blink reflex range, select the focus object from the objects to be recognized in the virtual driving scene. Identify the amplitude points of the blink reflection data in the second data set P, and record the set of sampling point data numbers of the amplitude points in the second data set P as R=(R1, R2...Ri,...Rn). In the embodiment of the present invention, the amplitude range of the blink reflection is obtained through the set R, and the object to be recognized corresponding to the data with a small blink reflection is taken as the focus object. In an optional implementation manner, the range of the blink reflection data in the eye movement information of the target can be determined, and the range of the blink reflection data is divided into multiple intervals (such as five intervals), and the corresponding interval of each interval is determined. Different degrees of concentration, so that the focus objects are selected from the objects to be recognized in the virtual driving scene in descending order of concentration.

其中，C_i表示第i个瞳孔直径数据，S_1j表示第j个重要等级的瞳孔直径标准值，n表示瞳孔直径数据的总个数；预设专注权重w₂(也即，瞬目反射的权重w₂)可表示为：

其中，D_p表示第p个瞬目反射数据，S_2q表示第q个重要等级的瞬目反射标准值，m表示瞬目反射数据的总个数。从而将不同驾驶任务下多视认目标环境中待视认对象的重要等级分为五个等级，包括：关键目标、重要目标、中等重要目标、一般目标、不重要目标。Step 3, according to the preset interest weight, preset focus weight, interest object set and focus object set, determine the importance level of the objects to be recognized in the virtual driving scene. Optionally, the larger the pupil diameter data, the greater the possibility that the object to be viewed and recognized corresponding to the pupil diameter data is an object of interest; the smaller the blink reflection data, the greater the possibility that the object to be recognized and recognized corresponding to the blink reflection data is focused The more likely the object is, the target object who is both interested and focused is regarded as a higher-level object to be recognized. In order to facilitate the understanding of the above step 3, the embodiment of the present invention exemplarily provides an implementation manner of determining the importance level of the object to be recognized. The present invention divides the pupil diameter data and the corresponding psychological state into five categories: pupil diameter < Pupil diameter 2.5mm is no interest object, pupil diameter 2.5-4mm is general interest object, pupil diameter 4-5.5mm is medium interest object, pupil diameter 5.5-7mm is important interest object, pupil diameter >7mm is extremely interest object; instantaneous The eye reflex data is the blinking movement, and the blink reflex data and the corresponding cognitive neural states are divided into five categories: blink reflex > 16 times/min is inattentive state, blink reflex 14-16 times/min is general concentration State, blink reflex 12-14 times/min is moderate concentration state, blink reflex 10-12 times/min is abnormal state, pupil diameter <10 times/min is extreme concentration state. Combining the clustering weight method, the data of the set Q and the set E are combined with the grade standard to calculate the weight: the measured value of the pupil diameter data is selected as Qi, the measured value of the blink reflex data is selected as Ei, and the j-th index of the i-th index is The standard values of grades are S _ij , i=1, 2; j=1, 2, 3, 4, 5. Let W _ij be the weight of the i-th indicator at the j-th level, then the preset interest weight w ₁ (that is, the weight w ₁ of the pupil diameter) can be expressed as:

Among them, C _i represents the i pupil diameter data, S _1j represents the pupil diameter standard value of the j important level, n represents the total number of pupil diameter data; preset focus weight w ₂ (that is, the blink reflex Weight w ₂ ) can be expressed as:

Among them, _Dp represents the pth blink reflection data, S _2q represents the blink reflection standard value of the qth important level, and m represents the total number of blink reflection data. Therefore, the importance levels of objects to be recognized in the multi-view target environment under different driving tasks are divided into five levels, including: key targets, important targets, moderately important targets, general targets, and unimportant targets.

For the above step S106, the steps shown in the following steps a to c can be performed based on the target eye movement information and scene information to determine the visual recognition order of the objects to be visually recognized in the virtual driving scene:

Step a, based on the eye movement information of the target and the scene information, the gaze point data is obtained.

In step b, the sight point data satisfying the preset condition is determined as the target visual recognition data. Wherein, the preset conditions include gaze time conditions and gaze angle conditions. In one embodiment, the gaze time condition may be that the gaze duration is greater than a preset time threshold, and the gaze angle condition may be that the gaze angle range is within a preset angle range. First of all, it is necessary to clarify the difference between the target visual recognition data and the conventional glance point of sight. The target visual recognition data is obtained when the eye movement speed is lower than 5deg/s, the viewing angle deviation is a*a where a<=0.41°, and the fixation duration>100ms Gaze point position; among them, fixation duration is the duration that the central position of the visual axis remains unchanged, that is, the time it takes to extract information from the gaze object; gaze angle is the angle at which the eyeball rotates in the horizontal and vertical directions relative to the head . Taking (0,0) as the origin, that is, the intersection of the vertical line from the eyeball to the vertical plane and the vertical plane, the angle a between the projection of the line between the eyeball and the fixation point on the horizontal plane and the vertical plane is The angle of the line of sight point position on the horizontal plane.

则实际的目标视认视线点位置

通过对应矩阵的时间项即可得到对应的目标视认顺序。Step c, according to the correlation between the visual recognition time and the target visual recognition data, determine the visual recognition sequence of the target visual recognition data corresponding to the objects to be recognized. Since the experiment is sampled at a frequency of 100 Hz, in the complex environment of different driving tasks and multiple visual recognition targets, using MATLAB image processing technology, the set of target visual recognition data (X2 , Y2), the location set of the target visual recognition data in the time domain is (Xt, Yt), and the corresponding matrix between the visual recognition time and the target visual recognition sight point position is established

Then the actual target visual recognition point position

The corresponding target visual recognition sequence can be obtained through the time item of the corresponding matrix.

In order to facilitate the understanding of the driving environment recognition method provided by the above embodiments, the embodiment of the present invention provides another driving environment recognition method, refer to the process diagram of a driving environment recognition method shown in Figure 3, the pupil The multi-scale geometric analysis of the diameter data and the harmonic analysis of the blink reflection data are combined to reflect the two types of indicators in cognitive neurology and cognitive psychology to obtain the importance level of the object to be recognized. In addition, the sight point The data is divided into sight points, and the target visual recognition data and the target visual recognition objects corresponding to the target visual recognition data are determined at different times. By analyzing the target visual recognition data corresponding to the target visual recognition data, the target visual recognition objects can be obtained. The sequence of visual recognition, finally using the Petri net discrete modeling method to establish a visual search strategy model, the visual search strategy model is used to identify the driving environment based on the method provided by the above embodiment, wherein the input of the visual search strategy model is Tasks, the output is importance level and recognition order. The embodiment of the present invention also provides an implementation mode of establishing a visual search strategy model using the Petri net discrete modeling method. Specifically, the driving environment for a single visual recognition target and the multi-visual recognition target driving environment under different driving tasks, using After the processed pupil diameter data Q, blink reflex data E, and the corresponding relationship between time and target visual recognition data position, a visual search strategy model with driving tasks as input and importance level and visual recognition sequence as output is constructed.

In order to facilitate the understanding of the above-mentioned visual search strategy model, an embodiment of the present invention provides a structural framework diagram of a visual search strategy model, as shown in FIG. 4 , let X={D0, B0, (X0, Y0), D1, B1, (X1, Y1), Q, E, (X2, Y2), I, R, (Xt, Yt), C, S, (Xi, Yi)}, where X refers to all the data used, Including original eye movement information, scene information and processed data, etc. D0 is the pupil diameter data before preprocessing, B0 is the blink reflex data before preprocessing, (X0, Y0) is the sight point position data before preprocessing , D1 is the preprocessed pupil diameter data, B1 is the preprocessed blink reflection data, (X1, Y1) is the preprocessed gaze point position data, Q is the first data set, E is the second data set , (X2, Y2) is the set of target visual recognition data, I is the set of sampling point data numbers of the target pupil diameter corresponding to the peak point in the first data set Q, and R is the amplitude point in the second data set P (Xt, Yt) is the location set of target visual recognition data in the time domain, C is the importance level, and D is the visual recognition sequence.

The process change set Σ={T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13} represented by Fig. 4, wherein, T1 is the data collection process (including collecting eye movement information and collected scene information), T2 is the process of preprocessing the pupil diameter data (including elimination, compensation and noise reduction), T3 is the process of preprocessing the blink reflection data, and T4 is the process of preprocessing the gaze point data , T5 is the multi-scale transformation process of pupil diameter data, T6 is the harmonic analysis process of blink reflection data, T7 is the data extraction process of Matlab, T8 is the peak point extraction process and the search process of the peak point interval, T9 is the extraction process of the amplitude point and the search process of the interval where the amplitude point is located, T10 is the position combination process in the time domain, T11 is the process of calculating the weight of the pupil diameter index, T12 is the process of calculating the weight of the blink reflex index, and T13 It is the process of establishing the corresponding matrix of the visual recognition position of the target in the time domain.

In summary, the driving environment recognition method provided by the embodiment of the present invention is based on a complex driving environment oriented to various driving tasks and multiple visual recognition targets, and the psychological state reflected by the driver's eye movement information is used as the target visual recognition method. The basis for the classification of the importance level is in line with the traffic safety principle based on traffic psychology, and the order of visual recognition is determined by processing the position of sight points at different times, and the visual search strategy model based on the complex environment is constructed by using the perti net system modeling method to solve the problem It overcomes the shortcomings of low perception accuracy and efficiency in other intelligent vehicle environment perception methods, improves the pertinence of the vehicle's multi-view target search in complex environments, reduces the number of targets that need to be continuously tracked, and shortens the time required for perception in complex environments. time, making smart cars safer and more reliable.

For the driving environment recognition method provided in the above embodiments, the embodiment of the present invention also provides a driving environment recognition device, refer to the schematic structural diagram of a driving environment recognition device shown in Figure 5, the device mainly includes the following parts :

The data collection module 502 is used to collect the original eye movement information of the target object and the scene information of the virtual driving scene, and preprocess the original eye movement information to obtain the target eye movement information.

A level determination module 504, configured to determine the importance level of the object to be recognized in the virtual driving scene based on the eye movement information of the target.

The sequence determination module 506 is configured to determine the sequence of visual recognition of the objects to be visually recognized in the virtual driving scene based on the target eye movement information and the scene information.

The environment identification module 508 is configured to identify the driving environment of the virtual driving scene based on the importance level and the order of visual recognition.

The driving environment recognition device provided by the embodiment of the present invention proposes a new method for perceiving the environment. The embodiment of the present invention determines the importance level of the object to be recognized based on the eye movement information, and determines the importance level of the object to be recognized based on the eye movement information and scene information. The visual recognition sequence of the visual recognition objects, so as to recognize the driving environment based on the visual recognition sequence and important classification, can effectively improve the recognition accuracy and recognition efficiency of the driving environment.

In one embodiment, the eye movement information of the target includes pupil diameter data and blink reflex data; the level determination module 504 is also used for: performing multi-scale geometric analysis on the pupil diameter data, so as to obtain the visually recognized objects from the virtual driving scene Select an object of interest to obtain a set of objects of interest; conduct harmonic analysis on the blink reflection data to select a focus object from the objects to be recognized in the virtual driving scene to obtain a set of focus objects; according to the preset interest weight, preset The focus weight, the set of interested objects and the set of focus objects determine the importance level of the objects to be recognized in the virtual driving scene.

In one embodiment, the level determination module 504 is also used to: perform wavelet transformation on the pupil diameter data to obtain the first data set; determine the first interval containing the peak point in the first data set, and identify the first interval corresponding to The target pupil diameter range; select the object of interest from the objects to be recognized in the virtual driving scene based on the degree of interest corresponding to the target pupil diameter range.

In one embodiment, the level determination module 504 is also used to: perform Fourier transform on the blink reflection data to obtain the second data set; determine the second interval containing the amplitude points in the second data set, and identify The target blink reflex range corresponding to the second interval; selecting the focus object from the objects to be recognized in the virtual driving scene based on the degree of concentration corresponding to the target blink reflex range.

其中，C_i表示第i个瞳孔直径数据，S_1j表示第j个重要等级的瞳孔直径标准值，n表示瞳孔直径数据的总个数；预设专注权重w₂的设定方式为：

其中，D_p表示第p个瞬目反射数据，S_2q表示第q个重要等级的瞬目反射标准值，m表示瞬目反射数据的总个数。In an implementation manner, the setting method of the preset interest weight w ₁ is:

Among them, C _i represents the i-th pupil diameter data, S _1j represents the pupil diameter standard value of the j-th important level, n represents the total number of pupil diameter data; the setting method of the preset focus weight w ₂ is:

Among them, _Dp represents the pth blink reflection data, S _2q represents the blink reflection standard value of the qth important level, and m represents the total number of blink reflection data.

In one embodiment, the sequence determination module 506 is also used to: obtain the sight point data based on the target eye movement information and scene information; determine the sight point data satisfying the preset condition as the target visual recognition data; wherein, the preset condition The gaze time condition and the gaze angle condition are included; according to the correlation between the visual recognition time and the target visual recognition data, the visual recognition sequence of the target visual recognition data corresponding to the objects to be recognized is determined.

In one embodiment, the above device further includes a preprocessing module, configured to: use Lagrangian interpolation algorithm and empirical mode decomposition method to preprocess the original eye movement information to obtain target eye movement information.

The implementation principles and technical effects of the device provided by the embodiment of the present invention are the same as those of the foregoing method embodiment. For brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment.

An embodiment of the present invention provides a simulated driver, specifically, the simulated driver includes a simulator screen, a processor and a storage device; the simulator screen is used to display a virtual driving scene, and a computer program is stored on the storage device, the computer program The method of any one of the above embodiments is performed when executed by the processor.

FIG. 6 is a schematic structural diagram of another simulated driver provided by an embodiment of the present invention. The simulated driver 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, and the processor 60, the communication interface 63 and The memory 61 is connected through a bus 62; the processor 60 is used to execute executable modules stored in the memory 61, such as computer programs.

The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the methods described in the foregoing method embodiments, specifically implemented Reference may be made to the foregoing method embodiments, and details are not repeated here.

Finally, it should be noted that: the above-described embodiments are only specific implementations of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them, and the scope of protection of the present invention is not limited thereto, although referring to the foregoing The embodiment has described the present invention in detail, and those of ordinary skill in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be covered by the scope of the present invention. within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. A method for recognizing a driving environment, comprising:

acquiring original eye movement information of a target object and scene information of a virtual driving scene, and preprocessing the original eye movement information to obtain target eye movement information;

determining an importance level of an object to be visually recognized in the virtual driving scene based on the target eye movement information;

determining a visual recognition sequence of objects to be visually recognized in the virtual driving scene based on the target eye movement information and the scene information;

identifying a driving environment of the virtual driving scene based on the importance level and the visual recognition sequence;

the target eye movement information comprises pupil diameter data and transient ocular reflex data;

the step of determining the importance level of the object to be visually recognized in the virtual driving scene based on the target eye movement information includes:

performing multi-scale geometric analysis on the pupil diameter data to select an object of interest from objects to be viewed and recognized in the virtual driving scene to obtain an object of interest set;

performing harmonic analysis on the transient reflection data to select a concentration object from objects to be viewed and recognized in the virtual driving scene to obtain a concentration object set;

determining the importance level of an object to be viewed in the virtual driving scene according to a preset interest weight, a preset concentration weight, the interest object set and the concentration object set;

the step of determining the visual recognition sequence of the object to be visually recognized in the virtual driving scene based on the target eye movement information and the scene information comprises the following steps:

obtaining sight point data based on the target eye movement information and the scene information;

determining sight point data meeting preset conditions as target visual recognition data; the preset conditions comprise a fixation time condition and a fixation angle condition;

and determining the visual recognition sequence of the target visual recognition data corresponding to the object to be visually recognized according to the incidence relation between the visual recognition time and the target visual recognition data.

2. The method according to claim 1, wherein the step of performing a multi-scale geometric analysis on the pupil diameter data to select an object of interest from objects to be viewed of the virtual driving scene comprises:

performing wavelet transformation on the pupil diameter data to obtain a first data set;

determining a first interval containing a peak point in the first data set, and identifying a target pupil diameter range corresponding to the first interval;

and selecting an interested object from objects to be viewed in the virtual driving scene based on the interested degree corresponding to the target pupil diameter range.

3. The method of claim 1, wherein the step of performing harmonic analysis on the transient reflection data to select a concentration object from the objects to be identified for viewing in the virtual driving scene comprises:

performing Fourier transform on the instantaneous reflection data to obtain a second data set;

determining a second interval containing amplitude points in the second data set, and identifying a target instantaneous reflection range corresponding to the second interval;

and selecting a concentration object from the objects to be visually recognized in the virtual driving scene based on the concentration degree corresponding to the target instantaneous reflection range.

4. The method of claim 1, wherein the presetting is performed in response to a user inputWeight of interest w ₁ The setting method is as follows:

wherein, C _i Represents the ith pupil diameter data, S _1j The pupil diameter standard value of the jth importance level is represented, and n represents the total number of pupil diameter data;

the preset concentration weight w ₂ The setting method is as follows:

wherein D is _p Representing the p-th instantaneous reflection data, S _2q And (3) representing a snapshot reflection standard value of the q-th importance level, and m represents the total number of the snapshot reflection data.

5. The method of claim 1, wherein the step of preprocessing the original eye movement information to obtain target eye movement information comprises:

and preprocessing the original eye movement information by utilizing a Lagrange interpolation algorithm and an empirical mode decomposition method to obtain target eye movement information.

6. An apparatus for recognizing a driving environment, comprising:

the data acquisition module is used for acquiring original eye movement information of a target object and scene information of a virtual driving scene, and preprocessing the original eye movement information to obtain target eye movement information;

the grade determining module is used for determining the importance grade of an object to be viewed in the virtual driving scene based on the target eye movement information;

the sequence determining module is used for determining the visual recognition sequence of the objects to be viewed in the virtual driving scene based on the target eye movement information and the scene information;

the environment identification module is used for identifying the driving environment of the virtual driving scene based on the importance level and the visual recognition sequence;

the target eye movement information comprises pupil diameter data and instantaneous reflection data;

the rank determination module is further to:

performing multi-scale geometric analysis on the pupil diameter data to select an interested object from objects to be viewed in the virtual driving scene to obtain an interested object set;

performing harmonic analysis on the transient reflection data to select a concentration object from objects to be viewed and recognized in the virtual driving scene to obtain a concentration object set;

determining the importance level of an object to be viewed in the virtual driving scene according to a preset interest weight, a preset concentration weight, the interest object set and the concentration object set;

the order determination module is further configured to:

obtaining sight point data based on the target eye movement information and the scene information;

determining sight point data meeting preset conditions as target visual recognition data; the preset conditions comprise a watching time condition and a watching angle condition;

and determining the visual recognition sequence of the target visual recognition data corresponding to the object to be visually recognized according to the incidence relation between the visual recognition time and the target visual recognition data.

7. A simulated driver comprising a simulator screen, a processor and a memory;

the simulator screen is for displaying the virtual driving scenario, the memory having stored thereon a computer program which, when executed by the processor, performs the method of any one of claims 1 to 5.

8. A computer storage medium storing computer software instructions for use in the method of any one of claims 1 to 5.

Images