JP2024045551A - Information Output Device - Google Patents

Abstract

[Problem] Detecting inattentive tendencies with a simple configuration. [Solution] In an information processing device (1), a visual saliency processing unit (3) acquires a visual saliency map in time series obtained by estimating the level of visual saliency in an image captured of the outside from a moving object, and a visual saliency peak detection unit (4) detects at least one peak position in the visual saliency map in time series. Then, an inattentive tendency determination unit (5) sets a gaze area G in the image, and if the peak position is out of the gaze area G for a predetermined period of time or more, an inattentive tendency warning unit (6) outputs information indicating that there is an inattentive tendency. [Selected Figure] Figure 1

Description

The present invention relates to an information output device that outputs predetermined information based on an image captured from a moving object.

Detection of driver inattentiveness is being carried out to reduce traffic accidents. For example, in Patent Document 1, visual feature points (for example, a vehicle ahead 44, a traffic light 45) that are objects to be visually recognized that exist in front of the vehicle are detected based on an image G captured by an in-vehicle camera 1, and visual features are detected. It is described that it is determined whether or not the person is in a distracted state based on the point and the gaze point of the person to be determined.

JP 2017-224067 A

In the case of the method described in Patent Document 1, the driver's line of sight, facial direction, posture, etc. are detected from the image of the driver, and where the driver is looking is judged by comparing it with the driving image. Therefore, in addition to the driving image, images from the driver's side are also required, which requires multiple cameras. Also, a huge amount of calculation processing is required to compare the driver's line of sight with the driving image.

An example of the problem to be solved by the present invention is to detect a tendency to look aside with a simple configuration.

In order to solve the above problem, the invention according to claim 1 provides visual saliency distribution information obtained by estimating the level of visual saliency in the image based on an image taken of the outside from a moving body. a peak position detection unit that detects at least one peak position in the visual saliency distribution information in chronological order; and a gaze range that sets a range that the driver of the mobile object should gaze at in the image. The present invention is characterized by comprising a setting section and an inattentiveness output section that outputs information indicating that there is a tendency to inattentiveness when the peak position is continuously out of the range to be watched.

FIG. 1 is a functional configuration diagram of an information output device according to a first embodiment of the present invention. FIG. 2 is a block diagram illustrating the configuration of a visual saliency processing section shown in FIG. 1. FIG. (a) is a diagram illustrating an image input to the determination device, and (b) is a diagram illustrating a visual saliency map estimated for (a). 2 is a flowchart illustrating a processing method of the visual saliency processing unit shown in FIG. 1 . FIG. 3 is a diagram illustrating in detail the configuration of a nonlinear mapping section. FIG. 2 is a diagram illustrating a configuration of an intermediate layer. 13A and 13B are diagrams illustrating an example of convolution processing performed by a filter. (a) is a diagram for explaining the processing of the first pooling section, (b) is a diagram for explaining the processing of the second pooling section, and (c) is a diagram for explaining the processing of the second pooling section. FIG. 2 is a diagram for explaining the process. FIG. 11 is an explanatory diagram of a method for setting a gaze area. FIG. 3 is an explanatory diagram of an inattentiveness detection area. FIG. 11 is an explanatory diagram of another inattentive looking detection area. 2 is a flowchart of an operation of the information output device shown in FIG. 1 . FIG. 2 is a configuration diagram of a system including an information processing device according to a second embodiment of the present invention. FIG. 14 is a functional configuration diagram of the information processing device shown in FIG. 13. 15 is a flowchart of the operation of the information processing device shown in FIG. 14 .

The information output device according to one embodiment of the present invention will be described below. In the information output device according to one embodiment of the present invention, the acquisition unit acquires visual saliency distribution information in a time series, which is obtained by estimating the level of visual saliency in an image based on an image of the outside taken from a moving body, and the peak position detection unit detects at least one peak position in the visual saliency distribution information in a time series. Then, the gaze range setting unit sets a range in the image that the driver of the moving body should gaze at, and the inattentive output unit outputs information indicating a tendency to inattentiveness if the peak position is out of the gaze range for a predetermined period of time or more. This visual saliency distribution information indicates the statistical likelihood of human gaze gathering. Therefore, the peak of the visual saliency distribution information indicates the position that is statistically most likely to attract human gaze. Therefore, by using the visual saliency distribution information, it is possible to detect a tendency to inattentiveness with a simple configuration without measuring the actual driver's gaze.

The gaze range setting unit may also set the gaze range based on the vanishing point of the image. In this way, it becomes possible to easily set the gaze range without detecting, for example, a vehicle ahead.

The inattentive output unit may also output information indicating a tendency to look away due to a fixed object if the peak position is located above or below the range to be watched for a predetermined period of time or more. The area above the range to be watched is generally an area in which fixed objects such as buildings, traffic signals, signs, and street lights are reflected, while the area below the range to be watched is generally an area in which road markings such as road signs are reflected. In other words, if the peak position is included in these ranges, it is possible to identify that the object that is the cause of inattentive looking is a fixed object.

The vehicle may also include a speed acquisition unit that acquires the moving speed of the moving body, and the gaze range setting unit may change the gaze range based on the moving speed. For example, it is known that the driver's field of vision narrows when driving at high speed, and the gaze range can be set taking this into consideration. The speed acquisition unit may acquire information about speed from a vehicle speed sensor, an acceleration sensor, etc., or may determine the speed from a captured image.

Furthermore, the apparatus may include a position acquisition section that acquires position information of the moving body, and the gaze range setting section may change the range to be gazed on based on the position information. By doing this, it is possible to set the range to be watched depending on the location where the vehicle is traveling, such as a residential area, a main road, or a downtown area.

The acquisition unit also includes an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and generates saliency estimation information that indicates a saliency distribution based on the mapped data. The nonlinear mapping section may include a feature extraction section that extracts features from the intermediate data, and an upsampling section that upsamples the data generated by the feature extraction section. By doing so, visual saliency can be estimated with low calculation cost. Furthermore, the visual saliency estimated in this way reflects the contextual attention state.

Further, in the information output method according to an embodiment of the present invention, in the acquisition step, the visual saliency obtained by estimating the level of visual saliency in the image based on the image captured from the outside from the moving object Distribution information is acquired in time series, and at least one peak position in the visual saliency distribution information is detected in time series in the peak position detection step. Then, in the gaze range setting step, the range that the driver of the moving object should gaze on in the image is set, and in the inattentiveness output step, if the peak position is out of the gaze range for a predetermined period of time or more, the inattentiveness is detected. Outputs information indicating that there is a trend. This visual saliency distribution information shows the statistical ease with which people's gaze is attracted. Therefore, the peak of the visual saliency distribution information indicates the position where human gaze is statistically most likely to be attracted. Therefore, by using visual saliency distribution information, it is possible to detect a tendency to look aside with a simple configuration without actually measuring the driver's line of sight.

Further, the information output method described above is executed by a computer. By doing so, it is possible to detect a tendency to look aside with a simple configuration without using a computer to measure the driver's actual line of sight.

Furthermore, the above-described information output program may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed.

An information output device according to a first embodiment of the present invention will be described with reference to Figs. 1 to 12. The information output device according to this embodiment is not limited to being installed in a moving object such as an automobile, but may also be configured as a server device or the like installed in a business establishment or the like. In other words, analysis does not need to be performed in real time, and analysis may be performed after driving, etc.

As shown in FIG. 1, the information output device 1 includes a driving image acquisition unit 2, a visual saliency processing unit 3, a visual saliency peak detection unit 4, an inattentive tendency determination unit 5, and an inattentive warning unit 6.

The driving image acquisition unit 2 receives input of images (e.g., video images) captured by, for example, a camera, and outputs the images as image data. The input video images are output as image data broken down into a time series, for example, for each frame. Although still images may be input as images to be input to the driving image acquisition unit 2, it is preferable to input them as an image group consisting of multiple still images in a time series.

The images input to the driving image acquisition unit 2 include, for example, images of the vehicle's travel direction. In other words, images of the outside world continuously captured from a moving body. These images may be so-called panoramic images or images captured using multiple cameras, and may include angles other than the travel direction, such as 180° or 360° in the horizontal direction. Furthermore, images input to the driving image acquisition unit 2 are not limited to images captured by a camera, and may also be images read from a recording medium such as a hard disk drive or memory card.

The visual saliency processing unit 3 receives image data from the driving image acquisition unit 2 and outputs a visual saliency map as visual saliency estimation information described below. In other words, the visual saliency processing unit 3 functions as an acquisition unit that acquires a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency in an image captured of the outside from a moving body.

FIG. 2 is a block diagram illustrating the configuration of the visual saliency processing section 3. As shown in FIG. The visual saliency processing unit 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts the image into intermediate data that can be mapped. The nonlinear mapping unit 320 converts intermediate data into mapping data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping section 320 includes a feature extraction section 321 that extracts features from intermediate data, and an upsampling section 322 that upsamples the data generated by the feature extraction section 321. The storage unit 390 stores image data input from the traveling image acquisition unit 2, coefficients of a filter described later, and the like. This will be explained in detail below.

FIG. 3(a) is a diagram illustrating an image input to the visual saliency processing unit 3, and FIG. 3(b) is a diagram illustrating an image showing the visual saliency distribution estimated for FIG. 3(a). It is a figure which illustrates. The visual saliency processing unit 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual conspicuousness means, for example, how easy it is to stand out or how easy it is to attract attention. Specifically, visual saliency is indicated by probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person viewing the image will turn to the position.

Figure 3(a) and Figure 3(b) correspond to each other in terms of position. In Figure 3(a), the higher the visual saliency is at a position, the higher the luminance is displayed in Figure 3(b). The image showing the visual saliency distribution as in Figure 3(b) is an example of a visual saliency map output by the output unit 330. In this example, visual saliency is visualized with 256 gradations of luminance values. An example of a visual saliency map output by the output unit 330 will be described in detail later.

FIG. 4 is a flowchart illustrating the operation of the visual saliency processing unit 3 according to this embodiment. The flowchart shown in FIG. 4 is part of an information output method executed by a computer, and includes an input step S115, a nonlinear mapping step S120, and an output step S130. In input step S115, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information (visual saliency distribution information) indicating the saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121.

Returning to FIG. 2, each component of the visual saliency processing unit 3 will be explained. In input step S115, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the traveling image acquisition unit 2 . The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, and is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data obtained by normalizing the brightness of the obtained image, or data obtained by converting each pixel of the obtained image into a slope of brightness. In input step S115, the input unit 310 may further perform image noise removal, resolution conversion, etc.

In the nonlinear mapping step S120, the nonlinear mapping section 320 obtains intermediate data from the input section 310. Then, the intermediate data is converted into mapping data in the nonlinear mapping section 320. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, etc., and is preferably a process using a function, a functional, or a neural network.

FIG. 5 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping section 320 includes the feature extraction section 321 and the up-sampling section 322. The feature extraction section 321 performs a feature extraction step S121, and the upsampling section 322 performs an upsampling step S122. Further, in the example shown in the figure, at least one of the feature extraction section 321 and the up-sampling section 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, multiple intermediate layers 323 are coupled.

In particular, it is preferable that the neural network is a convolutional neural network. Specifically, each of the multiple intermediate layers 323 includes one or more convolutional layers 324. In the convolutional layer 324, the input data is convolved by multiple filters 325, and activation processing is performed on the output of the multiple filters 325.

In the example of FIG. 5, the feature extraction unit 321 includes a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the up-sampling unit 322 includes a neural network including a plurality of intermediate layers 323, and includes an unpooling unit 328 between the plurality of intermediate layers 323. Further, the feature extraction section 321 and the up-sampling section 322 are connected to each other via a second pooling section 327 that performs overlap pooling.

Note that in the example shown in this figure, each intermediate layer 323 consists of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may consist of only one convolutional layer 324. The intermediate layers 323 that are adjacent to each other are separated by one of a first pooling section 326, a second pooling section 327, and an unpooling section 328. Here, when the intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the numbers of filters 325 in those convolutional layers 324 are equal to each other.

In this diagram, the intermediate layer 323 marked "A×B" is composed of B convolution layers 324, meaning that each convolution layer 324 includes A convolution filters for each channel. Such an intermediate layer 323 is also referred to below as an "A×B intermediate layer." For example, a 64×2 intermediate layer 323 is composed of two convolution layers 324, meaning that each convolution layer 324 includes 64 convolution filters for each channel.

In the example shown in the figure, the feature extraction unit 321 includes a 64×2 intermediate layer 323, a 128×2 intermediate layer 323, a 256×3 intermediate layer 323, and a 512×3 intermediate layer 323 in this order. Further, the up-sample section 322 includes a 512×3 intermediate layer 323, a 256×3 intermediate layer 323, a 128×2 intermediate layer 323, and a 64×2 intermediate layer 323 in this order. Further, the second pooling section 327 connects the two 512×3 intermediate layers 323 to each other. Note that the number of intermediate layers 323 constituting the nonlinear mapping section 320 is not particularly limited, and can be determined depending on, for example, the number of pixels of image data.

Note that this diagram is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have other configurations. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of convolutional layers 324 included in the intermediate layer 323 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of channels in the intermediate layer 323 may further reduce the computational cost. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the multiple intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed through. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling unit 326, and the second intermediate layer 323b is located after the first intermediate layer 323a. The first intermediate layer 323a is composed of a convolution layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolution layer 324 in which the number of filters 325 for each channel is N2. At this time, it is preferable that N2>N1 holds. It is more preferable that N2=N1×2 holds.

In addition, in the multiple intermediate layers 323 included in the upsampling unit 322, it is preferable that the number of filters 325 decreases each time the unpooling unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling unit 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. The third intermediate layer 323c is composed of a convolution layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolution layer 324 in which the number of filters 325 for each channel is N4. At this time, it is preferable that N4<N3 holds. It is more preferable that N3=N4×2 holds.

The feature extraction unit 321 extracts image features with multiple abstraction levels, such as gradients and shapes, from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 6 illustrates an example of the configuration of the 64×2 intermediate layer 323. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 has 64 filters 325. In the first convolution layer 324a, convolution processing using the filters 325 is performed on each channel of the data input to the intermediate layer 323. For example, if the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h0i (i=1..3). Also, in the example of this figure, the filters 325 are 64 types of 3×3 filters, that is, a total of 64×3 types of filters. As a result of the convolution process, 64 results h0i,j (i = 1..3, j = 1..64) are obtained for each channel i.

Next, the activation unit 329 performs activation processing on the outputs of the multiple filters 325. Specifically, activation processing is performed on the sum of the corresponding elements for the corresponding results j of all channels. This activation processing results in 64 channels h1i (i=1..64), i.e., the output of the first convolution layer 324a, as image features. The activation processing is not particularly limited, but processing using at least one of a hyperbolic function, a sigmoid function, and a normalized linear function is preferable.

Furthermore, the output data of the first convolutional layer 324a is used as input data for the second convolutional layer 324b, which performs the same processing as the first convolutional layer 324a, and the 64-channel result h2i (i = 1..64), that is, the output of the second convolutional layer 324b, is obtained as the image feature. The output of the second convolutional layer 324b becomes the output data of this 64 x 2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but it is preferable that the filter 325 is a two-dimensional filter of 3×3. The coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are stored in the memory unit 390, and the nonlinear mapping unit 320 can read them and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapping data. The memory unit 390 may be provided in the visual saliency processing unit 3 or may be provided outside the visual saliency processing unit 3. In addition, the nonlinear mapping unit 320 may obtain the correction information from the outside via a communication network.

FIGS. 7A and 7B are diagrams showing examples of convolution processing performed by the filter 325, respectively. 7(a) and 7(b) both show examples of 3×3 convolution. The example in FIG. 7(a) is a convolution process using the nearest elements. The example in FIG. 7(b) is a convolution process using adjacent elements having a distance of two or more. Note that convolution processing using adjacent elements having a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using adjacent elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The above describes the operation of the 64×2 intermediate layer 323. The operation of the other intermediate layers 323 (such as the 128×2 intermediate layer 323, the 256×3 intermediate layer 323, and the 512×3 intermediate layer 323) is the same as that of the 64×2 intermediate layer 323, except for the number of convolutional layers 324 and the number of channels. In addition, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsampling unit 322 are also the same as above.

Figure 8(a) is a diagram for explaining the processing of the first pooling unit 326, Figure 8(b) is a diagram for explaining the processing of the second pooling unit 327, and Figure 8(c) is a diagram for explaining the processing of the unpooling unit 328.

In the feature extraction unit 321, the data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. In the first pooling unit 326, for example, non-overlapping pooling processing is performed. FIG. 8(a) shows a process of associating four 2×2 elements 30 with one element 30 for the element group included in each channel. In the first pooling unit 326, such association is performed for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is performed on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of matching four 2×2 elements 30 to one element 30 while overlapping some of the elements 30. That is, in repeated matching, some of the four 2×2 elements 30 in a certain matching are also included in the four 2×2 elements 30 in the next matching. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. Note that the number of elements 30 before and after matching in the second pooling unit 327 is not particularly limited.

The method of each process performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, the maximum value of four elements 30 is associated with one element 30 (max pooling), An example of this is average pooling, in which the average value of two elements 30 is used as one element 30.

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. The output data from the intermediate layer 323 of the upsampling unit 322 is subjected to unpooling processing for each channel in the unpooling unit 328, and then input to the next intermediate layer 323. Figure 8(c) shows the process of expanding one element 30 to multiple elements 30. The method of expansion is not particularly limited, but one example is a method of duplicating one element 30 to four elements 30 (2 x 2).

The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which visual saliency is visualized by brightness values, as shown in FIG. 3B. The visual saliency map may also be, for example, an image that is colored according to visual saliency, such as a heat map, or an image in which visual saliency areas with visual saliency higher than a predetermined standard are marked in a manner that makes them distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information shown as an image or the like, and may be a table or the like that lists information indicating visual saliency areas.

The visual saliency peak detection unit 4 detects peak positions (pixels) in the visual saliency map acquired by the visual saliency processing unit 3. Here, in this embodiment, a peak is a pixel with high visual saliency where the pixel value is the maximum value (luminance is maximum), and the position is represented by coordinates. That is, the visual saliency peak detection unit 4 functions as a peak position detection unit that detects at least one peak position in the visual saliency map (visual saliency distribution information).

The inattentive tendency determination unit 5 determines whether the image input from the driving image acquisition unit 2 has an inattentive tendency based on the peak position detected by the visual saliency peak detection unit 4. The inattentive tendency determination unit 5 first sets a gaze area (range to be gazed at) for the image input from the driving image acquisition unit 2. The method of setting the gaze area will be described with reference to FIG. 9. That is, the inattentive tendency determination unit 5 functions as a gaze range setting unit that sets the range in the image that the driver of the moving object should gaze at.

In the image P shown in FIG. 9, the gaze area G is set around the vanishing point V. That is, the gaze area G (range to be watched) is set based on the vanishing point of the image. This gaze area G is determined by setting the size of the gaze area G (for example, 3 m in width and 2 m in height) in advance, and determining the number of horizontal pixels, the number of vertical pixels, the horizontal angle of view, the vertical angle of view, and the distance to the preceding vehicle. It is possible to calculate the number of pixels of the set size from the distance, the mounting height of the camera such as a drive recorder that is capturing the image, etc. Note that the vanishing point may be estimated from a white line or the like, or may be estimated using optical flow or the like. Furthermore, the inter-vehicle distance to the preceding vehicle may be set virtually without the need to actually detect the preceding vehicle.

Next, an inattentive detection area in image P is set based on the set gaze area G (shaded area in FIG. 10). The inattentive detection area is set to an upper area Iu, a lower area Id, a left side area Il, and a right side area Ir. These areas are divided by lines connecting the vanishing point V and each vertex of gaze area G. That is, the upper area Iu and the left side area Il are separated by a line segment L1 connecting the vanishing point V and the vertex Ga of gaze area G. The upper area Iu and the right side area Ir are separated by a line segment L2 connecting the vanishing point V and the vertex Gd of gaze area G. The lower area Id and the left side area Il are separated by a line segment L3 connecting the vanishing point V and the vertex Gb of gaze area G. The lower area Id and the right side area Ir are separated by a line segment L4 connecting the vanishing point V and the vertex Gc of gaze area G.

Note that the inattentiveness detection area is not limited to the division as shown in FIG. For example, the configuration shown in FIG. 11 may be used. In FIG. 11, the inattentiveness detection area is divided by line segments obtained by extending each side of the gaze area G. Since the method of FIG. 11 has a simple shape, it is possible to reduce the processing involved in dividing the inattentiveness detection areas.

Next, the determination of the inattentive tendency by the inattentive tendency determination unit 5 will be described. If the peak position detected by the visual saliency peak detection unit 4 is continuously outside the gaze area G for a predetermined period of time or more, it is determined that there is an inattentive tendency. Here, the predetermined period of time can be, for example, two seconds, but may be changed as appropriate. In other words, the inattentive tendency determination unit 5 determines whether the peak position has been continuously outside the range to be gazed at for a predetermined period of time or more.

The inattentive tendency determination unit 5 may also determine that there is a tendency to look inattentively due to fixed objects when the inattentive detection area is the upper area Iu or the lower area Id. This is because, in the case of an image captured of the front from the vehicle, fixed objects such as buildings, traffic signals, signs, and street lights are generally reflected in the upper area Iu, and road paint such as road signs is generally reflected in the lower area Id. On the other hand, moving objects other than the vehicle itself, such as vehicles traveling in other driving lanes, may be reflected in the left side area Il and right side area Ir, and it is difficult to determine the object of inattentiveness (whether it is a fixed object or a moving object) depending on the area.

The inattentiveness warning section 6 issues a warning or the like based on the determination result of the inattentiveness tendency determination section 5. The warning may be notified by displaying it on a display device that is visible to the driver, or by outputting it as a sound or vibration. That is, the inattentiveness warning section 6 functions as an inattentiveness output section that outputs information indicating that there is a tendency to look aside. In this embodiment, the warning is information that there is a tendency to look aside, but the information that there is a tendency to look aside includes information such as the time and position at the time in addition to information on detection of looking aside such as flags. The data may also be output to a storage medium or outside the information output device 1 via communication or the like. Further, based on the determination result of the inattentive tendency determination unit 5, the result may be output as information related to a near-miss.

Next, the operation (information output method) of the information output device 1 configured as described above will be explained with reference to the flowchart of FIG. 12. Further, by configuring this flowchart as a program executed by a computer functioning as the information output device 1, it can be made into an information processing program. Further, this information output program is not limited to being stored in the memory of the information output device 1, but may be stored in a storage medium such as a memory card or an optical disc.

First, the inattentive warning unit 6 determines whether the inattentive warning switch (SW) is ON or OFF (step S101). The inattentive warning SW is a switch that switches whether or not the inattentive warning unit 6 issues a warning, and is provided in the inattentive warning unit 6 and is switched under the control of the inattentive tendency determination unit 5.

When the inattentive warning SW is ON (step S101; SW = ON), the inattentive warning unit 6 performs a comparison with the warning timer threshold (step S102). The warning timer is a timer that measures the period during which the inattentive warning unit 6 issues a warning, and the warning timer threshold is a threshold that determines the period during which the warning is executed. In other words, the inattentive warning unit 6 issues a warning only for the period determined by the warning timer threshold.

If the warning timer threshold is exceeded (step S102; threshold exceeded), the inattentiveness warning section 6 turns off the inattentiveness warning SW to stop the warning timer (step S103), and step S104, which will be described later, is executed. If the warning timer threshold has not been exceeded (step S102; threshold not exceeded), step S104, which will be described later, is executed without doing anything.

On the other hand, if the inattentive warning switch is OFF or if the process proceeds from steps S102 or S103 described above, the driving image acquisition unit 2 acquires a driving image (step S104), and the visual saliency processing unit 3 performs visual saliency image processing (acquisition of a visual saliency map) (step S105). Then, the visual saliency peak detection unit 4 acquires (detects) a peak position based on the visual saliency map acquired by the visual saliency processing unit 3 in step S105 (step S106).

Next, the inattentive tendency determination unit 5 sets a gaze area G and compares the gaze area G with the peak position acquired by the visual saliency peak detection unit 4 (step S107). If the comparison result shows that the peak position is outside the gaze area G (step S107; outside the gaze area), the inattentive tendency determination unit 5 determines whether the dwell timer has started or is stopped (step S108). The dwell timer is a timer that measures the time that the peak position dwells outside the gaze area G. Note that the gaze area G may be set when the image is acquired in step S104.

If the dwell timer is stopped (step S108; stopped), the inattentive tendency judgment unit 5 starts the dwell timer (step S109). On the other hand, if the dwell timer has already started (step S108; started), the inattentive tendency judgment unit 5 performs a comparison with the dwell timer threshold (step S110). The dwell timer threshold is a threshold for the time that the peak position remains outside the gaze area G, and is set to, for example, 2 seconds as described above.

If the retention timer exceeds the threshold (step S110; exceeds the threshold), the inattentive tendency determination section 5 turns on the inattentiveness warning SW of the inattentiveness warning section 6 to start the warning timer (step S111). The inattentive tendency determining unit 5 then stops the retention timer (step S112). In other words, since the time during which the peak position remained outside the gaze area G was equal to or greater than the threshold value, the inattentiveness warning section 6 starts issuing a warning.

On the other hand, if the dwell timer does not exceed the threshold (step S110; threshold not exceeded), the inattentive tendency determination unit 5 does nothing and returns to step S101.

Also, if the comparison in step S107 shows that the peak position is within the gaze area G (step S107; within gaze area), the inattentive driving tendency determination unit 5 stops the dwell timer (step S112).

As is clear from the above explanation, step S105 functions as an acquisition process, step S106 functions as a peak position detection process, step S107 functions as a gaze range setting process, and steps S107 to S111 functions as an aside-gaze output process.

According to the present embodiment, the information output device 1 uses visual saliency processing unit 3 to estimate the level of visual saliency in the image based on an image taken of the outside from a moving body. The saliency map is acquired in time series, and the visual saliency peak detection unit 4 detects at least one peak position in the visual saliency map in time series. Then, the inattentiveness tendency determination unit 5 sets the gaze area G in the image, and if the peak position is continuously outside the gaze area G for a predetermined period of time or more, the inattentiveness warning unit 6 determines that there is a tendency to inattentiveness. Output information. This visual saliency map shows the statistical ease of attracting human gaze. Therefore, the peak of the visual saliency map indicates the position where human gaze is statistically most likely to be attracted. Therefore, by using a visual saliency map, it is possible to detect a tendency to look aside with a simple configuration without actually measuring the driver's line of sight.

The inattentive tendency determination unit 5 also sets the gaze area G based on the vanishing point V of the image. In this way, it becomes possible to easily set the gaze area G without detecting, for example, a vehicle ahead.

In addition, if the inattentive tendency determination unit 5 determines that the peak position is located above or below the gaze area G for a predetermined period of time or more, the inattentive tendency warning unit 6 may output information indicating that there is a tendency to look inattentively due to a fixed object. The area above the gaze area G is generally an area in which fixed objects such as buildings, traffic signals, signs, and street lights are reflected, and the area below the gaze area G is generally an area in which road paint such as road signs is reflected. In other words, if the range includes the peak position, it is possible to identify that the object of inattentive looking due to inattentive looking is a fixed object.

The visual saliency processing unit 3 also includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. In this way, visual saliency can be estimated with low computational cost. Furthermore, the visual saliency estimated in this way reflects the contextual attention state.

Note that the gaze area G is not limited to being set to a fixed range. For example, it may be changed according to the moving speed of the moving body. For example, it is known that the driver's field of vision narrows when driving at high speed. Therefore, for example, the inattentive tendency determination unit 5 may obtain the vehicle speed from a speed sensor or the like mounted on the vehicle, and narrow the range of the gaze area G as the speed increases. In addition, since the appropriate inter-vehicle distance also changes according to the moving speed, the range of the gaze area G calculated by the calculation method described with reference to FIG. 9 may also be changed. The speed of the vehicle is not limited to being obtained by a speed sensor, and may be obtained from an acceleration sensor or captured images.

Furthermore, the viewing area G may be changed depending on the driving position and situation of the vehicle or the like. If the situation requires attention to the surroundings, the gaze area G needs to be widened. For example, the range to be watched changes depending on where the vehicle is traveling, such as in a residential area, on a main road, or in a downtown area. If it is a residential area, there are few pedestrians, but it is necessary to be careful about sudden jumps, so the gaze area G cannot be narrowed. On the other hand, if the vehicle is on a main road, the traveling speed will be high and the field of view will be narrow as described above.

To give specific examples, school routes, parks, and areas near schools are considered to be at risk of children running out into the street. There are considered to be many pedestrians near train stations, schools, event locations, and tourist sites. There are considered to be many bicycles near bicycle parking lots and schools. There are considered to be many drunk people near entertainment districts. Such locations are situations that require attention to the surroundings, so the gaze area G may be widened and the area determined to have a tendency to look away may be narrowed. On the other hand, when driving on highways or in areas with low traffic volume and population density, there is a tendency for driving speeds to be high, so the gaze area G may be narrowed and the area determined to have a tendency to look away may be widened.

The gaze area G may also be changed depending on the time of day, events, etc. For example, during commuting hours, when attention to the surroundings is required, the gaze area G may be made wider than during normal hours, narrowing the area determined to be prone to looking away. Similarly, the gaze area G may be made wider from twilight to nighttime, narrowing the area determined to be prone to looking away. On the other hand, in the middle of the night, the gaze area G may be narrowed, widening the area determined to be prone to looking away.

Furthermore, the gaze area G may be changed depending on event information. For example, special events and similar events are held in locations and at times with high pedestrian traffic, so the gaze area G may be made wider than usual to loosen the judgment of the tendency to look away.

Information on such points can be obtained by the inattentiveness tendency determination unit 5 acquiring information from a means that can determine the current position and the area in which the vehicle is traveling, such as a GPS receiver or map data, and associating it with image data. , the range of the gaze area G can be changed. The time information may be acquired by the information output device 1 from inside or outside. Event information may be obtained from external sites, etc. Further, the change may be determined by combining the location, time, and date, or any one of them may be used to determine the change.

Furthermore, when driving at high speeds, the dwell timer threshold may be shortened. This is because even a brief moment of inattentive driving can be dangerous when driving at high speeds.

Next, an information processing device according to a second embodiment of the present invention will be described with reference to Figures 13 to 15. Note that the same parts as those in the first embodiment described above are given the same reference numerals and the description will be omitted.

FIG. 13 shows a typical system configuration example of this embodiment. The system according to this embodiment includes an information processing device 1A and a server device 10. An information processing device 1A according to this embodiment is mounted on a vehicle V. The information processing device 1A and the server device 10 can communicate via a network N such as the Internet.

FIG. 14 shows an information processing apparatus 1A according to this embodiment. The information processing device 1A includes a running image acquisition section 2, a visual saliency processing section 3, a visual saliency peak detection section 4, an inattentive tendency determination section 5A, and an output section 7.

The running image acquisition section 2, visual saliency processing section 3, and visual saliency peak detection section 4 are the same as those in the first embodiment. In addition to determining the tendency of looking aside, the looking aside tendency determination unit 5A also determines whether the looking aside object (peak position) when it is determined that there is a tendency of looking aside is a constant object. That is, when the peak position is out of the gaze area G (range to be watched) for a predetermined period of time or more, the inattentive tendency determination unit 5A determines whether the object corresponding to the peak position is a constant object. function as a department.

A constantly present object refers to a fixed object such as a building, traffic signal, sign, street lamp, or road paint as described in the first embodiment, and is an object that can be constantly observed from the video shooting location (a building or other object that is always present in that position).

The determination of whether a constant object is present is made not only when the peak position is in the upper area Iu or the lower area Id, as described in the first embodiment, but also when the peak position is in the left side area Il or the right side area Ir.

If the peak position is in the upper area Iu or the lower area Id, it is possible to determine whether the object is a resident object based only on the area. On the other hand, if the peak position is in the left area Il or the right area Ir, it is not possible to determine whether the inattentive object is a permanent object based only on the area, so the determination is performed using object recognition. Object recognition (also referred to as object detection) may be performed using a well-known algorithm, and the specific method is not particularly limited.

In addition, relative velocity may be used in addition to object recognition to determine a resident object. This calculates the relative speed from the speed of the own vehicle and the moving speed of the object to be looked at in between frames, and then determines from the relative speed whether the object to be looked at is a permanent object. Here, the inter-frame moving speed of the inattentive object can be determined by calculating the inter-frame moving speed of the peak position. If the determined relative velocity is greater than or equal to a predetermined threshold, it can be determined that the object is fixed at a certain position (a permanent object).

As described above, a permanent object is fixed at a certain position and can be observed at all times from the video shooting location (always existing at that position, such as a building), so by making the determination according to this embodiment, A position where the object of inattentiveness (peak position) is determined to be a constant object can be considered to be a position where it is easy to look aside.

When the inattentive tendency determining unit 5 determines that the inattentive object is a constant object, the output unit 7 transmits the determination result to the server device 10 . Alternatively, the presence or absence of a resident object may be always transmitted as a determination result. At this time, if the determination is made in real time, the determination time may be added, and if the image stored in a memory card or the like is determined, the imaging time or date of the image may be added. By adding time information, it is possible to extract locations where it is easy to look aside depending on the time of day. For example, it is possible to determine locations where it is easy to look aside due to buildings that are only visible during the day, locations where it is easy to look aside due to the effects of lighting, etc., locations where it is easy to look aside due to events such as fireworks, etc. Additionally, information on the imaging point may be added.

The server device 10 tally up the judgment results transmitted from the information processing device 1A. For example, by tallying up the judgment results transmitted from the information processing device 1A (vehicle) including the location information, it is possible to extract locations where it is easy to look away from the vehicle. Furthermore, by including time information, it is possible to extract locations where it is easy to look away from the vehicle depending on the time of day.

The operation of the information processing apparatus 1A according to this embodiment will be explained with reference to the flowchart of FIG. 15. In FIG. 15, steps S104 to S110 and S112 are the same as in FIG. 12. Furthermore, steps S101 to S103 in FIG. 12 are omitted because warnings are not essential in this embodiment.

In FIG. 15, if the dwell timer exceeds the threshold (step S110; exceeded threshold), the inattentive tendency determination unit 5 determines whether or not there is a habitual object and transmits the determination result (step S111A). In step S111A, the determination of the habitual object is performed by the object recognition described above, and the determination result is transmitted to the server device 10.

Note that in the flowchart shown in FIG. 15, steps related to warnings are omitted, but warnings may also be issued. If a warning is to be issued, the inattentiveness warning unit 6 shown in FIG. 1 may be provided, steps S101 to S103 may also be executed, and step S111 in FIG. 12 may be executed before, after, or in parallel with step S111A.

In addition, in this embodiment, the information processing device 1A on the vehicle side judges whether a target object is always present, but the server device 10 may also judge whether a target object is always present. In other words, the server device 10 may acquire driving images, etc., judge whether a target object is always present on the images, and output the judgment result to a storage device in the server device 10 or another server device, etc.

According to the present embodiment, the information processing device 1A uses visual saliency processing unit 3 to estimate the level of visual saliency in the image based on an image taken of the outside from a moving body. The saliency map is acquired in time series, and the visual saliency peak detection unit 4 detects at least one peak position in the visual saliency map in time series. Then, the inattentive tendency determination unit 5 sets a gaze area G in the image, and if the peak position is continuously away from the gaze area G for a predetermined time or more, it is determined that the object corresponding to the peak position is a constant object. Determine whether By doing this, it is determined whether the inattentive object detected based on the visual saliency map is a permanent object that can be observed at all times from the video shooting location (always existing at that position, such as a building) or a moving object. It becomes possible to do so. Therefore, it is possible to specify whether the neglected object is at least a constant object.

The inattentive tendency determination unit 5 may also determine whether the peak position is a resident object by object recognition when the peak position is to the left or right of the gaze area G. In this way, when the peak position is to the left or right of the gaze area G, the peak position corresponds to either a resident object or a moving object, so by performing object recognition, it is possible to accurately determine whether the object is a resident object such as a building or a moving object such as a car.

In addition, the inattentiveness tendency determination unit 5 obtains the moving speed of the moving object, and when the peak position is off to the left or right side of the gaze area G, the inattentiveness tendency determination unit 5 acquires the moving speed of the moving object, and when the peak position is off to the left or right side of the gaze area G, the object indicated by the peak position is determined based on the moving speed. The relative velocity may be calculated and it may be determined whether the object is a permanent object based on the relative velocity. By doing so, it is possible to determine whether the object is present based on the relative velocity, and the processing load at the time of determination can be reduced.

The system also includes an output unit 7 that outputs the judgment results. In this way, it is possible to transmit the results of the judgment made in the vehicle to a server device 10 or the like for compilation.

Further, the present invention is not limited to the above embodiments. That is, those skilled in the art can implement various modifications based on conventionally known knowledge without departing from the gist of the present invention. Of course, such modifications fall within the scope of the present invention as long as they still include the information processing apparatus of the present invention.

1 Information output device 2 Driving image acquisition unit 3 Visual saliency processing unit (acquisition unit)
4. Visual saliency peak detection unit (peak position detection unit)
5. Inattentive tendency determination unit (gaze range setting unit, inattentive output unit, speed acquisition unit, position acquisition unit, determination unit)
6 Inattentive warning section 7 Output section

Claims (1)

an acquisition unit that acquires visual saliency distribution information in time series obtained by estimating the level of visual saliency in the image based on an image taken of the outside from the moving body;
a peak position detection unit that detects at least one peak position in the visual saliency distribution information;
a gaze range setting unit that sets a range in the image that the driver of the mobile object should gaze at;
an inattentiveness output unit that outputs information indicating that there is a tendency to inattentiveness when the peak position is continuously out of the range to be watched;
An information output device comprising:

Images