JP7502051B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device that performs a predetermined process based on an image captured from a moving object.

For example, it has been proposed to reduce the risk of traffic accidents by providing drivers with information about locations where there is a high risk of traffic accidents occurring (accident risk locations). In this case, the accident risk locations are set by taking into consideration both predictions that are dependent on physical attributes of the traffic environment, such as traffic volume, and natural phenomena, such as the time of day and weather, as well as locations where accidents have actually occurred.

Patent Document 1 describes a method for estimating the degree to which the driving environment is likely to cause eye fatigue from an image captured in the direction of travel of the vehicle, by acquiring an image captured in the direction of travel of the vehicle, estimating the position in the captured image at which the driver will naturally gaze, estimating the position in the captured image that the driver should look at when driving the vehicle, and estimating the visual load based on the positional relationship between the position at which the driver will gaze and the position that should be looked at.

Patent No. 5482737

It is also possible to calculate an index relating to the safety and risk of the scene depicted in the video based on the visual load extracted using the method described in Patent Document 1. However, the method described in Patent Document 1 cannot reflect the contextual attention state in which the gaze tends to be unconsciously focused on objects such as signs and pedestrians contained in the image, and therefore there may be a discrepancy with the actual visual state of the driver, leaving room for improvement in the calculation accuracy.

One example of the problem that the present invention aims to solve is the accurate calculation of indicators related to safety and risk.

In order to solve the above problem, the invention described in claim 1 comprises an acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image based on an image of the outside taken from a moving body, a gaze position setting unit that sets a reference gaze position in the image according to a predetermined rule, and a visual attention concentration level calculation unit that calculates a degree of visual attention concentration in the image based on the visual saliency distribution information and the reference gaze position, wherein the visual attention concentration level calculation unit calculates the degree of visual attention concentration in the image based on the visual saliency distribution information and the reference gaze position, based on the value of each pixel that constitutes the visual saliency distribution information and the vector error between the position of each pixel and the coordinate position of the reference gaze position, and calculates the degree of visual attention concentration by the inverse of the sum of the weighted relationship between the vector error of the coordinates of all pixels from the coordinates of the reference gaze position set on the visual saliency distribution information and the value of the pixel.

The invention described in claim 4 is an information processing method executed by an information processing device that performs predetermined information processing based on an image captured of the outside from a moving body, and includes an acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in the image based on the image, a gaze position setting step of setting a reference gaze position in the image in accordance with a predetermined rule, and a visual attention concentration level calculation step of calculating a degree of visual attention concentration in the image based on the visual saliency distribution information and the reference gaze position, wherein the visual attention concentration level calculation step calculates the degree of visual attention concentration by the reciprocal of a weighted sum of the vector errors of the coordinates of all pixels from the coordinates of the reference gaze position set on the visual saliency distribution information and the values of the pixels based on a value of each pixel constituting the visual saliency distribution information and a vector error between the position of each pixel and a coordinate position of the reference gaze position .

The invention as set forth in claim 5 is characterized in that the information processing method as set forth in claim 4 is executed by a computer.

The sixth aspect of the present invention is characterized in that the information processing program according to the fifth aspect is stored.

1 is a functional configuration diagram of an information processing device according to a first embodiment of the present invention; FIG. 2 is a block diagram illustrating a configuration of a visual saliency extraction unit shown in FIG. 1 . FIG. 2A is a diagram illustrating an example of an image input to a determination device, and FIG. 2B is a diagram illustrating an example of a visual saliency map estimated for FIG. 2 is a flow chart illustrating a processing method of the visual saliency extraction means shown in FIG. 1 . FIG. 2 is a diagram illustrating in detail an example of the configuration of a nonlinear mapping unit. FIG. 2 is a diagram illustrating a configuration of an intermediate layer. 13A and 13B are diagrams illustrating an example of convolution processing performed by a filter. FIG. 1A is a diagram for explaining the processing of a first pooling unit, FIG. 1B is a diagram for explaining the processing of a second pooling unit, and FIG. 1C is a diagram for explaining the processing of an unpooling unit. FIG. 11 is an explanatory diagram of a vector error. 2 is an example of an image input to the image input unit shown in FIG. 1 and a visual saliency map obtained from the image. 11 is a graph showing an example of a change in visual attention concentration level over time. 2 is a flowchart of an operation of the information processing device shown in FIG. 1 . FIG. 11 is a diagram showing an example of an intersection targeted by an information processing device according to a second embodiment of the present invention. FIG. 14 is a diagram showing calculations of visual attention concentration levels by setting an ideal line of sight for the intersection shown in FIG. 13. 15 is a graph showing a change over time in the visual attention concentration level shown in FIG. 14. 16 is a graph showing the results of calculating the ratio of the visual attention concentration level shown in FIG. 15 when turning right or left and when going straight. 10 is a flowchart of an operation of an information processing apparatus according to a second embodiment of the present invention. This is an example of a curve that is the subject of a modification of the second embodiment.

An information processing device according to one embodiment of the present invention will be described below. In the information processing device according to one embodiment of the present invention, an acquisition unit acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image based on an image of the outside captured from a moving body, and a gaze position setting unit sets a reference gaze position in the image according to a predetermined rule. Then, a visual attention concentration calculation unit calculates the visual attention concentration level in the image based on the visual saliency distribution information and the gaze position. In this way, since the visual saliency distribution information is used, it is possible to reflect a contextual attention state in which the gaze tends to be unconsciously focused on objects such as signs and pedestrians contained in the image. Therefore, it is possible to accurately calculate indicators related to safety and risk.

The visual attention concentration calculation unit may also calculate the visual attention concentration level based on the value of each pixel constituting the visual saliency distribution information and the vector error between the position of each pixel and the coordinate position of the reference gaze position. In this way, a value according to the difference between a position of high visual saliency and the reference gaze position is calculated as the visual attention concentration level. Therefore, for example, the value of the visual attention concentration level can be changed according to the distance between the position of high visual saliency and the reference gaze position.

The system may also include an output unit that outputs information about the risk at the location indicated by the image based on the change over time in the degree of visual attention concentration. In this way, it is possible to output, for example, a location where the change over time in the degree of visual attention concentration is large as an accident risk location, etc.

The acquisition unit may also include an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and an output unit that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit may include a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit. In this way, visual saliency can be estimated with low computational cost. Furthermore, the visual saliency estimated in this way reflects the contextual attention state.

In addition, in an information processing method according to one embodiment of the present invention, in an acquisition step, visual saliency distribution information is acquired based on an image of the outside taken from a moving body, and the visual saliency distribution information is obtained by estimating the level of visual saliency in the image, and in a gaze position setting step, a reference gaze position in the image is set according to predetermined rules. Then, in a visual attention concentration calculation step, the visual attention concentration level in the image is calculated based on the visual saliency distribution information and the gaze position. In this manner, the use of visual saliency distribution information makes it possible to reflect a contextual attention state in which the gaze tends to be unconsciously focused on objects such as signs and pedestrians contained in the image. Therefore, it becomes possible to accurately calculate indicators related to safety and risk.

The above-mentioned information processing method is also executed by a computer. In this way, visual saliency distribution information that estimates visual saliency using a computer is used, so it is possible to reflect the contextual attention state in which the gaze tends to be unconsciously focused on objects such as signs and pedestrians contained in an image. This makes it possible to calculate safety and risk-related indicators with high accuracy.

The above-mentioned information processing program may also be stored on a computer-readable storage medium. In this way, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades, etc., can be easily performed.

An information processing device according to a first embodiment of the present invention will be described with reference to Figs. 1 to 12. The information processing device according to this embodiment is not limited to being installed in a moving object such as an automobile, but may be configured as a server device or the like installed in a business establishment or the like. In other words, analysis does not need to be performed in real time, and analysis may be performed after driving, etc.

As shown in FIG. 1, the information processing device 1 includes an image input unit 2, a visual saliency calculation unit 3, a gaze coordinate setting unit 4, a vector error calculation unit 5, and an output unit 6.

Image input unit 2 receives input of an image (e.g., a moving image) captured by, for example, a camera, and outputs the image as image data. The input moving image is output as image data broken down into a time series, for example, for each frame. Although still images may be input as images to be input to image input unit 2, it is preferable to input them as an image group consisting of multiple still images in a time series.

The images input to the image input unit 2 include, for example, images captured in the direction of travel of the vehicle. In other words, images are images captured continuously of the outside from a moving body. These images may be images that include angles other than the direction of travel, such as 180° or 360° in the horizontal direction, such as so-called panoramic images or images captured using multiple cameras. Furthermore, images input to the image input unit 2 are not limited to images captured by a camera, and may also be images read from a recording medium such as a hard disk drive or memory card.

The visual saliency calculation unit 3 receives image data from the image input unit 2 and outputs a visual saliency map as visual saliency estimation information described below. In other words, the visual saliency calculation unit 3 functions as an acquisition unit that acquires a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency based on an image captured of the outside from a moving body.

Figure 2 is a block diagram illustrating the configuration of the visual saliency calculation unit 3. The visual saliency calculation unit 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds image data input from the image input unit 2, filter coefficients described later, and the like. This will be described in detail below.

Figure 3(a) is a diagram illustrating an example of an image input to the visual saliency calculation unit 3, and Figure 3(b) is a diagram illustrating an example of an image showing a visual saliency distribution estimated for Figure 3(a). The visual saliency calculation unit 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual saliency means, for example, how easily something stands out or how easily it attracts attention. Specifically, visual saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds, for example, to the probability that the gaze of a person viewing the image will be directed to that position.

Figure 3(a) and Figure 3(b) correspond to each other in terms of position. In Figure 3(a), the higher the visual saliency is at a position, the higher the luminance is displayed in Figure 3(b). The image showing the visual saliency distribution as in Figure 3(b) is an example of a visual saliency map output by the output unit 330. In this example, visual saliency is visualized with 256 gradations of luminance values. An example of a visual saliency map output by the output unit 330 will be described in detail later.

Figure 4 is a flowchart illustrating the operation of the visual saliency calculation unit 3 according to this embodiment. The flowchart shown in Figure 4 is a part of an information processing method executed by a computer, and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, an image is converted into intermediate data that can be subjected to mapping processing. In the nonlinear mapping step S120, the intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information (visual saliency distribution information) indicating a saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 in which features are extracted from the intermediate data, and an upsampling step S122 in which the data generated in the feature extraction step S121 is upsampled.

Returning to FIG. 2, each component of the visual saliency calculation unit 3 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the image input unit 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, and is, for example, a high-dimensional tensor. The intermediate data is, for example, data in which the luminance of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a luminance gradient. In the input step S110, the input unit 310 may further perform noise removal and resolution conversion of the image.

In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. The intermediate data is then converted to mapping data in the nonlinear mapping unit 320. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data in the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, etc., and is preferably processing using a function, functional, or neural network.

Figure 5 is a diagram illustrating in detail the configuration of the nonlinear mapping unit 320, and Figure 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsampling step S122 is performed in the upsampling unit 322. In the example shown in this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including multiple intermediate layers 323. In the neural network, multiple intermediate layers 323 are connected.

In particular, it is preferable that the neural network is a convolutional neural network. Specifically, each of the multiple intermediate layers 323 includes one or more convolutional layers 324. In the convolutional layer 324, the input data is convolved by multiple filters 325, and activation processing is performed on the output of the multiple filters 325.

In the example of FIG. 5, the feature extraction unit 321 is configured to include a neural network including multiple intermediate layers 323, and includes a first pooling unit 326 between the multiple intermediate layers 323. The upsampling unit 322 is configured to include a neural network including multiple intermediate layers 323, and includes an unpooling unit 328 between the multiple intermediate layers 323. Furthermore, the feature extraction unit 321 and the upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

In the example shown in this figure, each intermediate layer 323 is composed of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may be composed of only one convolutional layer 324. Adjacent intermediate layers 323 are separated by either a first pooling unit 326, a second pooling unit 327, or an unpooling unit 328. Here, when an intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the number of filters 325 in those convolutional layers 324 is equal to each other.

In this diagram, the intermediate layer 323 marked "A×B" is composed of B convolution layers 324, meaning that each convolution layer 324 includes A convolution filters for each channel. Such an intermediate layer 323 is also referred to below as an "A×B intermediate layer." For example, a 64×2 intermediate layer 323 is composed of two convolution layers 324, meaning that each convolution layer 324 includes 64 convolution filters for each channel.

In the example shown in this figure, the feature extraction unit 321 includes a 64x2 intermediate layer 323, a 128x2 intermediate layer 323, a 256x3 intermediate layer 323, and a 512x3 intermediate layer 323, in this order. The upsampling unit 322 includes a 512x3 intermediate layer 323, a 256x3 intermediate layer 323, a 128x2 intermediate layer 323, and a 64x2 intermediate layer 323, in this order. The second pooling unit 327 connects the two 512x3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined, for example, according to the number of pixels of the image data.

Note that this diagram is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have other configurations. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of convolutional layers 324 included in the intermediate layer 323 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of channels in the intermediate layer 323 may further reduce the computational cost. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the multiple intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed through. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling unit 326, and the second intermediate layer 323b is located behind the first intermediate layer 323a. The first intermediate layer 323a is composed of a convolution layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolution layer 324 in which the number of filters 325 for each channel is N2. At this time, it is preferable that N2>N1 holds. It is more preferable that N2=N1×2 holds.

In addition, in the multiple intermediate layers 323 included in the upsampling unit 322, it is preferable that the number of filters 325 decreases each time the unpooling unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling unit 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. The third intermediate layer 323c is composed of a convolution layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolution layer 324 in which the number of filters 325 for each channel is N4. At this time, it is preferable that N4<N3 holds. It is more preferable that N3=N4×2 holds.

The feature extraction unit 321 extracts image features with multiple levels of abstraction, such as gradients and shapes, from the intermediate data acquired from the input unit 310 as channels for the intermediate layer 323.
The configuration of the intermediate layer 323 is illustrated. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 has 64 filters 325. In the first convolution layer 324a, convolution processing using the filter 325 is performed on each channel of data input to the intermediate layer 323. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h ⁰ _i (i = 1..3). In addition, in the example of this figure, the filter 325 is 64 types of 3 × 3 filters, that is, a total of 64 × 3 types of filters. As a result of the convolution processing, 64 results h ⁰ _i,j (i = 1..3, j = 1..64) are obtained for each channel i.

Next, the outputs of the multiple filters 325 are subjected to activation processing in an activation unit 329. Specifically, for the corresponding results j of all channels, activation processing is performed on the sum of each corresponding element. By this activation processing, the results h ¹ _i (i=1..64) of 64 channels are generated.
), that is, the output of the first convolution layer 324a is obtained as the image feature. The activation process is not particularly limited, but it is preferable to use at least one of a hyperbolic function, a sigmoid function, and a rectified linear function.

Furthermore, the output data of the first convolution layer 324a is used as input data for the second convolution layer 324b, which performs the same processing as the first convolution layer 324a to obtain the 64-channel result ^h2i ( _i =1..64), i.e., the output of the second convolution layer 324b, as the image feature. The output of the second convolution layer 324b becomes the output data of this 64×2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but it is preferable that the filter 325 is a two-dimensional filter of 3×3. The coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are stored in the memory unit 390, and the nonlinear mapping unit 320 can read them and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapping data. The memory unit 390 may be provided in the visual saliency calculation unit 3, or may be provided outside the visual saliency calculation unit 3. The nonlinear mapping unit 320 may also obtain the correction information from outside via a communication network.

7(a) and 7(b) are diagrams showing examples of convolution processing performed by the filter 325. In both of Figs. 7(a) and 7(b), examples of 3x3 convolution are shown. The example in Fig. 7(a) is convolution processing using nearest neighbor elements. The example in Fig. 7(b) is convolution processing using nearby elements with a distance of two or more. Note that convolution processing using nearby elements with a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using nearby elements with a distance of two or more. This is because it is possible to extract a wider range of features and further improve the estimation accuracy of visual saliency.

The above describes the operation of the 64×2 intermediate layer 323. The operation of the other intermediate layers 323 (such as the 128×2 intermediate layer 323, the 256×3 intermediate layer 323, and the 512×3 intermediate layer 323) is the same as that of the 64×2 intermediate layer 323, except for the number of convolutional layers 324 and the number of channels. In addition, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsampling unit 322 are also the same as above.

Figure 8(a) is a diagram for explaining the processing of the first pooling unit 326, Figure 8(b) is a diagram for explaining the processing of the second pooling unit 327, and Figure 8(c) is a diagram for explaining the processing of the unpooling unit 328.

In the feature extraction unit 321, the data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. In the first pooling unit 326, for example, non-overlapping pooling processing is performed. FIG. 8(a) shows a process of associating four 2×2 elements 30 with one element 30 for the element group included in each channel. In the first pooling unit 326, such association is performed for all elements 30. Here, the four 2×2 elements 30 are selected so that they do not overlap with each other. In this example, the number of elements in each channel is reduced to one-fourth. Note that, as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is performed on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of matching four 2×2 elements 30 to one element 30 while overlapping some of the elements 30. That is, in repeated matching, some of the four 2×2 elements 30 in a certain matching are also included in the four 2×2 elements 30 in the next matching. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. Note that the number of elements 30 before and after matching in the second pooling unit 327 is not particularly limited.

The method of each process performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but examples include matching in which the maximum value of four elements 30 is matched to one element 30 (max pooling) and matching in which the average value of four elements 30 is matched to one element 30 (average pooling).

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. The output data from the intermediate layer 323 of the upsampling unit 322 is subjected to unpooling processing for each channel in the unpooling unit 328, and then input to the next intermediate layer 323. Figure 8(c) shows the process of expanding one element 30 to multiple elements 30. The method of expansion is not particularly limited, but one example is a method of duplicating one element 30 to four elements 30 (2 x 2).

The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which visual saliency is visualized by brightness values, as shown in FIG. 3B. The visual saliency map may also be, for example, an image that is colored according to visual saliency, such as a heat map, or an image in which visual saliency areas with visual saliency higher than a predetermined standard are marked in a manner that makes them distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information shown as an image or the like, and may be a table or the like that lists information indicating visual saliency areas.

The gaze coordinate setting unit 4 sets the ideal gaze, which will be described later, on the visual saliency map. The ideal gaze is the gaze that a driver of a vehicle directs along the direction of travel in an ideal traffic environment in which there are no obstacles or other traffic participants other than the driver himself. It is handled as (x, y) coordinates on the image data and visual saliency map. Note that in this embodiment, the ideal gaze is a fixed value, but it may be handled as a function of the speed or road friction coefficient that affects the stopping distance of a moving object, or may be determined using set route information. In other words, the gaze coordinate setting unit 4 functions as a gaze position setting unit that sets the ideal gaze (reference gaze position) in the image according to predetermined rules.

The vector error calculation unit 4 calculates a vector error based on the visual saliency map output by the visual saliency calculation unit 3 and the ideal gaze set by the gaze coordinate setting unit 5 for the visual saliency map and image, and calculates a visual attention concentration level Ps (described below) indicating the degree of visual attention concentration based on the vector error. In other words, the vector error calculation unit 4 functions as a visual attention concentration level calculation unit that calculates the degree of visual attention concentration in an image based on visual saliency distribution information and gaze position.

Here, the vector error in this embodiment will be described with reference to FIG. 9. FIG. 9 shows an example of a visual saliency map. This visual saliency map is shown with 256 gradation brightness values of H pixels x V pixels, and as in FIG. 3, pixels with higher visual saliency are displayed with higher brightness. In FIG. 9, when the coordinates (x, y) of the ideal gaze are set to ( _xim , _yim ), a vector error between the pixel and any coordinate (k, m) in the visual saliency map is calculated. When the coordinates of the ideal gaze with high brightness in the visual saliency map are far from each other, it means that the position to be gazed upon and the position that is actually easy to gaze upon are far from each other, and it can be said that the image is likely to distract visual attention. On the other hand, when the coordinates of the ideal gaze with high brightness are close to each other, it means that the position to be gazed upon and the position that is actually easy to gaze upon are close to each other, and it can be said that the image is likely to concentrate visual attention on the position to be gazed upon.

Next, a description will be given of a method for calculating the visual attention concentration level Ps in the vector error calculation unit 4. In this embodiment, the visual attention concentration level Ps is calculated by the following formula (1).

In formula (1), V _vc is pixel depth (luminance value), f _w is weighting function, and d _err is vector error. This weighting function is a function that is weighted based on the distance from a pixel indicating the value of V _vc to the coordinates of the ideal gaze. α is a coefficient that makes the visual attention concentration level Ps equal to 1 when the coordinates of a bright point and the coordinates of the ideal gaze match in a visual saliency map (reference heat map) of one bright point.

That is, the vector error calculation unit 5 (visual attention concentration calculation unit) calculates the degree of visual attention concentration based on the value of each pixel that constitutes the visual saliency map (visual saliency distribution information) and the vector error between the position of each pixel and the coordinate position of the ideal gaze (reference gaze position).

The visual attention concentration level Ps obtained in this way is the reciprocal of the weighted sum of the relationship between the vector error and brightness value of the coordinates of all pixels from the coordinates of the ideal gaze set on the visual saliency map. This visual attention concentration level Ps is calculated as a low value when the distribution of high brightness on the visual saliency map is far from the coordinates of the ideal gaze. In other words, the visual attention concentration level Ps can also be said to be the concentration level with respect to the ideal gaze.

Figure 10 shows an example of an image input to the image input unit 2 and a visual saliency map obtained from that image. Figure 10 (a) is the input image, and (b) is the visual saliency map. In Figure 10, if the coordinates of the ideal line of sight are set on the road, for example, on a truck traveling ahead, the visual attention concentration level Ps in that case is calculated.

The output unit 6 outputs information regarding risk for the scene shown by the image for which the visual attention concentration level Ps was calculated based on the visual attention concentration level Ps calculated by the vector error calculation unit 5. For example, the information regarding risk may be such that a predetermined threshold is set for the visual attention concentration level Ps, and information is output indicating that the scene is high risk if the calculated visual attention concentration level Ps is equal to or below the threshold. For example, if the visual attention concentration level Ps calculated in FIG. 10 is equal to or below the threshold, it may be determined that the scene is high risk, and information such as "risk present (or high risk)" may be output.

The output unit 6 may also output information related to risk based on the change over time in the visual concentration level Ps calculated by the vector error calculation unit 5. Figure 11 shows an example of the change over time in the visual concentration level Ps. Figure 11 shows the change in the visual concentration level Ps in a 12-second video. In Figure 11, the visual concentration level Ps changes suddenly between approximately 6.5 seconds and approximately 7 seconds. This occurs, for example, when another vehicle cuts in front of the vehicle.

As shown in FIG. 11, a scene may be determined to be high risk by comparing the rate of change per short period of time or the change value of the visual concentration level Ps with a predetermined threshold, and information indicating the presence of risk (or high risk) may be output. In addition, the presence or absence of risk (high or low) may be determined based on a pattern of change, for example, the visual concentration level Ps decreasing once and then increasing.

Next, the operation (information processing method) of the information processing device 1 configured as described above will be described with reference to the flowchart in FIG. 12. In addition, this flowchart can be configured as a program executed by a computer that functions as the information processing device 1 to become an information processing program. In addition, this information processing program is not limited to being stored in a memory or the like possessed by the information processing device 1, but may also be stored in a storage medium such as a memory card or optical disk.

First, the image input unit 2 outputs the input image as image data to the visual saliency calculation unit 3 (step S11). In this step, the image data input to the image input unit 2 is decomposed into a time series such as image frames, and input to the visual saliency calculation unit 3. In this step, image processing such as noise removal and geometric transformation may also be performed.

Next, the visual saliency calculation unit 3 acquires a visual saliency map (step S12). The visual saliency calculation unit 3 outputs the visual saliency map in time series as shown in FIG. 3(b) by the method described above.

Meanwhile, in parallel with step S12, the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze (step S13). As described above, these coordinates are fixed positions such as forward gaze.

Next, the vector error calculation unit 5 calculates the visual attention concentration level Ps from the visual saliency map and the ideal gaze (step S14). That is, as described above, the vector error between the coordinates of the ideal gaze and the coordinates of the visual saliency map is calculated, and the visual attention concentration level Ps is calculated using formula (1) based on the vector error and the value of each pixel.

Next, the output unit 6 outputs risk information (step S15). In this step, as described above, risk information is output based on one calculated visual attention concentration level Ps or a change over time.

As is clear from the above explanation, step S12 functions as an acquisition process, step S13 functions as a gaze position setting process, and step S14 functions as a visual attention concentration calculation process.

According to this embodiment, the information processing device 1 acquires a visual saliency map obtained by estimating the level of visual saliency in an image captured from a moving object using an image of the outside world, and the gaze coordinate setting unit 4 sets the coordinates of an ideal gaze at a predetermined fixed position. Then, the vector error calculation unit 5 calculates the visual attention concentration level Ps in the image based on the visual saliency map and the ideal gaze. In this way, the use of the visual saliency map makes it possible to reflect a contextual attention state in which the gaze tends to be unconsciously focused on objects such as signs and pedestrians contained in the image. Therefore, it becomes possible to accurately calculate indicators related to safety and risk.

The vector error calculation unit 5 also calculates the visual attention concentration level Ps based on the value of each pixel constituting the visual saliency map and the vector error between the position of each pixel and the coordinate position of the ideal gaze. In this way, a value according to the difference between a position of high visual saliency and the ideal gaze is calculated as the visual attention concentration level Ps. Therefore, for example, it is possible to make the value of the visual attention concentration level Ps change depending on the distance between the position of high visual saliency and the ideal gaze.

The system also includes an output unit 6 that outputs risk information at the location indicated by the image based on the change over time in the visual concentration level Ps. In this way, it is possible to output, for example, a location where the change over time in the visual concentration level Ps is large as an accident risk location, etc. The risk information may also be output as information related to near misses based on the change over time in the visual concentration level Ps.

The visual saliency calculation unit 3 also includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. In this way, visual saliency can be estimated with low calculation cost. Furthermore, the visual saliency estimated in this way reflects the contextual attention state.

Next, a risk information output device according to a second embodiment of the present invention will be described with reference to Figures 13 to 16. Note that parts that are the same as those in the first embodiment described above will be given the same reference numerals and descriptions thereof will be omitted.

In this embodiment, the block configuration and the like are the same as in FIG. 1. In other words, the information processing device 1 functions as a risk information output device according to this embodiment. The image input from the image input unit 2 is an image of entering an intersection, and the method of risk determination in the output unit 6 is different.

Figure 13 shows an example of an intersection for which risk information is output in this embodiment. Figure 13 shows an intersection that is a four-way intersection (crossroads). Images of the intersection direction (travel direction) when entering from directions A, B, and C are shown. In other words, the image shown in Figure 13 is an image when directions A, B, and C are each considered to be the approach roads when entering the intersection.

For the image shown in Figure 13, a visual saliency map is obtained as described in the first embodiment. Then, for the image, an ideal gaze is set for each traveling direction of going straight, turning right, and turning left, and the visual attention concentration level Ps is calculated for each ideal gaze (Figure 14). In other words, for each exit road that is the road to exit after entering the intersection, an ideal gaze (reference gaze position) in the image is set, and the vector error calculation unit 5 calculates the visual attention concentration level Ps for each ideal gaze.

Figure 15 shows the change over time in visual concentration level Ps when entering an intersection from each approach road. Such changes over time are based on the calculation results of the vector error calculation unit 5. The graph in Figure 15 shows visual concentration level Ps on the vertical axis and time on the horizontal axis, with the thick line indicating a straight line, the thin line indicating a left turn, and the dashed line indicating a right turn when the ideal line of sight is set for each direction of travel. Figure 15(a) shows the case of entering from direction A, Figure 15(b) shows the case of entering from direction B, and Figure 15(c) shows the case of entering from direction C.

According to FIG. 15, visual concentration level Ps tends to decrease when approaching an intersection, but in some cases it drops sharply just before the intersection as in FIG. 15(b). Also, according to FIG. 15, visual concentration level Ps tends to be lower when it is assumed that the driver looks left or right to turn right or left than when it is assumed that the driver looks straight ahead to go straight.

Next, the output unit 6 calculates the ratio of the visual attention concentration level Ps in the right or left direction to the visual attention concentration level Ps in the straight direction by utilizing the change over time of the visual attention concentration level Ps calculated in FIG. 15. The change of the calculated ratio is shown in FIG. 16. In the graph of FIG. 16, the vertical axis indicates the ratio and the horizontal axis indicates the time, the thick line indicates the left turn/straight-going ratio (L/C), and the thin line indicates the right turn/straight-going ratio (R/C). FIG. 16(a) shows the case of entering from the A direction, FIG. 16(b) shows the case of entering from the B direction, and FIG. 16(c) shows the case of entering from the C direction. For example, I _AL in FIG. 16(a) indicates PS _LA (left visual attention concentration level in the A direction)/PS _CA (straight-going visual attention concentration level in the A direction), and I _AR indicates PS _RA (right visual attention concentration level in the A direction)/PS _CA (straight-going visual attention concentration level in the A direction). I _BL and I _BR in FIG. 16B and I _CL and I _CR in FIG. 16C have the same meaning but differ in the approach direction.

According to Fig. 16, when the ratio of visual attention concentration levels such as I _AL and I _AR is smaller than 1, it can be said that the intersection is one where the driver's concentration level (= visual attention concentration level Ps) is lower when the driver looks to turn right or left than when the driver looks to go straight. Conversely, when the ratio is larger than 1, it can be said that the intersection is one where the driver's concentration level is lower when the driver looks to go straight.

Therefore, the output unit 6 can determine the risk state of the target intersection based on the temporal change in the visual attention concentration level Ps and the ratio of the visual attention concentration levels Ps as described above, and output the determination result as information related to the risk.

Next, the operation (information processing method) of the information processing device 1 according to this embodiment will be described with reference to the flowchart in FIG. 17.

First, the image input unit 2 outputs the input image to the visual saliency calculation unit 3 as image data (step S21). In this step, as shown in FIG. 13, images of vehicles approaching the intersection from each approach road are acquired. At this time, it is advisable to acquire not only the images but also location information and time information at the same time. The location information makes it possible to determine the direction of approach, and the time information makes it possible to analyze by time period, such as morning, afternoon, and night, and also to calculate the approach speed to the intersection.

Next, the visual saliency calculation unit 3 acquires a visual saliency map (step S22). The visual saliency map is acquired based on images from each entry road. Meanwhile, in parallel with step S22, the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze (step S23). The ideal gaze in step S23 is set for each travel direction (exit road), as shown in FIG. 14.

Next, the vector error calculation unit 5 calculates the visual attention concentration level Ps from the visual saliency map and the ideal line of sight (step S24). In this step, the visual attention concentration level Ps is calculated for each exit route for which the ideal line of sight is set, as shown in FIG. 14. In addition, the visual attention concentration level Ps is calculated in time series, as shown in FIG. 15.

Next, the output unit 6 judges the risk of the intersection based on the visual attention concentration level Ps calculated in step S24 (step S25). For example, if the visual attention concentration level Ps changes suddenly when approaching the intersection and the intersection is an approach road from multiple directions, the intersection is judged to be risky (or high risk). Alternatively, if the above-mentioned ratio is smaller than 1 when approaching the intersection and the intersection is an approach road from multiple directions, the intersection is judged to be risky (or high risk). Alternatively, the intersection may be judged to be risky (or high risk) if both of these conditions are met. Furthermore, the intersection may be judged to be risky (or high risk) if the above conditions are met multiple times. "Multiple times" refers to the execution of this flowchart multiple times based on images taken of the same intersection at different times (or by different vehicles).

Then, the output unit 6 outputs the judgment result of step S25 as risk information (step S26).

As is clear from the above explanation, step S22 functions as an acquisition process, step S23 functions as a gaze position setting process, step S24 functions as a visual attention concentration calculation process, and steps S25 and S26 function as output processes.

According to this embodiment, the information processing device 1 has a visual saliency calculation unit 3 that obtains a visual saliency map for each entry road, which is a road used when entering an intersection, by estimating the level of visual saliency in the image from the image of each entry road, and a gaze coordinate setting unit 4 that sets the coordinates of the ideal gaze in the image for each exit road, which is a road used to exit the intersection after entering, for the visual saliency map. Then, the vector error calculation unit 5 calculates the visual attention concentration level Ps for each exit road in the image based on the visual saliency map and the ideal gaze, and the output unit 6 outputs risk information at the intersection based on the visual attention concentration level Ps calculated for each exit road. In this way, it is possible to evaluate the risk for a target intersection and output risk information.

The output unit 6 also outputs risk information based on the ratio of the visual attention concentration level Ps of the exit route going straight to the visual attention concentration level Ps of the exit route turning right or left. In this way, it is possible to evaluate which direction the driver's attention is more likely to be directed to, going straight or turning right or left, and output the evaluation result.

The output unit 6 may also output risk information based on a change over time in the visual concentration level Ps. In this way, it is possible to detect, for example, a sudden change in the visual concentration level Ps and output risk information.

In the second embodiment, for example, in FIG. 14, when entering the intersection from direction A, the visual attention concentration level Ps for turning right (toward direction B) decreases. When entering the intersection from direction B, the visual attention concentration level Ps for turning right or left (toward direction A or C) decreases. When entering the intersection from direction C, the visual attention concentration level Ps for turning right decreases.

In this case, for example, the route turning right from direction A and the route turning left from direction B are both routes in which the visual attention concentration level Ps is lower than other routes, and furthermore, they become the same route when the entry road and exit road are swapped. Therefore, at this intersection, information regarding risk, such as "risky (or high risk)" for this route, may be output.

In addition, while the second example describes an intersection, this concept can also be applied to curves on a road. This will be explained with reference to Figure 18.

Figure 18 is an example of a curved road. This road can be driven on as a left curve from direction D (bottom of the figure), or as a right curve from direction E (left side of the figure). Here, for example, when entering a curve from direction D, an ideal line of sight is not only set to the left, which is the direction in which the road curves, but also to the direction in which the road would have been if it had been going straight (direction D'), and a visual attention concentration level Ps is calculated for each. Similarly, when entering a curve from direction E, an ideal line of sight is not only set to the right, which is the direction in which the road curves, but also to the direction in which the road would have been if it had been going straight (direction E'), and a visual attention concentration level Ps is calculated for each.

Then, based on the calculated visual attention concentration level Ps, risk can be determined based on time series changes, ratios, etc., in the same way as at intersections.

When the curvature of the curve is large as in Figure 18, the virtual ideal line of sight may be set not only in the straight-ahead direction but also in the direction opposite to the curvature of the curve. In Figure 18, if the vehicle is entering from direction D, the ideal line of sight may be set not only in direction D' but also in direction E' to calculate the visual attention concentration level Ps. In other words, the ideal line of sight may be set in a direction different from the curvature of the curve.

That is, the visual saliency calculation unit 3 obtains a visual saliency map obtained by estimating the level of visual saliency in an image taken when entering a curve on a road, and the gaze coordinate setting unit 4 sets the coordinates of the ideal gaze in the image in the curvature direction of the curve and in a direction different from the curvature direction for the visual saliency map. Then, the vector error calculation unit 5 calculates the visual attention concentration level Ps in the curvature direction and in a direction different from the curvature direction in the image based on the visual saliency map and the ideal gaze, and the output unit 6 outputs risk information on the curve based on the visual attention concentration level Ps calculated for each exit route.

By doing this, it is possible to evaluate the risk for the target curve and output information related to the risk.

Furthermore, the present invention is not limited to the above-mentioned embodiment. In other words, a person skilled in the art can implement the present invention by modifying it in various ways in accordance with conventional knowledge without departing from the gist of the present invention. As long as the information processing device of the present invention is still included even after such modifications, it is of course included in the scope of the present invention.

1. Information processing device (risk information output device)
2 Image input unit 3 Visual saliency calculation unit (acquisition unit)
4. Gaze coordinate setting unit (gaze position setting unit)
5. Vector error calculation unit (visual attention concentration calculation unit)
6 Output section

Claims (6)

an acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in an image captured from a moving object;
a gaze position setting unit that sets a reference gaze position in the image according to a predetermined rule;
a visual attention concentration degree calculation unit that calculates a visual attention concentration degree in the image based on the visual saliency distribution information and the reference gaze position;
Equipped with
the visual attention concentration level calculation unit calculates the visual attention concentration level by the reciprocal of a weighted sum of the vector errors of the coordinates of all pixels from the coordinates of the reference gaze position set on the visual saliency distribution information and the values of the pixels, based on a value of each pixel constituting the visual saliency distribution information and a vector error between the position of each pixel and a coordinate position of the reference gaze position;
23. An information processing apparatus comprising: The information processing device according to claim 1 , further comprising an output unit that outputs information relating to a risk at a location indicated by the image based on a temporal change in the degree of visual attention concentration. The acquisition unit is
an input unit for converting the image into intermediate data that can be subjected to mapping processing;
a nonlinear mapping unit that converts the intermediate data into mapping data;
an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data,
The nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples data generated by the feature extraction unit.
3. The information processing apparatus according to claim 1 , wherein the information processing apparatus further comprises: a first input section; An information processing method executed by an information processing device that performs predetermined information processing based on an image captured from a moving object of the outside, comprising:
an acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in the image based on the image;
a gaze position setting step of setting a reference gaze position in the image according to a predetermined rule;
a visual attention concentration calculation step of calculating a visual attention concentration level in the image based on the visual saliency distribution information and the reference gaze position;
Including,
the visual attention concentration degree calculation step calculates the visual attention concentration degree by the reciprocal of a weighted sum of the vector errors of coordinates of all pixels from the coordinates of the reference gaze position set on the visual saliency distribution information and the values of the pixels, based on the values of each pixel constituting the visual saliency distribution information and the vector errors between the positions of the pixels and the coordinate positions of the reference gaze position;
23. An information processing method comprising: 5. An information processing program for causing a computer to execute the information processing method according to claim 4 . 6. A computer-readable storage medium storing the information processing program according to claim 5 .

Images