JP7405570B2 - Visual load estimation device - Google Patents

Description

The present invention relates to a visible load amount estimating device that estimates a visible load amount based on an image taken of the outside from a moving object.

Patent Document 1 discloses that in order to estimate how easy it is for the eyes to get tired in the driving environment from an image taken in the traveling direction of the own vehicle, a photographed image in the traveling direction of the own vehicle is acquired, and an image in the photographed image is , the position that the driver physiologically gazes at is estimated, the position in the captured image that the driver should see when driving the own vehicle is estimated, and the position that the driver looks at and the position that should be visually recognized are estimated. It is described that the amount of visible load is estimated based on the positional relationship with the vehicle.

Patent No. 5482737

However, what causes a visual burden is not limited to the gap between the position to be visually recognized and the physiological degree of gaze, as described in Patent Document 1. For example, scenes in which the line of sight changes significantly may also cause a visual burden, but Patent Document 1 does not take such scenes into consideration.

An example of the problem to be solved by the present invention is to automatically extract a visually burdensome part.

In order to solve the above problem, the invention according to claim 1 uses visual saliency distribution information obtained by estimating the level of visual saliency based on images continuously captured outside from a moving body. a generation unit that generates for each image, a calculation unit that calculates an estimated movement amount of the gaze point based on the generated visual saliency distribution information, and a calculation unit that calculates the movement amount of the estimated gaze point based on the temporal transition of the calculated movement amount of the estimated gaze point. and an estimating section that estimates the amount of visible load.

The invention according to claim 6 is a visible load amount estimation method executed by a visible load amount estimation device that estimates a visible load amount based on an image taken of the outside from the moving body, a generation step of generating visual saliency distribution information for each image, which is obtained by estimating the level of visual saliency based on continuously captured images, and based on the generated visual saliency distribution information; The present invention is characterized by comprising a calculation step of calculating the estimated movement amount of the gaze point, and an estimation step of estimating the amount of visual recognition load based on the temporal transition of the calculated movement amount of the estimated gaze point.

The invention according to claim 7 is characterized in that the visual load estimation method according to claim 6 is executed by a computer.

The invention according to claim 8 is characterized in that the visual load estimation program according to claim 7 is stored.

FIG. 1 is a functional configuration diagram of a visual load estimation device according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the configuration of visual saliency extraction means shown in FIG. 1. FIG. (a) is a diagram illustrating an image input to the determination device, and (b) is a diagram illustrating a visual saliency map estimated for (a). 2 is a flowchart illustrating a processing method of the visual saliency extraction means shown in FIG. 1. FIG. FIG. 3 is a diagram illustrating in detail the configuration of a nonlinear mapping section. FIG. 3 is a diagram illustrating a configuration of an intermediate layer. (a) and (b) are diagrams each showing an example of convolution processing performed by a filter. (a) is a diagram for explaining the processing of the first pooling section, (b) is a diagram for explaining the processing of the second pooling section, and (c) is a diagram for explaining the processing of the second pooling section. FIG. FIG. 2 is a functional configuration diagram of a visual load estimation means shown in FIG. 1. FIG. 10 is a waveform diagram showing the operation of each function in the visual load estimation means shown in FIG. 9. FIG. 2 is a flowchart of the operation of the visual load estimation device shown in FIG. 1. FIG.

A visual load estimation device according to an embodiment of the present invention will be described below. A visual recognition load estimation device according to an embodiment of the present invention uses visual saliency distribution information obtained by estimating the level of visual saliency based on images continuously captured outside from a moving object in a generation unit. The amount of movement of the estimated point of gaze is calculated based on the visual saliency distribution information generated for each image and generated by the calculation unit. Then, the amount of visual recognition load is estimated based on the temporal transition of the amount of movement of the estimated point of gaze calculated by the estimator. By doing this, it is possible to estimate a scene in which the change in the line of sight becomes large based on the time change in the amount of movement of the estimated point of gaze. Therefore, it is possible to automatically extract portions that are visually burdensome.

Further, the calculation unit may calculate the movement amount by estimating the estimated gaze point as a position on the image where the visual saliency is the maximum value in the visual saliency distribution information. By doing so, the amount of movement can be calculated based on the position that is estimated to be most visible.

Furthermore, the estimation unit may decompose the estimated movement amount of the gaze point into a plurality of base components, and calculate the visual recognition load amount based on the decomposed base components. By doing so, it becomes possible to use an algorithm such as Empirical Mode Decomposition (EMD).

The generation unit also includes an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapping data, and generates saliency estimation information that indicates a saliency distribution based on the mapping data. The nonlinear mapping section may include a feature extraction section that extracts features from the intermediate data, and an upsampling section that upsamples the data generated by the feature extraction section. By doing so, visual saliency can be estimated with low calculation cost.

Further, the information processing apparatus may include a presentation section that presents the estimation results of the estimation section. By doing this, for example, it becomes possible to notify that a point with a large amount of visible load is approaching.

Furthermore, in the visual recognition load estimation method according to an embodiment of the present invention, a visual saliency distribution obtained by estimating the level of visual saliency based on images continuously captured outside from a moving object in the generation step. Information is generated for each image, and the amount of movement of the estimated point of gaze is calculated based on the visual saliency distribution information generated in the calculation step. Then, the visual recognition load amount is estimated based on the temporal transition of the movement amount of the estimated gaze point calculated in the estimation step. By doing so, it is possible to estimate a position where the change in the line of sight is large based on the time change in the amount of movement of the estimated point of gaze. Therefore, it is possible to automatically extract portions that are visually burdensome.

Further, the above-described visual load estimation method is executed by a computer. By doing so, it is possible to use a computer to estimate a position where the shift in the line of sight increases based on the temporal shift in the amount of movement of the estimated point of gaze. Therefore, it is possible to automatically extract portions that are visually burdensome.

Further, the above-described visual load estimation program may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed.

A visual load estimation device according to an embodiment of the present invention will be described with reference to FIGS. 1 to 11. The visual load amount estimating device according to this embodiment is not limited to being installed in a moving body such as a car, but may be configured with a server device or the like installed in a business office or the like. That is, it is not necessary to analyze in real time, and analysis may be performed after driving.

As shown in FIG. 1, the visual load estimation device 1 includes an input means 2, a visual saliency extraction means 3, a visual load estimation means 4, and an information presentation means 5.

The input means 2 receives an image (moving image) captured by, for example, a camera, and outputs the image as image data. Note that the input moving image is output as image data that is decomposed into time series, such as for each frame, for example. Although a still image may be input as the image input to the input means 2, it is preferable to input it as an image group consisting of a plurality of still images in chronological order.

The image input to the input means 2 may be, for example, an image in which the direction of travel of the vehicle is captured. In other words, the images are images taken continuously of the outside from the moving body. This image may be a so-called panoramic image, which includes a horizontal direction other than the direction of travel, such as 180° or 360°. Moreover, what is input to the input means 2 is not limited to an image captured by a camera, but may also be an image read from a recording medium such as a hard disk drive or a memory card.

The visual saliency extraction means 3 receives image data from the input means 2 and outputs a visual saliency map as visual saliency estimation information to be described later. That is, the visual saliency extraction means 3 acts as a generation unit that generates a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency based on an image taken of the outside from a moving object. Function.

FIG. 2 is a block diagram illustrating the configuration of the visual saliency extraction means 3. As shown in FIG. The visual saliency extraction means 3 according to this embodiment includes an input section 310, a nonlinear mapping section 320, an output section 330, and a storage section 390. The input unit 310 converts the image into intermediate data that can be mapped. The nonlinear mapping unit 320 converts intermediate data into mapping data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping section 320 includes a feature extraction section 321 that extracts features from intermediate data, and an upsampling section 322 that upsamples the data generated by the feature extraction section 321. The storage unit 390 stores image data input from the input means 2, coefficients of a filter to be described later, and the like. This will be explained in detail below.

FIG. 3(a) is a diagram illustrating an image input to the visual saliency extraction means 3, and FIG. 3(b) is a diagram illustrating an image showing the visual saliency distribution estimated for FIG. 3(a). It is a figure which illustrates. The visual saliency extraction means 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual conspicuousness means, for example, how easy it is to stand out or how easy it is to attract attention. Specifically, visual saliency is indicated by probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person viewing the image will turn to the position.

The positions of FIGS. 3(a) and 3(b) correspond to each other. In FIG. 3(a), the higher the visual saliency of a position, the higher the brightness is displayed in FIG. 3(b). An image showing a visual saliency distribution as shown in FIG. 3(b) is an example of a visual saliency map output by the output unit 330. In the example of this figure, visual saliency is visualized using 256 gradations of brightness values. An example of the visual saliency map output by the output unit 330 will be described in detail later.

FIG. 4 is a flowchart illustrating the operation of the visual saliency extraction means 3 according to this embodiment. The flowchart shown in FIG. 4 is part of a visual load estimation method executed by a computer, and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information indicating saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121.

Returning to FIG. 2, each component of the visual saliency extraction means 3 will be explained. In input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the input means 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, and is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data obtained by normalizing the brightness of the obtained image, or data obtained by converting each pixel of the obtained image into a slope of brightness. In input step S110, the input unit 310 may further perform image noise removal, resolution conversion, and the like.

In the nonlinear mapping step S120, the nonlinear mapping section 320 obtains intermediate data from the input section 310. Then, the intermediate data is converted into mapping data in the nonlinear mapping section 320. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, etc., and is preferably a process using a function, a functional, or a neural network.

FIG. 5 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping section 320 includes the feature extraction section 321 and the up-sampling section 322. The feature extraction section 321 performs a feature extraction step S121, and the upsampling section 322 performs an upsampling step S122. Further, in the example shown in the figure, at least one of the feature extraction section 321 and the up-sampling section 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, multiple intermediate layers 323 are coupled.

In particular, it is preferable that the neural network is a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolution layer 324, the input data is convolved by a plurality of filters 325, and the outputs of the plurality of filters 325 are subjected to activation processing.

In the example of FIG. 5, the feature extraction unit 321 includes a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the up-sampling unit 322 includes a neural network including a plurality of intermediate layers 323, and includes an unpooling unit 328 between the plurality of intermediate layers 323. Further, the feature extraction section 321 and the up-sampling section 322 are connected to each other via a second pooling section 327 that performs overlap pooling.

Note that in the example shown in this figure, each intermediate layer 323 consists of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may consist of only one convolutional layer 324. The intermediate layers 323 that are adjacent to each other are separated by one of a first pooling section 326, a second pooling section 327, and an unpooling section 328. Here, when the intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the numbers of filters 325 in those convolutional layers 324 are equal to each other.

In this figure, the intermediate layer 323 labeled "A×B" is composed of B convolutional layers 324, and each convolutional layer 324 means that it includes A convolutional filters for each channel. . Such an intermediate layer 323 will also be referred to as an "A×B intermediate layer" below. For example, a 64×2 hidden layer 323 consists of two convolutional layers 324, meaning that each convolutional layer 324 includes 64 convolutional filters for each channel.

In the example shown in the figure, the feature extraction unit 321 includes a 64×2 intermediate layer 323, a 128×2 intermediate layer 323, a 256×3 intermediate layer 323, and a 512×3 intermediate layer 323 in this order. Further, the up-sample section 322 includes a 512×3 intermediate layer 323, a 256×3 intermediate layer 323, a 128×2 intermediate layer 323, and a 64×2 intermediate layer 323 in this order. Further, the second pooling section 327 connects the two 512×3 intermediate layers 323 to each other. Note that the number of intermediate layers 323 constituting the nonlinear mapping section 320 is not particularly limited, and can be determined depending on, for example, the number of pixels of image data.

Note that this figure is an example of the configuration of the nonlinear mapping section 320, and the nonlinear mapping section 320 may have another configuration. For example, instead of the 64×2 middle layer 323, a 64×1 middle layer 323 may be included. By reducing the number of convolutional layers 324 included in the intermediate layer 323, calculation costs may be further reduced. Further, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. By reducing the number of channels in the intermediate layer 323, calculation costs may be further reduced. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction section 321, it is preferable that the number of filters 325 increases each time the filter passes through the first pooling section 326. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling part 326, and the second intermediate layer 323a is disposed after the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N2. It is composed of layer 324. At this time, it is preferable that N2>N1 holds true. Further, it is more preferable that N2=N1×2 holds true.

Further, in the plurality of intermediate layers 323 included in the up-sampling section 322, it is preferable that the number of filters 325 decreases each time the signal passes through the unpooling section 328. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling section 328, and the fourth intermediate layer 323d is provided after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N4. It is composed of layer 324. At this time, it is preferable that N4<N3 holds true. Further, it is more preferable that N3=N4×2 holds true.

The feature extraction unit 321 extracts image features having multiple levels of abstraction, such as gradients and shapes, from the intermediate data obtained from the input unit 310 as channels of the intermediate layer 323. Figure 6 is 64×2
The configuration of the intermediate layer 323 is illustrated. Processing in the intermediate layer 323 will be described with reference to this figure. In the example shown, the intermediate layer 323 is composed of a first convolutional layer 324a and a second convolutional layer 324b, and each convolutional layer 324 includes 64 filters 325. In the first convolution layer 324a, convolution processing using a filter 325 is performed on each channel of data input to the intermediate layer 323. For example, if the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h ⁰ _i (i=1..3). Further, in the example of this figure, the filter 325 is a 3×3 filter of 64 types, that is, a total of 64×3 types of filters. As a result of the convolution process, 64 results h ⁰ _i,j (i=1..3, j=1..64) are obtained for each channel i.

Next, the activation unit 329 performs activation processing on the outputs of the plurality of filters 325. Specifically, for the corresponding result j of all channels, activation processing is performed on the sum of each corresponding element. Through this activation process, the result of 64 channels h ¹ _i (i=1..64
), that is, the output of the first convolutional layer 324a is obtained as an image feature. Although the activation process is not particularly limited, it is preferable to use at least one of a hyperbolic function, a sigmoid function, and a normalized linear function.

Furthermore, the output data of the first convolutional layer 324a is used as the input data of the second convolutional layer 324b, and the second convolutional layer 324b performs the same processing as the first convolutional layer 324a, resulting in the 64-channel result h ² _i (i=1..64), ie, the output of the second convolutional layer 324b, is obtained as an image feature. The output of the second convolutional layer 324b becomes the output data of this 64×2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but is preferably a 3×3 two-dimensional filter. Further, the coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are held in the storage section 390, and the nonlinear mapping section 320 can read them and use them for processing. Here, the coefficients of the plurality of filters 325 may be determined based on correction information generated and modified using machine learning. For example, the correction information includes coefficients of a plurality of filters 325 as a plurality of correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert intermediate data into mapped data. The storage unit 390 may be included in the visual saliency extraction means 3 or may be provided outside the visual saliency extraction means 3. Furthermore, the nonlinear mapping unit 320 may acquire the correction information from outside via a communication network.

FIGS. 7A and 7B are diagrams showing examples of convolution processing performed by the filter 325, respectively. 7(a) and 7(b) both show examples of 3×3 convolution. The example in FIG. 7(a) is a convolution process using the nearest elements. The example in FIG. 7(b) is a convolution process using adjacent elements having a distance of two or more. Note that convolution processing using adjacent elements having a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using adjacent elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The operation of the 64×2 intermediate layer 323 has been described above. Regarding the operation of other intermediate layers 323 (128×2 intermediate layer 323, 256×3 intermediate layer 323, 512×3 intermediate layer 323, etc.), except for the number of convolutional layers 324 and the number of channels, 64× The operation is the same as that of the second intermediate layer 323. Furthermore, the operation of the intermediate layer 323 in the feature extraction section 321 and the operation of the intermediate layer 323 in the up-sampling section 322 are similar to those described above.

8(a) is a diagram for explaining the processing of the first pooling unit 326, and FIG. 8(b) is a diagram for explaining the processing of the second pooling unit 327. (c) is a diagram for explaining the processing of the unpooling unit 328.

In the feature extraction unit 321, the data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. For example, the first pooling unit 326 performs non-overlapping pooling processing. FIG. 8A shows a process of associating four 2×2 elements 30 with one element 30 for a group of elements included in each channel. The first pooling unit 326 performs this kind of association for all elements 30. Here, the four 2×2 elements 30 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced by a factor of four. Note that as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after being associated is not particularly limited.

The data output from the feature extraction section 321 is input to the up-sampling section 322 via the second pooling section 327. The second pooling unit 327 performs overlap pooling on the output data from the feature extraction unit 321. FIG. 8B shows a process of associating four 2×2 elements 30 with one element 30 while making some of the elements 30 overlap. That is, in repeated associations, some of the four 2×2 elements 30 in one association are also included in the four 2×2 elements 30 in the next association. In the second pooling unit 327 as shown in this figure, the number of elements is not reduced. Note that the number of elements 30 before and after the elements 30 associated in the second pooling unit 327 is not particularly limited.

The method of each process performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, the maximum value of four elements 30 is associated with one element 30 (max pooling), An example of this is average pooling, in which the average value of two elements 30 is used as one element 30.

The data output from the second pooling section 327 is input to the intermediate layer 323 in the up-sampling section 322. Then, the output data from the intermediate layer 323 of the up-sampling section 322 is subjected to unpooling processing for each channel in an unpooling section 328, and then input to the next intermediate layer 323. FIG. 8C shows a process of expanding one element 30 into multiple elements 30. The method of enlarging is not particularly limited, but an example is a method of duplicating one element 30 into four 2×2 elements 30.

The output data of the last intermediate layer 323 of the up-sampling section 322 is outputted as mapping data from the nonlinear mapping section 320 and input to the output section 330. In output step S130, the output unit 330 generates a visual saliency map by performing normalization, resolution conversion, etc. on the data acquired from the nonlinear mapping unit 320, and outputs the map. The visual saliency map is, for example, an image (image data) in which visual saliency is visualized using brightness values, as illustrated in FIG. 3(b). Furthermore, the visual saliency map may be, for example, an image that is color-coded according to visual saliency like a heat map, or it may be a visual saliency map that maps visually salient areas whose visual saliency is higher than a predetermined standard to other positions. may be an image marked with identification. Further, the visual saliency estimation information is not limited to map information shown as an image or the like, but may also be a table listing information indicating visually salient regions.

The visual load estimation means 4 estimates the visual load amount estimation based on the visual saliency map output by the visual saliency extraction means 3. The visual load amount, which is the result estimated by the visual load amount estimating means 4, may be, for example, a scalar amount or a vector amount. Alternatively, it may be single data or multiple time series data. As shown in FIG. 9, the visual recognition load estimation means 4 includes a gaze point estimation means 41, a gaze point movement amount calculation means 42, a base component decomposition means 43, a base component selection means 44, and a power calculation means 45. and a power combining means 46.

The gaze point estimation means 41 estimates gaze point information from the time-series visual saliency map output by the visual saliency extraction means 3. Although the definition of the gaze point information is not particularly limited, it may be, for example, the position (coordinates) where the saliency value is the maximum value. That is, the gaze point estimating means 41 estimates the estimated gaze point to be the position on the image where the visual saliency has the maximum value in the visual saliency map (visual saliency distribution information).

The gaze point movement amount calculation means 42 calculates the time series gaze point movement amount from the time series gaze point information estimated by the gaze point estimation means 41. The gaze point movement amount calculated by the gaze point movement amount calculation means 42 also becomes time series data. Although the calculation method is not particularly limited, it may be, for example, a Euclidean distance between gaze point coordinates that are in a sequential relationship in time series. That is, the gaze point movement amount calculating means 42 calculates the movement amount of the gaze point (estimated gaze point) based on the generated visual saliency map (visual saliency distribution information).

The base component decomposition means 43 decomposes the time-series change in the gaze point movement amount calculated by the gaze point movement amount calculation means 42 into one or more base component groups. Each base component decomposed by the base component decomposition means 43 also becomes time series data corresponding to the amount of movement. Although the decomposition method is not particularly limited, for example, empirical mode decomposition (EMD) is desirable, and in this case, the basis component is an intrinsic mode function (IMF). Alternatively, a Fourier transform family (base component is a sine wave) may be used.

The base component selection means 44 selects one or more base components from the base component group decomposed by the base component decomposition means 43. Note that the selection method is not limited and various methods can be used.

The power calculation means 45 calculates the power for each of the base components selected by the base component selection means 44. Here, the power is also time series data corresponding to the base component. There are no particular limitations on the calculation method, but if empirical mode decomposition is used as the basis component decomposition means, the amplitude component and phase component are calculated using Hilbert transform for the basis component (eigenmode function), and the amplitude component is It is desirable to use it as power.

The power synthesizing means 46 synthesizes the plurality of power components calculated by the power calculating means 45 to calculate a visual load amount. Here, the visual load amount is also time series data corresponding to the power. The combining means is not particularly limited, but may be, for example, simple addition of a plurality of power components. That is, the visual recognition load amount is estimated by the means from the base component decomposition means 43 to the power synthesis means 46 based on the temporal transition of the movement amount of the calculated gaze point (estimated gaze point).

FIG. 10 shows the amount of gaze point movement output by the gaze point movement amount calculation means 42, the base component group obtained by decomposing the gaze point movement amount by the base component decomposition means 43, and the power calculation means 45 calculated from the base component group. An example of the waveform of the visual load amount output by combining the power by the power combining means 46 is shown. The top row of FIG. 10 shows the amount of movement of the gaze point. The 2nd to 16th rows in FIG. 10 are the base component group. The 17th row (lowest row) in FIG. 10 is the visual load amount.

In FIG. 10, it is estimated that the visible load amount is large in the portion where the amplitude of the visible load amount shown at the bottom is large.

The information presenting means 5 presents the amount of visible load outputted by the amount of visible load estimating means 4. Examples of the information presentation means include a display device such as a liquid crystal display, and an audio output means such as a speaker.

Next, the operation (visual load amount estimation method) of the visible load amount estimating device 1 having the above-described configuration will be described with reference to the flowchart of FIG. 11. Further, by configuring this flowchart as a program executed by a computer functioning as the visible load amount estimation device 1, it can be made into a visible load amount estimation program. Further, this visual load estimation program is not limited to being stored in the memory of the visual load estimation device 1, but may be stored in a storage medium such as a memory card or an optical disk.

First, the input means 2 outputs the input image as image data to the visual saliency extraction means 3 (step S210). In this step, the image data input to the input means 2 is decomposed into time series such as image frames and inputted to the visual saliency extraction means 3. Further, image processing such as noise removal and geometric transformation may be performed in this step.

Next, the visual saliency extraction means 3 extracts a visual saliency map (step S220). The visual saliency map as shown in FIG. 3(b) is outputted in time series by the visual saliency extracting means 3 using the method described above.

Next, the visual recognition load amount is estimated from the visual saliency map (step S230). To estimate the visual load amount, the visual load amount estimating means 4 estimates the visual load amount by the method described above based on the visual saliency map output by the visual saliency extraction means 3.

Next, the visual recognition load amount is presented (step S240). In this embodiment, the graph of the change in visual load amount shown in FIG. 10 may be shown, or a corresponding image (frame), traveling position, etc. may be displayed together with the graph.

Then, if reading has been completed for all frames (step S250: Yes), the flowchart is ended, and if reading has not been completed for all frames (step S250: No), the process returns to step S220 and the next frame is read out. Processing is performed.

According to the present embodiment, the visual recognition load amount estimating device 1 estimates the level of visual saliency based on the images continuously captured outside the vehicle by the visual saliency extracting means 3, and the visual saliency A map is generated for each image, and the amount of movement of the gaze point is calculated based on the visual saliency map generated by the visual load estimation means 4. Then, the visual load amount estimating means 4 estimates the visual load amount based on the temporal transition of the calculated estimated movement amount of the gaze point. By doing this, it is possible to estimate a position where the change in the line of sight is large based on the change in the amount of movement of the gaze point over time. Therefore, it is possible to automatically extract portions that are visually burdensome.

Further, the visual load amount estimating means 4 decomposes the estimated movement amount of the gaze point into base components, and calculates the visual load amount based on the decomposed base components. By doing so, it becomes possible to use an algorithm suitable for analyzing the amount of movement of the gaze point, such as empirical mode decomposition.

Further, the visual load estimation means 4 calculates the amount of movement by estimating the point of gaze to be the position on the image where the visual saliency becomes the maximum value in the visual saliency map. By doing so, the amount of movement can be estimated based on the position that is estimated to be most visible.

The visual saliency extraction means 3 also includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and a saliency that shows a saliency distribution based on the mapped data. The nonlinear mapping section 320 includes a feature extraction section 321 that extracts features from intermediate data, and an up-sampling section that upsamples the data generated by the feature extraction section 321. A sample section 322 is provided. By doing so, visual saliency can be estimated with low calculation cost.

It also includes information presentation means 5 for presenting the estimation results of the visual load amount estimating means 4. By doing this, for example, it becomes possible to notify that a point with a large visual load is approaching. Specifically, it will be possible to automatically extract from in-vehicle camera images and notify drivers in advance that they are approaching a location where the amount of load is likely to increase visually. Become. Therefore, reductions in driving errors and accidents due to driver distraction can be expected.

In addition, for example, by identifying locations where the amount of visual load is likely to increase from actual landscape images corresponding to any route on a map, we can analyze risks related to the traffic environment (road structure, sign placement, etc.). This can be expected to improve efficiency.

Further, the present invention is not limited to the above embodiments. That is, those skilled in the art can implement various modifications based on conventionally known knowledge without departing from the gist of the present invention. Such modifications are of course included in the scope of the present invention as long as the visual load amount estimating device of the present invention is provided.

1 Visual recognition load estimation device 2 Input means 3 Visual saliency extraction means (generation unit)
4 Visual load estimation means (calculation section, estimation section)
5 Information presentation means (presentation part)

Claims (8)

a generation unit that generates visual saliency distribution information for each image, which is obtained by estimating the level of visual saliency based on images continuously captured outside from a moving object;
a calculation unit that calculates a movement amount of the estimated gaze point based on the generated visual saliency distribution information;
an estimation unit that estimates a visual load amount based on a temporal change in the calculated movement amount of the estimated gaze point;
A visual load estimation device comprising: The calculation unit calculates the movement amount by estimating the estimated gaze point to be a position on the image where the visual saliency has a maximum value in the visual saliency distribution information. 1. The visual load estimation device according to 1. 3. The estimation unit decomposes the movement amount of the estimated gaze point into a plurality of base components, and estimates the visual recognition load amount based on the decomposed base components. Visual load estimation device. The generation unit is
an input unit that converts the image into intermediate data that can be mapped;
a nonlinear mapping unit that converts the intermediate data into mapping data;
an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data,
The nonlinear mapping section includes a feature extraction section that extracts features from the intermediate data, and an upsampling section that upsamples the data generated by the feature extraction section.
The visual load amount estimating device according to any one of claims 1 to 3. The visual load amount estimating device according to any one of claims 1 to 4, further comprising a presentation unit that presents the estimation result from the estimation unit. A visible load amount estimation method executed by a visible load amount estimating device that estimates a visible load amount based on an image taken of the outside from a moving body, the method comprising:
a generation step of generating visual saliency distribution information for each of the images, which is obtained by estimating the level of visual saliency based on images continuously captured outside the moving body;
a calculation step of calculating the movement amount of the estimated gaze point based on the generated visual saliency distribution information;
an estimating step of estimating a visual load amount based on a temporal change in the calculated movement amount of the estimated point of gaze;
A method for estimating a visible load amount, comprising: A visual load amount estimating program characterized by causing a computer to execute the visual load amount estimating method according to claim 6 . A computer-readable storage medium storing the visual load estimation program according to claim 7 .

Images