JP2021144312A - Determination apparatus - Google Patents

Abstract

To detect a monotonous tendency from a captured image.SOLUTION: A determination apparatus 1 comprises: a landscape image processing unit 3 for acquiring, in chronological order, a visual saliency map obtained by estimating the level of visual saliency in an image based on the image which is obtained by capturing outside from a moving body; and a monotonous determination unit 4 for determining whether the image tends to be monotonous based on the standard deviation and the amount of eye movement that are calculated based on the visual saliency map.SELECTED DRAWING: Figure 1

Description

The present invention relates to a determination device for determining whether an image has a monotonous tendency based on one or a plurality of images obtained by photographing the outside from a moving body.

In general, a phenomenon in which a person falls into a hypnotic state such as drowsiness when driving for a long time on a road such as a highway where monotonous scenery or scenery continues is known as a highway hypnosis phenomenon. In addition, drowsiness may occur on roads where there is no change in scenery or changes in scenery are scarce, or on roads with street lights installed at regular intervals.

As an invention for detecting such a monotonous road, for example, Patent Document 1 describes the monotonicity of a landscape corresponding to an external appearance image based on an image acquisition means for acquiring an external appearance image and an external appearance image acquired by the image acquisition means. A landscape monotonicity arithmetic unit including a monotonicity arithmetic means for calculating the above is described.

Further, Patent Document 2 describes a wakefulness determination system that determines a driver's wakefulness state with high accuracy.

Japanese Patent No. 4550116 Japanese Unexamined Patent Publication No. 2010-128649

In the case of the invention described in Patent Document 1, the feature determination process is performed to classify the landscape, and the monotonicity is obtained based on the number of changes in the landscape. Therefore, even if the landscape is complicated as in the "cityscape", if the time change or the position change is small, the monotonicity is judged to be low. Further, in the case of the invention described in Patent Document 2, the arousal level is determined from the movement of the head in the section determined to be a monotonous section using road information, vehicle information, and inter-vehicle distance information. Since the monotonous section detection is determined based on the road information, the visual information is not taken into consideration. Therefore, even if the signboards and signs are crowded, it may be determined that the road is monotonous if it is a straight single road.

One example of the problem to be solved by the present invention is to detect a monotonous tendency from an captured image.

In order to solve the above problem, the invention according to claim 1 is based on an image obtained by capturing an image of the outside from a moving body, and the visual saliency distribution information obtained by estimating the level of visual saliency in the image. It is characterized by including an acquisition unit for acquiring the image and a determination unit for determining whether the image has a monotonous tendency by using the statistic calculated based on the visual saliency distribution information.

The invention according to claim 6 is a determination method executed by a determination device for determining whether or not the image has a monotonous tendency based on an image obtained by capturing an image of the outside from a moving body, and in the image based on the image. Judgment as to whether the image tends to be monotonous using the acquisition process of acquiring the visual saliency distribution information obtained by estimating the height of the visual saliency and the statistics calculated based on the visual saliency distribution information. It is characterized by including steps.

The invention according to claim 7 is characterized in that the determination method according to claim 6 is executed by a computer.

The invention according to claim 8 is characterized in that the determination program according to claim 7 is stored.

It is a functional block diagram of the determination apparatus which concerns on 1st Example of this invention. It is a block diagram which illustrates the structure of the landscape image processing part shown in FIG. (A) is a diagram exemplifying an image to be input to the determination device, and (b) is a diagram exemplifying a visual saliency map estimated with respect to (a). It is a flowchart which illustrates the processing method of the landscape image processing unit shown in FIG. It is a figure which exemplifies the structure of the nonlinear mapping part in detail. It is a figure which illustrates the structure of the intermediate layer. (A) and (b) are diagrams showing an example of a convolution process performed by a filter, respectively. (A) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the second pooling unit. It is a figure for demonstrating the process of. It is a flowchart of the operation of the monotonic determination unit shown in FIG. It is a flowchart of the operation of the determination apparatus which concerns on 2nd Embodiment of this invention. This is an example of the calculation result of autocorrelation.

Hereinafter, a determination device according to an embodiment of the present invention will be described. In the determination device according to the embodiment of the present invention, the acquisition unit estimates the visual saliency distribution information in the image based on the image obtained by capturing the outside from the moving body, and obtains the visual saliency distribution information. The image is acquired, and the determination unit determines whether the image tends to be monotonous using the statistic calculated based on the visual saliency distribution information. By doing so, it is possible to determine from the captured image whether the tendency is monotonous based on a position that is easy for a human to gaze at. Since the determination is based on a position that is easy for a human (driver) to gaze at, the determination can be made with a tendency close to that of the driver feeling monotonous, and the determination can be made more accurately.

In addition, it is equipped with a standard deviation calculation unit that calculates the standard deviation of the brightness of each pixel in the image obtained as visual saliency distribution information, and the determination unit determines whether the image tends to be monotonous based on the calculated standard deviation. You may. By doing so, it can be determined that there is a monotonous tendency when the positions that are easy to gaze at are concentrated in one image.

In addition, it is provided with an average value calculation unit that calculates the average value of the brightness of each pixel in the image obtained as visual saliency distribution information, and the determination unit determines whether the image tends to be monotonous based on the calculated average value. You may. By doing so, it can be determined that there is a monotonous tendency when the positions that are easy to gaze at are concentrated in one image. Further, since the determination is made based on the average value, the arithmetic processing can be simplified.

In addition, the acquisition unit acquires the visual saliency distribution information in a time series, and the determination unit calculates a statistic from the visual saliency distribution information acquired in the time series, and based on the statistic obtained in the time series. A first determination unit for determining whether the tendency is monotonous and a second determination unit for determining whether the tendency is monotonous based on autocorrelation may be provided. By doing so, it is possible to determine the monotonous tendency due to periodically appearing objects such as street lights appearing during traveling, which is difficult to determine only by the statistic, by autocorrelation.

In addition, it is provided with a line-of-sight movement amount calculation unit that calculates the line-of-sight movement amount between frames based on images obtained in time series as visual saliency distribution information, and the determination unit is the image based on the calculated line-of-sight movement amount. May be determined whether is a monotonous tendency. By doing so, for example, when the amount of movement of the line of sight is small, it can be determined that the tendency is monotonous.

In addition, the acquisition unit generates an input unit that converts an image into intermediate data that can be mapped, a non-linear mapping unit that converts intermediate data into mapping data, and saliency estimation information that shows a saliency distribution based on the mapping data. The non-linear mapping unit may include an output unit, a feature extraction unit that extracts features from intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit. By doing so, the visual prominence can be estimated at a small calculation cost. Moreover, the visual saliency estimated in this way reflects the contextual attention state.

Further, the determination method according to the embodiment of the present invention is a visual saliency distribution obtained by estimating the level of visual saliency in the image based on an image obtained by capturing the outside from a moving body in the acquisition step. Information is acquired, and in the determination step, it is determined whether the image tends to be monotonous using the statistics calculated based on the visual saliency distribution information. By doing so, it is possible to determine from the captured image whether the tendency is monotonous based on a position that is easy for a human to gaze at. Since the determination is based on a position that is easy for a human (driver) to gaze at, the determination can be made with a tendency close to that of the driver feeling monotonous, and the determination accuracy can be further improved.

Further, the above-mentioned determination method is executed by a computer. By doing so, it is possible to determine whether or not the image has a monotonous tendency based on a position that is easy for a human to gaze from an image captured by a computer.

Further, the determination program described above may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a single unit in addition to being incorporated in the device, and version upgrades and the like can be easily performed.

The determination device according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 9. The determination device according to this embodiment is not limited to being installed in a moving body such as an automobile, for example, and may be configured by a server device or the like installed in a business establishment or the like. That is, it is not necessary to analyze in real time, and the analysis may be performed after running or the like.

As shown in FIG. 1, the determination device 1 includes a landscape image acquisition unit 2, a landscape image processing unit 3, and a monotonous determination unit 4.

The landscape image acquisition unit 2 inputs, for example, an image captured by a camera or the like (for example, a moving image), and outputs the image as image data. The input moving image is output as image data decomposed in time series such as for each frame. A still image may be input as an image to be input to the landscape image acquisition unit 2, but it is preferable to input as an image group composed of a plurality of still images in chronological order.

Examples of the image input to the landscape image acquisition unit 2 include an image in which the traveling direction of the vehicle is captured. That is, the image is obtained by continuously capturing the outside from the moving body. This image may be an image including a so-called panoramic image, an image acquired by using a plurality of cameras, or the like, which includes a direction other than the traveling direction such as 180 ° or 360 ° in the horizontal direction. Further, what is input to the landscape image acquisition unit 2 is not limited to the image captured by the camera, but may be an image read from a recording medium such as a hard disk drive or a memory card.

The landscape image processing unit 3 receives image data from the landscape image acquisition unit 2 and outputs a visual saliency map as visual saliency estimation information described later. That is, the landscape image processing unit 3 acquires a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency in the image based on an image obtained by capturing the outside from a moving body. Functions as an acquisition unit.

FIG. 2 is a block diagram illustrating the configuration of the landscape image processing unit 3. The landscape image processing unit 3 according to this embodiment includes an input unit 310, a non-linear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts the image into intermediate data that can be mapped. The non-linear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information showing a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds the image data input from the landscape image acquisition unit 2, the coefficients of the filter described later, and the like. This will be described in detail below.

FIG. 3 (a) is a diagram illustrating an image to be input to the landscape image processing unit 3, and FIG. 3 (b) illustrates an image showing a visual saliency distribution estimated with respect to FIG. 3 (a). It is a figure to do. The landscape image processing unit 3 according to this embodiment is a device that estimates the visual prominence of each part in the image. Visual prominence means, for example, the ease of conspicuousness and the ease of gathering eyes. Specifically, the visual prominence is indicated by a probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person who sees the image points to the position.

The positions of FIGS. 3 (a) and 3 (b) correspond to each other. Then, in FIG. 3A, the higher the visual prominence is, the higher the brightness is displayed in FIG. 3B. The image showing the visual saliency distribution as shown in FIG. 3B is an example of the visual saliency map output by the output unit 330. In the example of this figure, the visual saliency is visualized by the luminance value of 256 gradations. An example of the visual saliency map output by the output unit 330 will be described in detail later.

FIG. 4 is a flowchart illustrating the operation of the landscape image processing unit 3 according to the present embodiment. The flowchart shown in FIG. 4 is part of a determination method performed by a computer and includes an input step S110, a non-linear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information (visual saliency distribution information) showing a saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 for extracting features from the intermediate data and an upsampling step S122 for upsampling the data generated in the feature extraction step S121.

Returning to FIG. 2, each component of the landscape image processing unit 3 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the landscape image acquisition unit 2. Then, the input unit 310 converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, but is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a slope of the brightness. In the input step S110, the input unit 310 may further perform image noise removal, resolution conversion, and the like.

In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. Then, the non-linear mapping unit 320 converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by a parameter or the like, and is preferably a function, a functional, or a neural network process.

FIG. 5 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsample step S122 is performed in the upsample unit 322. Further, in the example of this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, a plurality of intermediate layers 323 are connected.

In particular, the neural network is preferably a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolution layer 324, the input data is convolved by the plurality of filters 325, and the outputs of the plurality of filters 325 are activated.

In the example of FIG. 5, the feature extraction unit 321 is configured to include a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323, and an amplifiering unit 328 is provided between the plurality of intermediate layers 323. Further, the feature extraction unit 321 and the upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

In the example of this figure, each intermediate layer 323 is composed of two or more convolutional layers 324. However, at least a part of the intermediate layer 323 may consist of only one convolution layer 324. The intermediate layers 323 adjacent to each other are separated by one of a first pooling portion 326, a second pooling portion 327, and an amplifiering portion 328. Here, when the intermediate layer 323 includes two or more convolution layers 324, it is preferable that the number of filters 325 in the convolution layers 324 is equal to each other.

In this figure, the intermediate layer 323 marked "AxB" is composed of B convolution layers 324, meaning that each convolution layer 324 includes A convolution filters for each channel. .. Such an intermediate layer 323 will also be referred to as an "A × B intermediate layer" below. For example, the 64 × 2 intermediate layer 323 consists of two convolution layers 324, meaning that each convolution layer 324 includes 64 convolution filters for each channel.

In the example of this figure, the feature extraction unit 321 includes a 64 × 2 intermediate layer 323, a 128 × 2 intermediate layer 323, a 256 × 3 intermediate layer 323, and a 512 × 3 intermediate layer 323 in this order. Further, the upsampling unit 322 includes 512 × 3 intermediate layer 323, 256 × 3 intermediate layer 323, 128 × 2 intermediate layer 323, and 64 × 2 intermediate layer 323 in this order. Further, the second pooling portion 327 connects two 512 × 3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined according to, for example, the number of pixels of the image data.

Note that this figure is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have another configuration. For example, the 64 × 1 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of convolution layers 324 included in the intermediate layer 323, the calculation cost may be further reduced. Further, for example, the 32 × 2 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of channels in the intermediate layer 323, the calculation cost may be further reduced. Further, both the number of convolution layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling portion 326, and the second intermediate layer is behind the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolution layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is a convolution layer in which the number of filters 325 for each channel is N2. It is composed of layers 324. At this time, it is preferable that N2> N1 holds. Further, it is more preferable that N2 = N1 × 2 holds.

Further, in the plurality of intermediate layers 323 included in the upsample unit 322, it is preferable that the number of filters 325 decreases each time the amplifier ring unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the amplifiering portion 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolution layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is a convolution layer in which the number of filters 325 for each channel is N4. It is composed of layers 324. At this time, it is preferable that N4 <N3 holds. Further, it is more preferable that N3 = N4 × 2 holds.

The feature extraction unit 321 extracts image features having a plurality of abstractions such as gradients and shapes from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 6 shows 64 × 2.
The configuration of the intermediate layer 323 is illustrated. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 includes 64 filters 325. In the first convolution layer 324a, each channel of the data input to the intermediate layer 323 is subjected to a convolution process using the filter 325. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the ^{three channels h 0} _{i (i = 1.3).} Further, in the example of this figure, the filter 325 is 64 types of 3 × 3 filters, that is, a total of 64 × 3 types of filters. As a result of the convolution process, 64 results h ⁰ _{i, j} (i = 1..3, j = 1...64) are obtained for each channel i.

Next, the activation process is performed on the output of the plurality of filters 325 in the activation unit 329. Specifically, for the corresponding result j of all channels, the activation process is applied to the sum of the corresponding elements. This activation treatment resulted in 64 channels h ¹ _i (i = 1..64).
), That is, the output of the first convolution layer 324a is obtained as an image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

Further, the output data of the first convolution layer 324a is used as the input data of the second convolution layer 324b, and the second convolution layer 324b performs the same processing as that of the first convolution layer 324a, resulting in 64 channels. ^{The output of 2} _i (i = 1..64), i.e., the second convolution layer 324b, is obtained as an image feature. The output of the second convolution layer 324b becomes the output data of the 64 × 2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but a 3 × 3 two-dimensional filter is preferable. Further, the coefficient of each filter 325 can be set independently. In this embodiment, the coefficient of each filter 325 is stored in the storage unit 390, and the nonlinear mapping unit 320 can read it out and use it for processing. Here, the coefficients of the plurality of filters 325 may be determined based on the correction information generated and corrected by using machine learning. For example, the correction information includes the coefficients of the plurality of filters 325 as a plurality of correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert intermediate data into mapping data. The storage unit 390 may be provided in the landscape image processing unit 3 or may be provided outside the landscape image processing unit 3. Further, the nonlinear mapping unit 320 may acquire the correction information from the outside via the communication network.

7 (a) and 7 (b) are diagrams showing an example of the convolution process performed by the filter 325, respectively. In both FIGS. 7 (a) and 7 (b), an example of 3 × 3 convolution is shown. The example of FIG. 7A is a convolution process using the closest element. The example of FIG. 7B is a convolution process using proximity elements having a distance of two or more. It should be noted that a convolution process using proximity elements having a distance of three or more is also possible. The filter 325 preferably performs a convolution process using proximity elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The operation of the 64 × 2 intermediate layer 323 has been described above. The operation of the other intermediate layers 323 (128 × 2 intermediate layer 323, 256 × 3 intermediate layer 323, 512 × 3 intermediate layer 323, etc.) is also 64 ×, excluding the number of convolution layers 324 and the number of channels. 2 The operation is the same as that of the intermediate layer 323. Further, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsampling unit 322 are the same as described above.

FIG. 8A is a diagram for explaining the processing of the first pooling unit 326, and FIG. 8B is a diagram for explaining the processing of the second pooling unit 327, and FIG. 8B is a diagram for explaining the processing of the second pooling unit 327. (C) is a diagram for explaining the processing of the amplifier ring unit 328.

The data output from the intermediate layer 323 in the feature extraction unit 321 is input to the next intermediate layer 323 after the pooling process is performed for each channel in the first pooling unit 326. In the first pooling unit 326, for example, a non-overlapping pooling process is performed. FIG. 8A shows a process of associating four 2 × 2 elements 30 with one element 30 for an element group included in each channel. In the first pooling unit 326, such a correspondence is made for all the elements 30. Here, the four elements 30 of 2 × 2 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced to a quarter. As long as the number of elements in the first pooling unit 326 is reduced, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is applied to the output data from the feature extraction unit 321. FIG. 8B shows a process of associating four 2 × 2 elements 30 with one element 30 while overlapping some elements 30. That is, in the repeated association, a part of the 2 × 2 four elements 30 in one association is also included in the 2 × 2 four elements 30 in the next association. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. The number of elements 30 before and after being associated with the second pooling unit 327 is not particularly limited.

The method of each processing performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, a mapping (max pooling) in which the maximum value of the four elements 30 is set as one element 30 or 4 An association (average pooling) in which the average value of one element 30 is set as one element 30 can be mentioned.

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. Then, the output data from the intermediate layer 323 of the upsample unit 322 is input to the next intermediate layer 323 after the amplifiering process is performed for each channel in the amplifiering unit 328. FIG. 8C shows a process of expanding one element 30 to a plurality of elements 30. The method of enlargement is not particularly limited, and an example is a method of duplicating one element 30 into four 2 × 2 elements 30.

The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which the visual saliency as illustrated in FIG. 3B is visualized by a luminance value. Further, the visual saliency map may be an image color-coded according to the visual saliency, such as a heat map, or a visual saliency region having a visual saliency higher than a predetermined reference can be set at other positions. May be an image marked so as to be identifiable. Further, the visual prominence estimation information is not limited to the map information shown as an image or the like, and may be a table or the like listing information indicating the visually prominent region.

The monotonous determination unit 4 determines whether the image input to the landscape image acquisition unit 2 has a monotonous tendency based on the visual saliency map acquired by the landscape image processing unit 3. In this embodiment, various statistics are calculated from the visual saliency map, and it is determined whether the tendency is monotonous based on the statistics. That is, the monotonous determination unit 4 functions as a determination unit for determining whether the image has a monotonous tendency by using the statistic calculated based on the visual saliency map (visual saliency distribution information).

FIG. 9 shows a flowchart of the operation of the monotonous determination unit 4. First, the standard deviation of the brightness of each pixel in the image (for example, FIG. 3B) constituting the visual saliency map is calculated (step S11). In this step, first, the average value of the brightness of each image in the images constituting the visual saliency map is calculated. Assuming that the image constituting the visual saliency map is H pixel × V pixel and the luminance value at an arbitrary coordinate (k, m) is _{VVC (k, m)} , the average value is calculated by the following equation (1). Will be done.

From the average value calculated by the equation (1), the standard deviation of the brightness of each image in the image constituting the visual saliency map is calculated. The standard deviation SDEV is calculated by the following equation (2).

With respect to the standard deviation calculated in step S11, it is determined whether or not there are a plurality of output results (step S12). In this step, the image input from the landscape image acquisition unit 2 is a moving image, the visual saliency map is acquired in frame units, and in step S11, it is determined whether the standard deviations for a plurality of frames have been calculated. ..

When there are a plurality of output results (step S12; Yes), the line-of-sight movement amount is calculated (step S13). In this embodiment, the line-of-sight movement amount is obtained from the coordinate distance at which the brightness value in the visual saliency map of each of the frames before and after is the maximum (maximum) in terms of time. The line-of-sight movement amount VSA is calculated by the following equation (3), where the coordinates of the maximum luminance value in the previous frame are (x1, y1) and the coordinates of the maximum luminance value in the subsequent frame are (x2, y2). ..

Then, it is determined whether the tendency is monotonous based on the standard deviation calculated in step S11 and the line-of-sight movement amount calculated in S13 (step S14). In this step, when step S12 is No, a threshold value may be set for the standard deviation calculated in step S11, and it may be determined whether the tendency is monotonous by comparing with the threshold value and comparing with the threshold value. On the other hand, when step S12 is Yes, a threshold value may be set for the line-of-sight movement amount calculated in step S13, and it may be determined whether the tendency is monotonous by comparing with the threshold value.

That is, the monotonous determination unit 4 functions as a standard deviation calculation unit for calculating the standard deviation of the brightness of each pixel in the image obtained as a visual saliency map (visual saliency distribution information), and the visual saliency map (visual saliency map). It functions as a line-of-sight movement amount calculation unit that calculates the line-of-sight movement amount between frames based on images obtained in time series as saliency distribution information).

The processing result (determination result) of the monotonous determination unit 4 is output to the outside of the determination device 1. For example, based on this processing result, the in-vehicle display device or the like may be notified that the vehicle is traveling on a visually monotonous road, or the speaker or the like may notify the vehicle by hearing or by vibration of the seat or the like. May be done. Further, based on the processing result of the monotonous determination unit 4, the user may be guided to a place where he / she can rest. In addition, the determination result may be used as a factor analysis of the hiyari hat in driving.

Further, in the above description, whether the tendency is monotonous is determined by the standard deviation, but it may be determined whether the tendency is monotonous based on the average value of the brightness of the visual saliency map, that is, the result of the equation (1). In the case of the average value of the brightness, a threshold value may be set in the same manner as the standard deviation, and it may be determined whether the tendency is monotonous by comparing with the threshold value. In this case, the monotonous determination unit 4 functions as an average value calculation unit.

Further, by setting the configuration shown in FIG. 1 as, for example, a program executed by a computer, it is possible to obtain a determination program that executes the determination method. Further, this determination program is not limited to being stored in the memory or the like of the determination device 1, and may be stored in a storage medium such as a memory card or an optical disk.

According to this embodiment, the determination device 1 is obtained by the landscape image processing unit 3 estimating the level of visual saliency in the image based on the image obtained by capturing the outside from the moving body. The map is acquired in chronological order, and the monotonic determination unit 4 determines whether the image tends to be monotonous based on the standard deviation calculated based on the visual saliency map, the amount of movement of the line of sight, and the like. By doing so, it is possible to determine from the captured image whether the tendency is monotonous based on a position that is easy for a human to gaze at. Since the determination is based on a position that is easy for a human (driver) to gaze at, the determination can be made with a tendency close to that of the driver feeling monotonous, and the determination can be made more accurately.

Further, the monotonic determination unit 4 calculates the standard deviation of the brightness of each pixel in the image obtained as the visual saliency map, and determines whether the image has a monotonous tendency based on the calculated standard deviation. .. By doing so, it can be determined that there is a monotonous tendency when the positions that are easy to gaze at are concentrated in one image.

Further, the monotonic determination unit 4 calculates the average value of the brightness of each pixel in the image obtained as the visual saliency map, and determines whether the image has a monotonous tendency based on the calculated average value. good. By doing so, it can be determined that there is a monotonous tendency when the positions that are easy to gaze at are concentrated in one image. Further, since the determination is made based on the average value, the arithmetic processing can be simplified.

Further, the monotonic determination unit 4 calculates the amount of line-of-sight movement between frames based on the images obtained in time series as a visual saliency map, and determines whether the tendency is monotonous based on the calculated amount of line-of-sight movement. There is. By doing so, when determining a moving image, for example, when the amount of movement of the line of sight is small, it can be determined that there is a monotonous tendency.

Further, the landscape image processing unit 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapping data, and a saliency distribution that shows a saliency distribution based on the mapping data. The non-linear mapping unit 320 includes an output unit 330 for generating estimation information, and the non-linear mapping unit 320 upsamples the feature extraction unit 321 that extracts features from the intermediate data and the data generated by the feature extraction unit 321. A unit 322 and a unit 322 are provided. By doing so, the visual prominence can be estimated at a small calculation cost. Moreover, the visual saliency estimated in this way reflects the contextual attention state.

Next, the risk information output device according to the second embodiment of the present invention will be described with reference to FIGS. 10 and 11. The same parts as those in the first embodiment described above are designated by the same reference numerals, and the description thereof will be omitted.

In this embodiment, the method of the first embodiment, particularly the case where the detection is omitted when there are a plurality of output results (moving image), can be determined as a monotonous tendency. The block configuration and the like are the same as those in the first embodiment. FIG. 10 shows a flowchart of the operation of the monotonous determination unit 4 according to this embodiment.

In the flowchart of FIG. 10, steps S11 and S13 are the same as those of FIG. In this embodiment, since autocorrelation is used as described later, the target image is a moving image, so step S12 is omitted. In step S14A, the determination content is the same as in step S14. In this embodiment, this step S14A is performed as a primary determination for a monotonous tendency.

Next, when it is determined that the tendency is monotonous as a result of the determination in step S14A (step S15; Yes), the determination result is output to the outside of the determination device 1 in the same manner as in FIG. On the other hand, if it is determined in step S14A that the tendency is not monotonous as a result of the determination (step S15; No), an autocorrelation calculation is performed (step S16).

In this embodiment, the autocorrelation is calculated using the standard deviation (luminance average value) and the line-of-sight movement amount calculated in steps S11 and S13. It is known that the autocorrelation R _(k) is calculated by the following equation (4), where the expected value is E, the average of X is μ, the variance of X is σ ^{2, and the lag is k.} In this embodiment, k is changed within a predetermined range to perform the calculation of Eq. (4), and the largest calculated value is set as the autocorrelation value.

Then, it is determined whether the tendency is monotonous based on the calculated autocorrelation value (step S17). The determination may be made by setting a threshold value for the autocorrelation value as in the first embodiment and comparing it with the threshold value to determine whether the autocorrelation tendency is monotonous. For example, if the autocorrelation value at k = k1 is larger than the threshold value, it means that the same landscape is repeated every k1. If it is determined to be monotonous, the landscape image is classified as a monotonous image. By calculating such an autocorrelation value, it becomes possible to determine a road having a monotonous tendency due to objects arranged periodically such as street lights regularly installed at equal intervals.

FIG. 11 shows an example of the calculation result of autocorrelation. FIG. 11 is a correlogram of the average luminance of the visual saliency map for the running moving image. The vertical axis of FIG. 11 shows the correlation function (autocorrelation value), and the horizontal axis shows the lag. Further, in FIG. 11, the shaded portion has a confidence interval of 95% (dominant level αs = 0.05), the null hypothesis is “no periodicity when lag k”, and the alternative hypothesis is “when lag k”. If "there is periodicity", it is judged that the data in this shaded part has no periodicity because the null hypothesis cannot be rejected, and that the data beyond the shaded part has periodicity regardless of whether it is positive or negative. Will be done.

FIG. 11A is a moving image of tunnel running, which is an example having periodicity. According to FIG. 11A, it can be seen that periodicity is observed at the 10th and 17th pieces. In the case of a tunnel, since the lighting in the tunnel is arranged at regular intervals, it is possible to determine a monotonous tendency due to the lighting or the like. On the other hand, FIG. 11B is a moving image of driving on a general road, which is an example without periodicity. According to FIG. 11B, it can be seen that most of the data is in the confidence interval.

By operating as shown in the flowchart of FIG. 10, it is possible to first determine whether the average, standard deviation, and line-of-sight movement amount are monotonous, and then make a secondary determination from the leaked items from the viewpoint of periodicity.

According to this embodiment, the landscape image processing unit 3 acquires the visual saliency map in time series, and the monotonic determination unit 4 calculates the statistic from the visually saliency map acquired in chronological order and converts it into time series. It functions as a first determination unit for determining whether the tendency is monotonous based on the obtained statistics and a second determination unit for determining whether the tendency is monotonous based on the autocorrelation. By doing so, it is possible to determine the monotonous tendency due to periodically appearing objects such as street lights appearing during traveling, which is difficult to determine only by the statistic, by autocorrelation.

Further, the present invention is not limited to the above examples. That is, those skilled in the art can carry out various modifications according to conventionally known knowledge within a range that does not deviate from the gist of the present invention. Even with such a modification, as long as the determination device of the present invention is still provided, it is, of course, included in the category of the present invention.

1 Judgment device 2 Landscape image acquisition unit 3 Landscape image processing unit (acquisition unit)
4 Monotonic judgment unit (judgment unit, standard deviation calculation unit, average value calculation unit, line-of-sight movement amount calculation unit)

Claims (9)

An acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in the image based on an image of the outside taken from a moving body.
A determination unit that determines whether the image has a monotonous tendency using the statistic calculated based on the visual saliency distribution information, and
A determination device comprising. A standard deviation calculation unit for calculating the standard deviation of the brightness of each pixel in the image obtained as the visual saliency distribution information is provided.
The determination unit determines whether the image tends to be monotonous based on the calculated standard deviation.
The determination device according to claim 1, wherein the determination device is characterized by the above. It is provided with an average value calculation unit that calculates the average value of the brightness of each pixel in the image obtained as the visual saliency distribution information.
The determination unit determines whether the image tends to be monotonous based on the calculated average value.
The determination device according to claim 1, wherein the determination device is characterized by the above. The acquisition unit acquires the visual saliency distribution information in chronological order.
The determination unit calculates the statistic from the visual saliency distribution information acquired in the time series, and determines whether the tendency is monotonous based on the statistic obtained in the time series.
A second determination unit that determines whether the tendency is monotonous based on the autocorrelation of the statistic,
The determination device according to claim 1, further comprising. It is provided with a line-of-sight movement amount calculation unit that calculates the line-of-sight movement amount between frames based on images obtained in time series as the visual saliency distribution information.
The determination unit determines whether the image tends to be monotonous based on the calculated amount of movement of the line of sight.
The determination device according to claim 1, wherein the determination device is characterized by the above. The acquisition unit
An input unit that converts the image into intermediate data that can be mapped,
A non-linear mapping unit that converts the intermediate data into mapping data,
It is provided with an output unit that generates saliency estimation information showing a saliency distribution based on the mapping data.
The nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit.
The determination device according to any one of claims 1 to 5, wherein the determination device is characterized by the above. It is a judgment method executed by a judgment device for judging whether or not the image has a monotonous tendency based on an image obtained by capturing an image of the outside from a moving body.
An acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in the image based on the image, and
A determination step of determining whether the image has a monotonous tendency using the statistic calculated based on the visual saliency distribution information, and
A determination method characterized by including. A determination program, characterized in that the determination method according to claim 7 is executed by a computer. A computer-readable storage medium comprising storing the determination program according to claim 8.

Images