JP2024045532A - Judgment device - Google Patents

Abstract

[Problem] Detecting objects that may be visually overlooked. [Solution] In a determination device 1, a visual saliency extraction means 3 generates a visual saliency map obtained by estimating the level of visual saliency from image data captured of the outside from a moving body such as a vehicle. Meanwhile, an object region detection means 4 is set with a type of object to be detected, and detects objects of the set type from the image data. Then, an oversight determination means 5 performs an oversight determination for the object detected by the object region detection means 4 based on the visual saliency map. [Selected Figure] Figure 1

Description

The present invention relates to a determination device that determines the possibility of oversight based on an image taken of the outside from a moving object.

BACKGROUND ART Conventionally, it has been proposed to give various warnings to a moving object such as a driver of a vehicle. For example, Patent Document 1 describes a road sign visibility determination system that includes a visibility estimation section, a visibility completion time calculation section, and a visibility determination section.

To explain Patent Document 1 in detail, the visibility estimation unit calculates the complexity of the road sign based on the content of the road sign, and estimates the visibility of the road sign according to the calculated complexity of the road sign. do. Next, the visibility completion time calculation unit calculates the distance from the driver to the road sign, and uses the speed of the vehicle, visibility of the road sign, and the calculated distance from the driver to the road sign to Calculate the time required to complete visual recognition of a road sign. Then, the visibility determination unit calculates the gaze time during which the driver continuously gazes at the road sign based on the vehicle position information, the road sign information, and the line of sight information including the direction of the driver's line of sight. , it is determined whether the driver has recognized the content of the road sign based on the gaze time and the visual recognition completion time.

Japanese Patent Application Publication No. 2017-111469

The road sign visibility determination system described in Patent Document 1 targets road signs, and does not take any moving objects such as other vehicles or pedestrians into account. Since there are usually other moving objects (cars, motorbikes, bicycles, pedestrians, etc.) on the road, it is desirable to warn drivers about overlooking these objects. Further, the road sign visibility determination system described in Patent Document 1 needs to detect the driver's line of sight, and therefore, equipment for detecting the line of sight must be installed inside the vehicle.

An example of the problem to be solved by the present invention is to detect an object that may be visually overlooked.

In order to solve the above problems, we have developed a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on images taken of the outside from a moving object, and the type of object to be detected. a setting unit that sets an object of a set type from the image; and an object detection unit that detects a set type of object from the image; and an object detection unit that detects a set type of object from the image; The present invention is characterized by comprising a determination unit that determines the possibility of oversight based on a value obtained by dividing the sum of the values of the characteristics by the area of the region.

The invention described in claim 7 is characterized by comprising a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image of the outside captured from a moving body, a setting unit that sets the type of object to be detected, an object detection unit that detects objects of the type set in the setting unit in an area including the imaging range of the image, and a determination unit that performs a probability of overlooking determination for an object detected by the object detection unit based on a value obtained by dividing the sum of the visual saliency values within the object's area in the visual saliency distribution information by the area of the area.

The invention according to claim 8 is a determination method that is executed by a determination device that performs oversight possibility determination based on an image taken of the outside from a moving body, the determination method being performed by estimating the level of visual saliency from the image. a generation step of generating the obtained visual saliency distribution information; a setting step of setting the type of object to be detected; an object detection step of detecting the set type of object from the image; and the object detection step. A determination step of determining the possibility of overlooking the detected object based on a value obtained by dividing the sum of the visual saliency values in the area of the object in the visual saliency distribution information by the area of the area; It is characterized by including.

The invention according to claim 9 is characterized in that the determination method according to claim 8 is executed by a computer.

The invention described in claim 10 is characterized in that the determination program described in claim 9 is stored.

The invention described in claim 11 is a determination method executed by a determination device that performs an oversight probability determination based on an image captured of the outside from a moving body, and is characterized by including a generation process for generating visual saliency distribution information obtained by estimating the level of visual saliency from the image, a setting process for setting the type of object to be detected, an object detection process for detecting objects of the type set in the setting process in an area including the imaging range of the image, and a determination process for performing an oversight probability determination for the object detected in the object detection process based on a value obtained by dividing the sum of the visual saliency values within the object area in the visual saliency distribution information by the area of the area.

The invention according to claim 12 is characterized in that the determination method according to claim 11 is executed by a computer.

The invention described in claim 13 is characterized in that the judgment program described in claim 12 is stored.

FIG. 2 is a functional configuration diagram of a determination device according to an embodiment of the present invention. 2 is a block diagram illustrating a configuration of a visual saliency extraction unit shown in FIG. 1 . (a) is a diagram illustrating an image input to the determination device, and (b) is a diagram illustrating a visual saliency map estimated for (a). 2 is a flow chart illustrating a processing method of the visual saliency extraction means shown in FIG. 1 . FIG. 3 is a diagram illustrating in detail the configuration of a nonlinear mapping section. FIG. 3 is a diagram illustrating a configuration of an intermediate layer. (a) and (b) are diagrams each showing an example of convolution processing performed by a filter. (a) is a diagram for explaining the processing of the first pooling section, (b) is a diagram for explaining the processing of the second pooling section, and (c) is a diagram for explaining the processing of the second pooling section. FIG. 2 is a flowchart of the operation of the determining means shown in FIG. 1. FIG. 5 is a diagram illustrating an example of region information output from an object region detection means. FIG.

The following describes a determination device according to one embodiment of the present invention. In the determination device according to one embodiment of the present invention, a generation unit generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving body of the outside. Meanwhile, a setting unit sets the type of object to be detected, and an object detection unit detects objects of the set type from the image. The determination unit then performs an oversight possibility determination based on the visual saliency distribution information for the object detected by the object detection unit. In this way, it is possible to determine the oversight possibility by combining visual saliency distribution information and object detection. Therefore, it is possible to detect objects that may be visually overlooked. In addition, since the oversight possibility can be determined only from an image captured from the outside of a moving body, it can be determined from images such as a drive recorder, and gaze detection is not required.

Further, the determination unit may determine the possibility of overlooking the object detected by the object detection unit by comparing it with the visual saliency distribution information. By doing so, the possibility of oversight can be determined by comparing the distribution of visual saliency distribution information and the captured image.

The determination unit may also determine that, with regard to objects detected by the object detection unit, objects that do not overlap with areas determined to have high visual saliency are likely to be overlooked. By doing so, it is possible to determine that objects located in parts of the image that do not have high visual saliency are likely to be overlooked.

The generation unit may also include an input unit that converts the image into intermediate data that can be mapped, a nonlinear mapping unit that converts the intermediate data into mapped data, and an output unit that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit may include a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples the data generated by the feature extraction unit. In this way, visual saliency can be estimated with low computational cost.

The device may also include a presentation unit that presents the results of the determination made by the determination unit. In this way, the results of the determination can be presented to the driver to warn them of the possibility of overlooking something.

Further, in a determination device according to another embodiment of the present invention, the generation unit generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image taken of the outside from a moving object. On the other hand, the setting section sets the type of object to be detected, and the object detection section detects the object of the type set in the setting section of the area including the image capturing range. The determination unit then determines the possibility of overlooking the object detected by the object detection unit based on the visual saliency distribution information. By doing so, it is possible to determine the possibility of overlooking by combining visual saliency distribution information and object detection. Therefore, objects that may be visually overlooked can be detected. Furthermore, object detection does not have to be based on images; for example, object detection results from other sensors such as lidar (Light Detection and Ranging) can be used.

Further, in the determination method according to an embodiment of the present invention, visual saliency distribution information is generated by estimating the level of visual saliency based on an image taken of the outside from a moving body in the generation step. On the other hand, in the setting step, the type of object to be detected is set, and in the object detection step, the set type of object is detected from the image. In the determination step, the possibility of overlooking the object detected in the object detection step is determined based on the visual saliency distribution information. By doing so, it is possible to determine the possibility of overlooking by combining visual saliency distribution information and object detection. Therefore, objects that may be visually overlooked can be detected. Furthermore, since the possibility of overlooking can be determined only from an image taken of the outside from a moving object, it is possible to determine the possibility of overlooking, for example, from an image of a drive recorder, etc., and line of sight detection etc. are not required.

Further, the above-described determination method is executed by a computer. By doing so, it is possible to use a computer to determine the possibility of overlooking by combining visual saliency distribution information and object detection. Therefore, objects that may be visually overlooked can be detected.

The above-mentioned determination program may also be stored on a computer-readable storage medium. In this way, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades, etc., can be easily performed.

In addition, in a determination method according to another embodiment of the present invention, in a generation process, visual saliency distribution information is generated by estimating the level of visual saliency based on an image of the outside captured from a moving body. Meanwhile, in a setting process, the type of object to be detected is set, and in an object detection process, objects of the type set in a setting section of an area including the imaging range of the image are detected. Then, in a determination process, a likelihood of overlooking is determined for the object detected in the object detection process based on the visual saliency distribution information. In this way, it is possible to determine the likelihood of overlooking by combining visual saliency distribution information and object detection. Therefore, it is possible to detect objects that may be visually overlooked. In addition, object detection does not have to be based on images, and object detection results from other sensors such as lidar, for example, can be used.

The above-mentioned determination method is also executed by a computer. In this way, the computer can be used to determine the likelihood of overlooking by combining visual saliency distribution information and object detection. Therefore, it is possible to detect objects that may be visually overlooked.

Further, the above-described determination program may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed.

A determination device according to one embodiment of the present invention will be described with reference to Figs. 1 to 10. The determination device according to this embodiment is installed in a mobile body such as an automobile. However, it is not limited to mounting all of the components shown in Fig. 1 on the mobile body. As long as at least the information presentation means 6 described below is installed on the mobile body, the other means may be configured as, for example, a server device, etc., and communication may be performed between the server device and the mobile body.

As shown in FIG. 1, the determination device 1 includes an input unit 2, a visual saliency extraction unit 3, an object region detection unit 4, an oversight determination unit 5, and an information presentation unit 6.

The input means 2 receives an image (still image or video image) captured by, for example, a camera, and outputs the image as image data. If the input image is a video image, it outputs the image data broken down into a time series, for example, for each frame. The image input to the input means 2 may be, for example, an image captured in the direction of travel of the vehicle, but it may also be an image that includes a horizontal direction other than the travel direction, such as 180° or 360°, such as a so-called panoramic image. Furthermore, the images input to the input means 2 are not limited to images captured by a camera, and may also be images read from a recording medium such as a hard disk drive or memory card.

The visual saliency extraction means 3 receives image data from the input means 2 and outputs a visual saliency map as visual saliency estimation information described below. In other words, the visual saliency extraction means 3 functions as a generation unit that generates a visual saliency map (visual saliency distribution information) obtained by estimating the level of visual saliency based on an image captured of the outside from a moving body.

Figure 2 is a block diagram illustrating the configuration of the visual saliency extraction means 3. The visual saliency extraction means 3 according to this embodiment includes an input unit 310, a nonlinear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts an image into intermediate data that can be subjected to mapping processing. The nonlinear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information indicating a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds image data input from the input unit 2 and coefficients of a filter, which will be described later. This will be explained in detail below.

Figure 3(a) is a diagram illustrating an example of an image input to the visual saliency extraction means 3, and Figure 3(b) is a diagram illustrating an example of an image showing a visual saliency distribution estimated for Figure 3(a). The visual saliency extraction means 3 according to this embodiment is a device that estimates the visual saliency of each part in an image. Visual saliency means, for example, how easily something stands out or how easily it attracts attention. More specifically, visual saliency is expressed as a probability or the like. Here, the magnitude of the probability corresponds, for example, to the probability that the gaze of a person viewing the image will be directed to that position.

Figure 3(a) and Figure 3(b) correspond to each other in terms of position. In Figure 3(a), the higher the visual saliency is at a position, the higher the luminance is displayed in Figure 3(b). The image showing the visual saliency distribution as in Figure 3(b) is an example of a visual saliency map output by the output unit 330. In this example, visual saliency is visualized with 256 gradations of luminance values. An example of a visual saliency map output by the output unit 330 will be described in detail later.

FIG. 4 is a flowchart illustrating the operation of the visual saliency extraction means 3 according to this embodiment. The flowchart shown in FIG. 4 is part of a determination method executed by a computer, and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information indicating saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 that extracts features from intermediate data, and an upsampling step S122 that upsamples the data generated in the feature extraction step S121.

Returning to FIG. 2, each component of the visual saliency extraction means 3 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the input means 2. The input unit 310 then converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, and is, for example, a high-dimensional tensor. The intermediate data is, for example, data in which the luminance of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a luminance gradient. In the input step S110, the input unit 310 may further perform noise removal and resolution conversion of the image.

In the nonlinear mapping step S120, the nonlinear mapping section 320 obtains intermediate data from the input section 310. Then, the intermediate data is converted into mapping data in the nonlinear mapping section 320. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process performed on the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters, etc., and is preferably a process using a function, a functional, or a neural network.

Figure 5 is a diagram illustrating in detail the configuration of the nonlinear mapping unit 320, and Figure 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsampling step S122 is performed in the upsampling unit 322. In the example shown in this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including multiple intermediate layers 323. In the neural network, multiple intermediate layers 323 are connected.

In particular, it is preferable that the neural network is a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolution layer 324, the input data is convolved by a plurality of filters 325, and the outputs of the plurality of filters 325 are subjected to activation processing.

In the example of FIG. 5, the feature extraction unit 321 is configured to include a neural network including multiple intermediate layers 323, and includes a first pooling unit 326 between the multiple intermediate layers 323. The upsampling unit 322 is configured to include a neural network including multiple intermediate layers 323, and includes an unpooling unit 328 between the multiple intermediate layers 323. Furthermore, the feature extraction unit 321 and the upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

Note that in the example shown in this figure, each intermediate layer 323 consists of two or more convolutional layers 324. However, at least some of the intermediate layers 323 may consist of only one convolutional layer 324. The intermediate layers 323 that are adjacent to each other are separated by one of a first pooling section 326, a second pooling section 327, and an unpooling section 328. Here, when the intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the numbers of filters 325 in those convolutional layers 324 are equal to each other.

In this figure, the intermediate layer 323 labeled "A×B" is composed of B convolutional layers 324, and each convolutional layer 324 means that it includes A convolutional filters for each channel. . Such an intermediate layer 323 will also be referred to as an "A×B intermediate layer" below. For example, a 64×2 hidden layer 323 consists of two convolutional layers 324, meaning that each convolutional layer 324 includes 64 convolutional filters for each channel.

In the example shown in the figure, the feature extraction unit 321 includes a 64×2 intermediate layer 323, a 128×2 intermediate layer 323, a 256×3 intermediate layer 323, and a 512×3 intermediate layer 323 in this order. Further, the up-sample section 322 includes a 512×3 intermediate layer 323, a 256×3 intermediate layer 323, a 128×2 intermediate layer 323, and a 64×2 intermediate layer 323 in this order. Further, the second pooling section 327 connects the two 512×3 intermediate layers 323 to each other. Note that the number of intermediate layers 323 constituting the nonlinear mapping section 320 is not particularly limited, and can be determined depending on, for example, the number of pixels of image data.

Note that this diagram is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have other configurations. For example, a 64×1 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of convolutional layers 324 included in the intermediate layer 323 may further reduce the computational cost. Also, for example, a 32×2 intermediate layer 323 may be included instead of the 64×2 intermediate layer 323. Reducing the number of channels in the intermediate layer 323 may further reduce the computational cost. Furthermore, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction section 321, it is preferable that the number of filters 325 increases each time the filter passes through the first pooling section 326. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling part 326, and the second intermediate layer 323a is disposed after the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N2. It is composed of layer 324. At this time, it is preferable that N2>N1 holds true. Further, it is more preferable that N2=N1×2 holds true.

Further, in the plurality of intermediate layers 323 included in the up-sampling section 322, it is preferable that the number of filters 325 decreases each time the signal passes through the unpooling section 328. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the unpooling section 328, and the fourth intermediate layer 323d is provided after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N4. It is composed of layer 324. At this time, it is preferable that N4<N3 holds true. Further, it is more preferable that N3=N4×2 holds true.

The feature extraction unit 321 extracts image features having multiple levels of abstraction, such as gradients and shapes, from the intermediate data obtained from the input unit 310 as channels of the intermediate layer 323. FIG. 6 illustrates the configuration of the 64×2 intermediate layer 323. Processing in the intermediate layer 323 will be described with reference to this figure. In the example shown, the intermediate layer 323 is composed of a first convolutional layer 324a and a second convolutional layer 324b, and each convolutional layer 324 includes 64 filters 325. In the first convolution layer 324a, convolution processing using a filter 325 is performed on each channel of data input to the intermediate layer 323. For example, if the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h0i (i=1..3). Further, in the example of this figure, the filter 325 is a 3×3 filter of 64 types, that is, a total of 64×3 types of filters. As a result of the convolution process, 64 results h0i,j (i=1..3, j=1..64) are obtained for each channel i.

Next, an activation process is performed on the outputs of the plurality of filters 325 in an activation unit 329 . Specifically, for the corresponding result j of all channels, activation processing is performed on the sum of each corresponding element. Through this activation process, the 64-channel result h1i (i=1..64), that is, the output of the first convolutional layer 324a, is obtained as an image feature. Although the activation process is not particularly limited, it is preferable to use at least one of a hyperbolic function, a sigmoid function, and a normalized linear function.

Furthermore, the output data of the first convolutional layer 324a is used as the input data of the second convolutional layer 324b, and the second convolutional layer 324b performs the same processing as the first convolutional layer 324a, resulting in 64 channels h2i (i=1..64), that is, the output of the second convolutional layer 324b is obtained as an image feature. The output of the second convolutional layer 324b becomes the output data of this 64×2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but it is preferable that the filter 325 is a two-dimensional filter of 3×3. The coefficients of each filter 325 can be set independently. In this embodiment, the coefficients of each filter 325 are stored in the memory unit 390, and the nonlinear mapping unit 320 can read them and use them for processing. Here, the coefficients of the multiple filters 325 may be determined based on correction information generated and corrected using machine learning. For example, the correction information includes the coefficients of the multiple filters 325 as multiple correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapping data. The memory unit 390 may be provided in the visual saliency extraction means 3, or may be provided outside the visual saliency extraction means 3. The nonlinear mapping unit 320 may also obtain the correction information from the outside via a communication network.

FIGS. 7A and 7B are diagrams showing examples of convolution processing performed by the filter 325, respectively. 7(a) and 7(b) both show examples of 3×3 convolution. The example in FIG. 7(a) is a convolution process using the nearest elements. The example in FIG. 7(b) is a convolution process using adjacent elements having a distance of two or more. Note that convolution processing using adjacent elements having a distance of three or more is also possible. It is preferable that the filter 325 performs convolution processing using adjacent elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The operation of the 64×2 intermediate layer 323 has been described above. Regarding the operation of other intermediate layers 323 (128×2 intermediate layer 323, 256×3 intermediate layer 323, 512×3 intermediate layer 323, etc.), except for the number of convolutional layers 324 and the number of channels, 64× The operation is the same as that of the second intermediate layer 323. Furthermore, the operation of the intermediate layer 323 in the feature extraction section 321 and the operation of the intermediate layer 323 in the up-sampling section 322 are similar to those described above.

Figure 8(a) is a diagram for explaining the processing of the first pooling unit 326, Figure 8(b) is a diagram for explaining the processing of the second pooling unit 327, and Figure 8(c) is a diagram for explaining the processing of the unpooling unit 328.

In the feature extraction unit 321, the data output from the intermediate layer 323 is subjected to pooling processing for each channel in the first pooling unit 326, and then input to the next intermediate layer 323. For example, the first pooling unit 326 performs non-overlapping pooling processing. FIG. 8A shows a process of associating four 2×2 elements 30 with one element 30 for a group of elements included in each channel. The first pooling unit 326 performs this kind of association for all elements 30. Here, the four 2×2 elements 30 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced by a factor of four. Note that as long as the number of elements is reduced in the first pooling unit 326, the number of elements 30 before and after being associated is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is performed on the output data from the feature extraction unit 321. FIG. 8(b) shows a process of matching four 2×2 elements 30 to one element 30 while overlapping some of the elements 30. That is, in repeated matching, some of the four 2×2 elements 30 in a certain matching are also included in the four 2×2 elements 30 in the next matching. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. Note that the number of elements 30 before and after matching in the second pooling unit 327 is not particularly limited.

The method of each process performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, the maximum value of four elements 30 is associated with one element 30 (max pooling), An example of this is average pooling, in which the average value of two elements 30 is used as one element 30.

The data output from the second pooling section 327 is input to the intermediate layer 323 in the up-sampling section 322. Then, the output data from the intermediate layer 323 of the up-sampling section 322 is subjected to unpooling processing for each channel in an unpooling section 328, and then input to the next intermediate layer 323. FIG. 8C shows a process of expanding one element 30 into multiple elements 30. The method of enlarging is not particularly limited, but an example is a method of duplicating one element 30 into four 2×2 elements 30.

The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which visual saliency is visualized by brightness values, as shown in FIG. 3B. In addition, the visual saliency map may be, for example, an image that is colored according to visual saliency, such as a heat map, or an image in which visual saliency areas with visual saliency higher than a predetermined standard are marked so as to be distinguishable from other positions. Furthermore, the visual saliency estimation information is not limited to map information shown as an image or the like, and may be a table or the like that lists information indicating visual saliency areas.

The object area detection means 4 detects (detects) a designated object in the image data based on the image data input from the input means 2 and object designation information that designates the type of object to be detected. The object detection method performed by the object area detection means 4 is not particularly limited, and may be a well-known method such as SSD (Single Shot Multiple Detector), for example. The object designation information can be, for example, a label of the object to be detected. Further, the object designation information may be held as internal information of the object area detection means 4. The label of the object specified in the object specification information includes, for example, a moving object such as a car, a motorcycle, a bicycle, or a pedestrian, but may also include a road marking or a road sign.

The object area detection means 4 outputs the detection result as area information indicating an area (detection area) on the image. The area indicated by this area information does not have to follow the shape of the object, and may be, for example, a rectangular or circular area that includes the object. That is, the object area detection means 4 functions as a setting section that sets the type of object to be detected, and also functions as an object detection section that detects the set type of object from the image.

The oversight determination means 5 compares the visual saliency map output by the visual saliency extraction means 3 with the object region information detected by the object region detection means 4 to determine which objects may be overlooked based on predetermined criteria information, and outputs the determination result as oversight object information. The criteria information is information used as a criterion for determining whether an object has been overlooked, and may be, for example, a specific threshold value (scalar value or vector value). This criteria information may also be held as internal information of the oversight determination means 5. The oversight object information output by the oversight determination means 5 may be, for example, rectangular region information including the oversight object or information indicating pixel coordinates. In other words, the oversight determination means 5 functions as a determination unit that performs an oversight possibility determination based on the visual saliency map (visual saliency distribution information) for the object detected by the object region detection means 4 (object detection unit).

The information presentation means 6 presents the object oversight information output by the oversight determination means 5. The information presentation means can be configured as a display device that displays the object oversight information. It is desirable for this display device to be installed in a position that is easily visible to the driver, such as a head-up display or in the meter.

Next, the operation (determination method) of the determination device 1 having the above-described configuration will be described with reference to the flowchart of FIG. 9. Further, by configuring this flowchart as a program executed by a computer functioning as the determination device 1, it can be made into a determination program. Further, this determination program is not limited to being stored in the memory of the determination device 1, but may be stored in a storage medium such as a memory card or an optical disk.

First, the input means 2 outputs the input image as image data to the visual saliency extraction means 3 and the object area detection means 4 (step S210). In this step, the image data input to the input means 2 is decomposed into time series in the case of a moving image and input to the visual saliency extraction means 3 and the object area detection means 4. Further, image processing such as noise removal and geometric transformation may be performed in this step.

Next, the visual saliency extraction means 3 extracts a visual saliency map (step S220). The visual saliency map is outputted by the visual saliency extracting means 3 as shown in FIG. 3(b) using the method described above.

In parallel with step S220, the object region detection means 4 outputs region information (step S230). The object region detection means 4 detects the region of an object present in the image data input from the input means 2 based on the object designation information and outputs the region information. An example of region information is shown in FIG. 10. FIG. 10 shows the image data shown in FIG. 3(a) to which region information has been added. As shown in FIG. 10, the region information is composed of region sections 41, 42 indicating the region including the detected object, and label names 43, 44 indicating the type, name, etc. of the detected object.

The area portion 41 is indicated by a rectangular frame as illustrated. In FIG. 10, a frame is shown to include a "dog" detected as an object. Similarly, the area portion 42 is also indicated by a rectangular frame as illustrated. In FIG. 10, a frame is shown to include a "car" detected as an object. Note that the shape of the regions 41 and 42 is not limited to a rectangle, but may be a circle, an ellipse, or the like.

The label name 43 is shown adjacent to the area portion 41 as shown. In FIG. 10, "dog" which is the label name of the detected object is shown. The label name 44 is shown adjacent to the area portion 42 as shown. In FIG. 10, "car" is shown as the label name of the detected object.

Returning to the explanation of FIG. 9. Next, an oversight determination is performed (step S240). The oversight determination means 5 uses the visual saliency map outputted by the visual saliency extraction means 3, the area information outputted by the object area detection means 4, and the object included in the area information based on the determination criterion information. Objects that may be visually overlooked are selected and output as oversight determination information.

For example, in the image illustrated in FIG. 3, when "dog" and "car" are detected in FIG. 3(a), the part with high brightness located at the center right in the visual saliency map of FIG. 3(b) If this exceeds the criterion information and is determined to have high visual saliency, then when the areas determined to have high visual saliency are superimposed on FIG. is unlikely to be overlooked. On the other hand, when regions determined to have high visual saliency are superimposed on FIG. 3(a), it is determined that there is a high possibility that the object "car" in the non-overlapping region will be overlooked. That is, the oversight determination means 5 (determination unit) detects an area determined to have high visual saliency in the visual saliency map (visual saliency distribution information) for the object detected by the object area detection unit 4 (object detection unit). It is determined that objects that do not overlap are likely to be overlooked.

The following describes in detail the determination method used by the above-mentioned oversight determination means 5. For each object detected by the object region detection means 4, if the average visual saliency is equal to or less than a threshold value obtained as the determination criterion information, it is determined that there is a possibility of oversight using the following formula (1).

In equation (1), area (obj _i ) is the area of object i, (x, y)∈obj _i is the coordinate of all pixels in object i, and sal (x, y) is the visual field of coordinate (x, y). The saliency value, Th, indicates a threshold value.

Then, the overlooked object is presented (step S250). In this embodiment, for example, the area and label name corresponding to the object that may be overlooked may be displayed in a color or font that stands out more than the area and label names corresponding to other objects, such as red. For example, if the "car" shown in FIG. 10 is determined to be an object that may be overlooked, the area 41 and label name 43 are displayed in red. Alternatively, for example, the area 41 and label name 43 may be displayed only for the object that may be overlooked.

According to this embodiment, the determination device 1 generates a visual saliency map obtained by the visual saliency extraction means 3 estimating the level of visual saliency from image data captured from a moving body such as a vehicle. Meanwhile, the object region detection means 4 is set with a type of object to be detected, and detects objects of the set type from the image data. Then, the oversight determination means 5 performs an oversight determination for the object detected by the object region detection means 4 based on the visual saliency map. In this way, oversight can be determined by combining the visual saliency map and object detection. Therefore, objects that may be visually overlooked can be detected. In addition, since oversight can be determined only from images captured from the outside of a moving body, it can be determined from images captured by, for example, a drive recorder or an in-vehicle camera for ADAS (advanced driving system), and gaze detection, etc. is not required.

The oversight determination means 5 also performs oversight determination by comparing the object detected by the object region detection means 4 with the visual saliency map. In this way, oversight can be determined by comparing the visual saliency map with the captured image.

Moreover, the oversight determining means 5 determines that, among the objects detected by the object area detecting means 4, objects that do not overlap with the area determined to have high visual saliency are likely to be overlooked. By doing so, it can be determined that objects located in parts of the image that are not highly visually salient are likely to be overlooked.

The visual saliency extraction means 3 also includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapped data, and an output unit 330 that generates saliency estimation information indicating a saliency distribution based on the mapped data, and the nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. In this way, visual saliency can be estimated with low calculation costs.

Further, information presentation means 6 for presenting the judgment result of the oversight judgment means 5 is provided. By doing so, it is possible to present the determination result to the driver and warn him of the possibility of oversight.

Note that in the embodiment described above, the object area detecting means 4 does not need to perform object detection based on the image data from the input means 2. For example, results detected by other sensors such as lidar may be used. In this case, the object detection range of the other sensor is preferably the same range as the imaging range of the image data, and needs to include at least the imaging range of the image data.

Further, the present invention is not limited to the above embodiments. That is, those skilled in the art can implement various modifications based on conventionally known knowledge without departing from the gist of the present invention. Of course, such modifications fall within the scope of the present invention as long as the determination device of the present invention is still provided.

1 Determination device 2 Input means 3 Visual saliency extraction means (generation unit)
4. Object region detection means (setting unit, object detection unit)
5. Oversight determination means (determination unit)
6. Information presentation means (presentation unit)

Claims (13)

a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving object of the outside;
A setting unit that sets the type of object to be detected;
an object detection unit that detects an object of a set type from the image;
a determination unit that performs a likelihood of overlooking an object detected by the object detection unit based on a value obtained by dividing a sum of the visual saliency values within a region of the object in the visual saliency distribution information by an area of the region;
A determination device comprising: The determination device according to claim 1, characterized in that the determination unit compares the value with a predetermined threshold value to determine the possibility of overlooking the object. 3. The determination device according to claim 1, wherein the determination unit determines the possibility of overlooking the object detected by the object detection unit by comparing it with the visual saliency distribution information. The determination unit is characterized in that, regarding the objects detected by the object detection unit, it is determined that an object that does not overlap with a region determined to have high visual saliency in the visual saliency distribution information is likely to be overlooked. The determination device according to claim 3. The generation unit is
an input unit for converting the image into intermediate data that can be subjected to mapping processing;
a nonlinear mapping unit that converts the intermediate data into mapping data;
an output unit that generates saliency estimation information indicating a saliency distribution based on the mapping data,
The nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsampling unit that upsamples data generated by the feature extraction unit.
5. The determination device according to claim 1, wherein the determination device is a detection device. The determination device according to any one of claims 1 to 5, further comprising a presentation unit that presents the determination result of the determination unit. a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image captured from a moving object of the outside;
A setting unit for setting a type of object to be detected;
an object detection unit that detects an object of a type set in the setting unit in an area including an imaging range of the image;
a determination unit that performs a likelihood of overlooking an object detected by the object detection unit based on a value obtained by dividing a sum of the visual saliency values within a region of the object in the visual saliency distribution information by an area of the region;
A determination device comprising: A determination method executed by a determination device that determines the possibility of oversight based on an image taken of the outside from a moving object, the method comprising:
a generation step of generating visual saliency distribution information obtained by estimating the level of visual saliency from the image;
a setting step of setting the type of object to be detected;
an object detection step of detecting a set type of object from the image;
For the object detected in the object detection step, a possibility of overlooking is determined based on a value obtained by dividing the sum of the visual saliency values in the area of the object in the visual saliency distribution information by the area of the area. Judgment process;
A determination method characterized by comprising: A determination program that causes a computer to execute the determination method described in claim 8. A computer-readable storage medium storing the determination program according to claim 9. A method for determining a possibility of overlooking an object based on an image captured from a moving object, comprising:
A generating step of generating visual saliency distribution information obtained by estimating the level of visual saliency from the image;
A setting step for setting a type of object to be detected;
an object detection step of detecting an object of the type set in the setting step in an area including an imaging range of the image;
a determination step of determining a possibility of overlooking an object detected in the object detection step based on a value obtained by dividing a sum of the visual saliency values within a region of the object in the visual saliency distribution information by an area of the region;
A method for determining whether or not a A determination program that causes a computer to execute the determination method according to claim 11. A computer-readable storage medium storing the determination program according to claim 12.

Images