JP2021077249A - Determination device - Google Patents

Abstract

To detect an object that can be missed visually.SOLUTION: A determination device 1 generates a visual saliency map obtained by estimating the level of visual saliency from image data which visual saliency extraction means 3 obtained by taking an image of the outside from such a mobile body as a vehicle. The type of an object to be detected is set in object region detection means 4, and the set type of the object is detected from the image data. Oversight determination means 5 determines an oversight of the object detected by the object region detection means 4 on the basis of the visual saliency map.SELECTED DRAWING: Figure 1

Description

The present invention relates to a determination device that determines the possibility of oversight based on an image obtained by capturing an image of the outside from a moving body.

Conventionally, it has been proposed to give various warnings to, for example, the driver of the own vehicle as a moving body. For example, Patent Document 1 describes a road sign visibility determination system including a visibility estimation unit, a visibility completion time calculation unit, and a visibility determination unit.

Explaining Patent Document 1 in detail, the visibility estimation unit calculates the complexity of the road sign based on the content of the road sign, and estimates the visibility of the road sign according to the calculated complexity of the road sign. To do. Next, the visual recognition completion time calculation unit calculates the distance from the driver to the road sign, and the driver uses the speed of the vehicle, the visibility of the road sign, and the calculated distance from the driver to the road sign. Calculate the visibility completion time required to visually recognize the road sign. Then, the visual recognition determination unit calculates the gaze time at which the driver continuously gazes at the road sign based on the position information of the vehicle, the road sign information, and the line-of-sight information including the direction of the driver's line of sight. , It is determined whether or not the driver recognizes the content of the road sign based on the gaze time and the visual recognition completion time.

Japanese Unexamined Patent Publication No. 2017-11469

The road sign visual determination system described in Patent Document 1 targets road signs, and does not consider moving objects such as other vehicles and pedestrians. Usually, there are other moving objects (cars, motorcycles, bicycles, pedestrians, etc.) on the road, so it is desirable to call attention to these oversights. Further, the road sign visual recognition determination system described in Patent Document 1 needs to detect the line of sight of the driver, and therefore, equipment for detecting the line of sight must be installed in the vehicle.

One example of the problem to be solved by the present invention is to detect an object that may be visually overlooked.

In order to solve the above problems, a generator that generates visual saliency distribution information obtained by estimating the height of visual saliency based on an image of the outside taken from a moving body, and a type of object to be detected. A determination unit for setting the above, an object detection unit for detecting an object of the type set from the image, and a determination for determining the possibility of oversight of an object detected by the object detection unit based on the visual saliency distribution information. It is characterized by having a part and.

The invention according to claim 6 is a generation unit that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image of an external image taken from a moving body, and an object to be detected. The visual saliency distribution of the setting unit for setting the type, the object detection unit for detecting an object of the type set in the setting unit in the area including the imaging range of the image, and the object detected by the object detection unit. It is characterized by including a determination unit that determines the possibility of oversight based on information.

The invention according to claim 7 is a determination method executed by a determination device that determines the possibility of oversight based on an image obtained by capturing an image of the outside from a moving body, and estimates the level of visual prominence based on the image. A generation step of generating visual saliency distribution information obtained in the above process, a setting step of setting the type of an object to be detected, an object detection step of detecting an object of the set type from the image, and the object. The object detected in the detection step is characterized by including a determination step of determining the possibility of oversight based on the visual saliency distribution information.

The invention according to claim 8 is characterized in that the determination method according to claim 7 is executed by a computer.

The invention according to claim 9 is characterized in that the determination program according to claim 8 is stored.

The invention according to claim 10 is a determination method executed by a determination device that determines the possibility of oversight based on an image obtained by capturing an image of the outside from a moving body, and estimates the level of visual prominence based on the image. The generation step of generating the visual saliency distribution information obtained in the above process, the setting step of setting the type of the object to be detected, and the type of object set in the setting step of the region including the imaging range of the image. It is characterized by including an object detection step for detecting an object, and a determination step for determining the possibility of oversight of an object detected in the object detection step based on the visual saliency distribution information.

The invention according to claim 11 is characterized in that the determination method according to claim 10 is executed by a computer.

The invention according to claim 12 is characterized in that the determination program according to claim 11 is stored.

It is a functional block diagram of the determination apparatus which concerns on one Example of this invention. It is a block diagram which illustrates the structure of the visual saliency extraction means shown in FIG. (A) is a diagram exemplifying an image input to the determination device, and (b) is a diagram exemplifying a visual saliency map estimated with respect to (a). It is a flowchart which illustrates the processing method of the visual saliency extraction means shown in FIG. It is a figure which exemplifies the structure of the nonlinear mapping part in detail. It is a figure which illustrates the structure of the intermediate layer. (A) and (b) are diagrams showing an example of a convolution process performed by a filter, respectively. (A) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the second pooling unit. It is a figure for demonstrating the process of. It is a flowchart of the operation of the determination means shown in FIG. It is a figure which illustrated the area information output from the object area detecting means.

Hereinafter, a determination device according to an embodiment of the present invention will be described. The determination device according to the embodiment of the present invention generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image obtained by an image of the outside from a moving body. On the other hand, the type of the object to be detected is set by the setting unit, and the object detection unit detects the set type of object from the image. Then, the determination unit determines the possibility of oversight of the object detected by the object detection unit based on the visual saliency distribution information. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. Further, since the possibility of oversight can be determined only by the image obtained by capturing the outside from the moving body, the determination can be made from the image of, for example, a drive recorder, and the line-of-sight detection or the like becomes unnecessary.

In addition, the determination unit may determine the possibility of oversight of the object detected by the object detection unit by comparing it with the visual saliency distribution information. By doing so, the possibility of oversight can be determined by comparing the distribution of the visual saliency distribution information with the captured image.

Further, the determination unit may determine that the object detected by the object detection unit is likely to be overlooked if the object does not overlap with the region determined to have high visual prominence. By doing so, it can be determined that an object located in a portion of the image where the visual prominence is not high is easily overlooked.

In addition, the generation unit generates an input unit that converts an image into intermediate data that can be mapped, a non-linear mapping unit that converts intermediate data into mapping data, and saliency estimation information that shows a saliency distribution based on the mapping data. The non-linear mapping unit may include an output unit, a feature extraction unit that extracts features from the intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit. By doing so, the visual prominence can be estimated at a small calculation cost.

In addition, a presentation unit that presents the determination result in the determination unit may be provided. By doing so, it is possible to present the determination result to the driver and warn the driver of the possibility of oversight.

In addition, the determination device according to another embodiment of the present invention generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image obtained by an image of the outside from a moving body. On the other hand, the type of the object to be detected is set by the setting unit, and the object detection unit detects the type of object set in the setting unit of the area including the image capturing range of the image. Then, the determination unit determines the possibility of oversight of the object detected by the object detection unit based on the visual saliency distribution information. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. Further, the object detection does not have to be based on an image, and the object detection result of another sensor such as a lidar (LiDAR: Light Detection and Ranging) can be used.

In addition, the determination method according to the embodiment of the present invention generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image obtained by capturing the outside from a moving body in the generation step. On the other hand, the type of the object to be detected is set in the setting process, and the object of the type set from the image is detected in the object detection process. Then, in the determination step, the possibility of oversight is determined based on the visual saliency distribution information for the object detected in the object detection step. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. Further, since the possibility of oversight can be determined only by the image obtained by capturing the outside from the moving body, the determination can be made from the image of, for example, a drive recorder, and the line-of-sight detection or the like becomes unnecessary.

Further, the above-mentioned determination method is executed by a computer. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection using a computer. Therefore, it is possible to detect an object that may be visually overlooked.

Further, the determination program described above may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a single unit in addition to being incorporated in the device, and version upgrades and the like can be easily performed.

In addition, the determination method according to another embodiment of the present invention generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image obtained by capturing the outside from a moving body in the generation step. On the other hand, the type of the object to be detected is set in the setting process, and the object of the type set in the setting unit of the region including the imaging range of the image is detected in the object detection process. Then, in the determination step, the possibility of oversight is determined based on the visual saliency distribution information for the object detected in the object detection step. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. Further, the object detection does not have to be based on an image, and the object detection result of another sensor such as a rider can be used.

Further, the above-mentioned determination method is executed by a computer. By doing so, it is possible to determine the possibility of oversight by combining the visual saliency distribution information and the object detection using a computer. Therefore, it is possible to detect an object that may be visually overlooked.

Further, the determination program described above may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a single unit in addition to being incorporated in the device, and version upgrades and the like can be easily performed.

A determination device according to an embodiment of the present invention will be described with reference to FIGS. 1 to 10. The determination device according to this embodiment is installed in a moving body such as an automobile. However, not all of the configurations shown in FIG. 1 are mounted on the moving body. If at least only the information presenting means 6 described later is installed in the mobile body, the other means may be configured by, for example, a server device or the like, and may be configured to communicate between the server device and the mobile body.

As shown in FIG. 1, the determination device 1 includes an input means 2, a visual prominence extraction means 3, an object area detection means 4, an oversight determination means 5, and an information presenting means 6.

The input means 2 inputs an image (still image or moving image) captured by, for example, a camera, and outputs the image as image data. If the input image is a moving image, it is output as time-series decomposed image data such as for each frame. The image input to the input means 2 includes, for example, an image in which the traveling direction of the vehicle is captured, but even if the image includes a direction other than the traveling direction such as 180 ° or 360 ° in the horizontal direction such as a so-called panoramic image. Good. Further, what is input to the input means 2 is not limited to the image captured by the camera, but may be an image read from a recording medium such as a hard disk drive or a memory card.

The visual saliency extracting means 3 inputs image data from the input means 2 and outputs a visual saliency map as the visual saliency estimation information described later. That is, the visual saliency extracting means 3 serves as a generation unit that generates a visual saliency map (visual saliency distribution information) obtained by estimating the height of the visual saliency based on an image obtained by capturing the outside from a moving body. Function.

FIG. 2 is a block diagram illustrating the configuration of the visual prominence extraction means 3. The visual prominence extraction means 3 according to the present embodiment includes an input unit 310, a non-linear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts the image into intermediate data that can be mapped. The non-linear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information showing a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds the image data input from the input means 2, the coefficient of the filter described later, and the like. This will be described in detail below.

FIG. 3 (a) is a diagram illustrating an image to be input to the visual saliency extracting means 3, and FIG. 3 (b) is an image showing a visual saliency distribution estimated with respect to FIG. 3 (a). It is a figure which exemplifies. The visual saliency extracting means 3 according to the present embodiment is a device for estimating the visual saliency of each part in the image. Visual prominence means, for example, the ease of conspicuousness and the ease of gathering eyes. Specifically, the visual prominence is indicated by a probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person who sees the image points to the position.

The positions of FIGS. 3 (a) and 3 (b) correspond to each other. Then, in FIG. 3A, the higher the visual prominence is, the higher the brightness is displayed in FIG. 3B. The image showing the visual saliency distribution as shown in FIG. 3B is an example of the visual saliency map output by the output unit 330. In the example of this figure, the visual prominence is visualized by the luminance value of 256 gradations. An example of the visual saliency map output by the output unit 330 will be described in detail later.

FIG. 4 is a flowchart illustrating the operation of the visual prominence extraction means 3 according to the present embodiment. The flowchart shown in FIG. 4 is part of a determination method performed by a computer and includes an input step S110, a non-linear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapping data. In output step S130, visual saliency estimation information showing a saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 for extracting features from the intermediate data and an upsampling step S122 for upsampling the data generated in the feature extraction step S121.

Returning to FIG. 2, each component of the visual saliency extracting means 3 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the input means 2. Then, the input unit 310 converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, but is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a slope of the brightness. In the input step S110, the input unit 310 may further perform image noise removal, resolution conversion, and the like.

In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. Then, the non-linear mapping unit 320 converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by a parameter or the like, and is preferably a function, a functional, or a neural network process.

FIG. 5 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 6 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsample step S122 is performed in the upsample unit 322. Further, in the example of this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, a plurality of intermediate layers 323 are connected.

In particular, the neural network is preferably a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolution layer 324, the input data is convolved by the plurality of filters 325, and the outputs of the plurality of filters 325 are activated.

In the example of FIG. 5, the feature extraction unit 321 is configured to include a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323, and an amplifiering unit 328 is provided between the plurality of intermediate layers 323. Further, the feature extraction unit 321 and the upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

In the example of this figure, each intermediate layer 323 is composed of two or more convolutional layers 324. However, at least a part of the intermediate layer 323 may consist of only one convolution layer 324. The intermediate layers 323 adjacent to each other are separated by one of a first pooling portion 326, a second pooling portion 327, and an amplifiering portion 328. Here, when the intermediate layer 323 includes two or more convolution layers 324, it is preferable that the number of filters 325 in the convolution layers 324 is equal to each other.

In this figure, the intermediate layer 323 marked "AxB" is composed of B convolution layers 324, which means that each convolution layer 324 includes A convolution filters for each channel. .. Such an intermediate layer 323 will also be referred to as an "A × B intermediate layer" below. For example, the 64 × 2 intermediate layer 323 consists of two convolution layers 324, meaning that each convolution layer 324 includes 64 convolution filters for each channel.

In the example of this figure, the feature extraction unit 321 includes a 64 × 2 intermediate layer 323, a 128 × 2 intermediate layer 323, a 256 × 3 intermediate layer 323, and a 512 × 3 intermediate layer 323 in this order. Further, the upsampling unit 322 includes 512 × 3 intermediate layer 323, 256 × 3 intermediate layer 323, 128 × 2 intermediate layer 323, and 64 × 2 intermediate layer 323 in this order. Further, the second pooling unit 327 connects two 512 × 3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined according to, for example, the number of pixels of the image data.

Note that this figure is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have another configuration. For example, a 64x1 intermediate layer 323 may be included instead of the 64x2 intermediate layer 323. By reducing the number of convolution layers 324 included in the intermediate layer 323, the calculation cost may be further reduced. Further, for example, a 32 × 2 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of channels in the intermediate layer 323, the calculation cost may be further reduced. Further, both the number of convolution layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling portion 326, and the second intermediate layer is behind the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolution layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is a convolution layer in which the number of filters 325 for each channel is N2. It is composed of layers 324. At this time, it is preferable that N2> N1 holds. Further, it is more preferable that N2 = N1 × 2 holds.

Further, in the plurality of intermediate layers 323 included in the upsample unit 322, it is preferable that the number of filters 325 decreases each time the amplifier ring unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the amplifiering portion 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolution layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is a convolution layer in which the number of filters 325 for each channel is N4. It is composed of layers 324. At this time, it is preferable that N4 <N3 holds. Further, it is more preferable that N3 = N4 × 2 holds.

The feature extraction unit 321 extracts image features having a plurality of abstractions such as gradients and shapes from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 6 shows 64 × 2.
The configuration of the intermediate layer 323 is illustrated. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 includes 64 filters 325. In the first convolution layer 324a, each channel of the data input to the intermediate layer 323 is subjected to a convolution process using the filter 325. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the ^{three channels h 0} _{i (i = 1.3).} Further, in the example of this figure, the filter 325 is 64 types of 3 × 3 filters, that is, a total of 64 × 3 types of filters. As a result of the convolution process, 64 results h ⁰ _{i, j} (i = 1..3, j = 1...64) are obtained for each channel i.

Next, the activation process is performed on the output of the plurality of filters 325 in the activation unit 329. Specifically, for the corresponding result j of all channels, the activation process is applied to the sum of the corresponding elements. This activation treatment resulted in 64 channels h ¹ _i (i = 1..64).
), That is, the output of the first convolution layer 324a is obtained as an image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

Further, the output data of the first convolution layer 324a is used as the input data of the second convolution layer 324b, and the second convolution layer 324b performs the same processing as that of the first convolution layer 324a, resulting in 64 channels. ^{The output of 2} _i (i = 1..64), i.e., the second convolution layer 324b, is obtained as an image feature. The output of the second convolution layer 324b becomes the output data of the 64 × 2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but a 3 × 3 two-dimensional filter is preferable. Further, the coefficient of each filter 325 can be set independently. In this embodiment, the coefficient of each filter 325 is stored in the storage unit 390, and the nonlinear mapping unit 320 can read it out and use it for processing. Here, the coefficients of the plurality of filters 325 may be determined based on the correction information generated and corrected by using machine learning. For example, the correction information includes the coefficients of the plurality of filters 325 as a plurality of correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert intermediate data into mapping data. The storage unit 390 may be provided in the visual saliency extracting means 3, or may be provided outside the visual saliency extracting means 3. Further, the nonlinear mapping unit 320 may acquire the correction information from the outside via the communication network.

7 (a) and 7 (b) are diagrams showing an example of the convolution process performed by the filter 325, respectively. In both FIGS. 7 (a) and 7 (b), an example of 3 × 3 convolution is shown. The example of FIG. 7A is a convolution process using the closest element. The example of FIG. 7B is a convolution process using proximity elements having a distance of two or more. It should be noted that a convolution process using proximity elements having a distance of three or more is also possible. The filter 325 preferably performs a convolution process using proximity elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The operation of the 64 × 2 intermediate layer 323 has been described above. The operation of the other intermediate layers 323 (128 × 2 intermediate layer 323, 256 × 3 intermediate layer 323, 512 × 3 intermediate layer 323, etc.) is also 64 ×, excluding the number of convolution layers 324 and the number of channels. 2 The operation is the same as that of the intermediate layer 323. Further, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsample unit 322 are the same as described above.

FIG. 8A is a diagram for explaining the processing of the first pooling unit 326, and FIG. 8B is a diagram for explaining the processing of the second pooling unit 327, and FIG. 8B is a diagram for explaining the processing of the second pooling unit 327. (C) is a diagram for explaining the processing of the amplifiering unit 328.

The data output from the intermediate layer 323 in the feature extraction unit 321 is input to the next intermediate layer 323 after the pooling process is performed for each channel in the first pooling unit 326. In the first pooling unit 326, for example, a non-overlapping pooling process is performed. FIG. 8A shows a process of associating four 2 × 2 elements 30 with one element 30 for an element group included in each channel. In the first pooling unit 326, such a correspondence is made for all the elements 30. Here, the four elements 30 of 2 × 2 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced to a quarter. As long as the number of elements in the first pooling unit 326 is reduced, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is applied to the output data from the feature extraction unit 321. FIG. 8B shows a process of associating four 2 × 2 elements 30 with one element 30 while overlapping some elements 30. That is, in the repeated association, a part of the 2 × 2 four elements 30 in one association is also included in the 2 × 2 four elements 30 in the next association. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. The number of elements 30 before and after being associated with the second pooling unit 327 is not particularly limited.

The method of each processing performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, a mapping (max pooling) in which the maximum value of the four elements 30 is set as one element 30 or 4 An association (average pooling) in which the average value of one element 30 is set as one element 30 can be mentioned.

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. Then, the output data from the intermediate layer 323 of the upsample unit 322 is input to the next intermediate layer 323 after the amplifiering process is performed for each channel in the amplifiering unit 328. FIG. 8C shows a process of expanding one element 30 to a plurality of elements 30. The method of enlargement is not particularly limited, and an example is a method of duplicating one element 30 into four 2 × 2 elements 30.

The output data of the last intermediate layer 323 of the upsampling unit 322 is output as mapping data from the nonlinear mapping unit 320 and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which the visual saliency as illustrated in FIG. 3B is visualized by a luminance value. Further, the visual saliency map may be an image color-coded according to the visual saliency, such as a heat map, or a visual saliency region having a visual saliency higher than a predetermined reference can be set at other positions. May be an image marked so as to be identifiable. Further, the visual prominence estimation information is not limited to the map information shown as an image or the like, and may be a table or the like listing information indicating the visually prominent region.

The object area detecting means 4 detects (detects) a designated object in the image data based on the image data input from the input means 2 and the object designation information that specifies the type of the object to be detected. The object detection method performed by the object area detection means 4 may be a well-known method such as SSD (Single Shot multiple Detector), and is not particularly limited. The object designation information can be, for example, a label of an object to be detected. Further, the object designation information may be retained as internal information of the object area detecting means 4. The label of the object specified in the object designation information includes, for example, a moving body such as an automobile, a motorcycle, a bicycle, or a pedestrian, but may include a road marking or a road sign.

The object area detecting means 4 outputs the detection result as area information indicating an area (detection area) on the image. The region indicated by this region information does not have to follow the shape of the object, and may be, for example, a rectangular or circular region including the object. That is, the object area detecting means 4 functions as a setting unit for setting the type of the object to be detected, and also functions as an object detecting unit for detecting the type of the object set from the image.

The oversight determination means 5 may be overlooked based on a predetermined determination criterion information by comparing the visual saliency map output by the visual saliency extracting means 3 with the object area information detected by the object area detection means 4. An object with a property is determined, and the determination result is overlooked and output as object information. The determination standard information is information that is used as a reference for determining oversight, and can be, for example, a specific threshold value (scalar value or vector value). Further, this determination standard information may be retained as internal information of the oversight determination means 5. The overlooked object information output by the overlooked determination means 5 can be, for example, rectangular area information including the overlooked object or information indicating pixel coordinates. That is, the oversight determination means 5 functions as a determination unit that determines the possibility of oversight of an object detected by the object area detection means 4 (object detection unit) based on the visual saliency map (visual saliency distribution information).

The information presenting means 6 presents the object oversight information output by the oversight determination means 5. The information presenting means can be configured by a display device that displays overlooked object information. It is desirable that this display device is installed at a position that is easy for the driver to see, such as in a head-up display or a meter.

Next, the operation (determination method) in the determination device 1 having the above-described configuration will be described with reference to the flowchart of FIG. Further, the determination program can be obtained by configuring this flowchart as a program executed by a computer functioning as the determination device 1. Further, this determination program is not limited to being stored in the memory or the like of the determination device 1, and may be stored in a storage medium such as a memory card or an optical disk.

First, the input means 2 outputs the input image as image data to the visual saliency extracting means 3 and the object area detecting means 4 (step S210). In this step, in the case of a moving image, the image data input to the input means 2 is decomposed in time series and input to the visual prominence extracting means 3 and the object area detecting means 4. In addition, image processing such as noise removal and geometric transformation may be performed in this step.

Next, the visual saliency extracting means 3 extracts the visual saliency map (step S220). The visual saliency map outputs the visual saliency map as shown in FIG. 3B by the method described above in the visual saliency extracting means 3.

In parallel with step S220, the object area detecting means 4 outputs the area information (step S230). The area information is output as area information by detecting the area of the object existing in the image data based on the object designation information with respect to the image data input from the input means 2 in the object area detecting means 4. FIG. 10 shows an example of area information. FIG. 10 shows the image data shown in FIG. 3A with region information added. As shown in FIG. 10, the area information is composed of the area portions 41 and 42 indicating the area including the detected object and the label names 43 and 44 indicating the type and name of the detected object. There is.

The region portion 41 is shown by a rectangular frame as shown. In FIG. 10, the frame is shown to include the "dog" detected as an object. Similarly, the region portion 42 is also shown by a rectangular frame as shown. In FIG. 10, the frame is shown to include the "car" detected as an object. The shapes of the regions 41 and 42 are not limited to rectangles, but may be circles, ellipses, or the like.

The label name 43 is shown adjacent to the region portion 41 as shown. In FIG. 10, "dog", which is the label name of the detected object, is shown. The label name 44 is shown adjacent to the region portion 42 as shown. In FIG. 10, the label name of the detected object, "car", is shown.

Returning to the description of FIG. Next, an oversight determination is performed (step S240). The oversight determination is an object included in the area information based on the determination reference information from the visual saliency map output by the visual saliency extracting means 3 and the area information output by the object area detection means 4 in the oversight determination means 5. Select an object that may be visually overlooked from the above and output it as oversight judgment information.

For example, in the image illustrated in FIG. 3, when a "dog" and a "car" are detected in FIG. 3 (a), a portion having a high brightness located in the center on the right side in the visual saliency map of FIG. 3 (b). If it is determined that the visual luminosity is high beyond the judgment criterion information, the "dog" which is an object in the overlapping region when the region determined to have the high visual luminance is overlapped with FIG. 3 (a). Is unlikely to be overlooked. On the other hand, when the regions determined to have high visual prominence are overlapped with FIG. 3A, it is determined that there is a high possibility that the "car", which is an object in the regions that do not overlap, will be overlooked. That is, the oversight determination means 5 (determination unit) determines that the object detected by the object area detection means 4 (object detection unit) has high visual saliency in the visual saliency map (visual saliency distribution information). It is judged that objects that do not overlap with are likely to be overlooked.

The details of the determination method in the oversight determination means 5 described above will be described. For each object detected by the object area detecting means 4, it is determined that there is a possibility of oversight when the average value of the visual saliency is equal to or less than the threshold value obtained as the determination reference information by using the following equation (1).

In equation (1), area (obj _i ) is the area of the object i, (x, y) ∈ obj _i is the coordinates of all pixels in the object i, and sal (x, y) is the visual of the coordinates (x, y). The value of saliency, Th, indicates a threshold.

Then, the overlooked object is presented (step S250). In this embodiment, for example, the area portion or label name corresponding to an object that may be overlooked is displayed in red or the like, and the area portion or label name corresponding to another object is displayed in a color or typeface that is more conspicuous than the label name. Be done. For example, when the "car" shown in FIG. 10 is determined to be an object that may be overlooked, the area portion 41 and the label name 43 are displayed in red. Alternatively, for example, the area portion 41 or the label name 43 may be displayed only for an object that may be overlooked.

According to this embodiment, the determination device 1 generates a visual saliency map obtained by estimating the height of the visual saliency from the image data obtained by the visual saliency extracting means 3 capturing the outside from a moving body such as a vehicle. To do. On the other hand, the type of the object to be detected is set in the object area detecting means 4, and the set type of object is detected from the image data. Then, the oversight determination means 5 performs an oversight determination on the object detected by the object area detection means 4 based on the visual saliency map. By doing so, the oversight can be determined by combining the visual saliency map and the object detection. Therefore, it is possible to detect an object that may be visually overlooked. In addition, since the oversight can be determined only by the image of the outside captured from the moving body, it can be determined from the image captured by the drive recorder, the in-vehicle camera for ADAS (advanced driver assistance system), etc. It becomes.

Further, the oversight determination means 5 makes an oversight determination by comparing the object detected by the object area detection means 4 with the visual saliency map. By doing so, the oversight can be determined by comparing the visual saliency map with the captured image.

Further, the oversight determination means 5 determines that there is a high possibility that the object detected by the object area detection means 4 will be overlooked if the object does not overlap with the region determined to have high visual prominence. By doing so, it can be determined that an object located in a portion of the image where the visual prominence is not high is easily overlooked.

Further, the visual saliency extracting means 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapping data, and a remarkableness that shows a saliency distribution based on the mapping data. The non-linear mapping unit 320 includes an output unit 330 for generating sex estimation information, and the non-linear mapping unit 320 up-samples the feature extraction unit 321 that extracts features from the intermediate data and the data generated by the feature extraction unit 321. A sample unit 322 and a sample unit 322 are provided. By doing so, the visual prominence can be estimated at a small calculation cost.

Further, the information presenting means 6 for presenting the determination result in the oversight determination means 5 is provided. By doing so, it is possible to present the determination result to the driver and warn the driver of the possibility of oversight.

In the above-described embodiment, the object area detecting means 4 does not have to detect the object based on the image data from the input means 2. For example, the result detected by another sensor such as a rider may be used. In this case, the object detection range of the other sensor is preferably the same range as the image data imaging range, and must include at least the image data imaging range.

Further, the present invention is not limited to the above examples. That is, those skilled in the art can carry out various modifications according to conventionally known knowledge within a range that does not deviate from the gist of the present invention. Even with such a modification, as long as the determination device of the present invention is still provided, it is, of course, included in the category of the present invention.

1 Judgment device 2 Input means 3 Visual prominence extraction means (generation unit)
4 Object area detection means (setting unit, object detection unit)
5 Oversight judgment means (judgment unit)
6 Information presentation means (presentation section)

Claims (12)

A generator that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image of the outside taken from a moving body, and
A setting unit that sets the type of object to be detected, and
An object detection unit that detects an object of the set type from the image,
A determination unit that determines the possibility of oversight of an object detected by the object detection unit based on the visual saliency distribution information.
A determination device comprising. The determination device according to claim 1, wherein the determination unit determines the possibility of oversight of an object detected by the object detection unit in comparison with the visual saliency distribution information. The determination unit is characterized in that, regarding an object detected by the object detection unit, it is highly likely that an object that does not overlap with a region determined to have high visual saliency in the visual saliency distribution information is likely to be overlooked. The determination device according to claim 2. The generator
An input unit that converts the image into intermediate data that can be mapped,
A non-linear mapping unit that converts the intermediate data into mapping data,
It includes an output unit that generates saliency estimation information showing a saliency distribution based on the mapping data.
The nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit.
The determination device according to any one of claims 1 to 3, wherein the determination device is characterized by the above. The determination device according to any one of claims 1 to 4, further comprising a presentation unit that presents a determination result in the determination unit. A generator that generates visual saliency distribution information obtained by estimating the level of visual saliency based on an image of the outside taken from a moving body, and
A setting unit that sets the type of object to be detected, and
An object detection unit that detects an object of the type set in the setting unit in the area including the imaging range of the image, and an object detection unit.
A determination unit that determines the possibility of oversight of an object detected by the object detection unit based on the visual saliency distribution information.
A determination device comprising. It is a judgment method executed by a judgment device that judges the possibility of oversight based on an image of the outside taken from a moving body.
A generation step of generating visual saliency distribution information obtained by estimating the height of visual saliency from the image, and
A setting process for setting the type of object to be detected, and
An object detection process that detects an object of the type set from the image, and
A determination step of determining the possibility of oversight of an object detected in the object detection step based on the visual saliency distribution information, and a determination step.
A determination method characterized by including. A determination program, characterized in that the determination method according to claim 7 is executed by a computer. A computer-readable storage medium comprising storing the determination program according to claim 8. It is a judgment method executed by a judgment device that judges the possibility of oversight based on an image of the outside taken from a moving body.
A generation step of generating visual saliency distribution information obtained by estimating the height of visual saliency from the image, and
A setting process for setting the type of object to be detected, and
An object detection step of detecting an object of the type set in the setting step of the region including the imaging range of the image, and an object detection step.
A determination step of determining the possibility of oversight of an object detected in the object detection step based on the visual saliency distribution information, and a determination step.
A determination device comprising. A determination program, characterized in that the determination method according to claim 10 is executed by a computer. A computer-readable storage medium comprising storing the determination program according to claim 11.

Images