JP2021157247A - Determination device - Google Patents

Abstract

To determine the possibility of occurrence of a safety problem such as an accident or a near-miss caused by a psychological burden.SOLUTION: In a determination device 1, a visual saliency calculation part 3, on the basis of an image of the outside photographed from a moving body, acquires a visual saliency map obtained by estimating the high/low level of visual saliency in the image, and a visual line coordinate setting part 4 sets a coordinate of an ideal visual line. Then, a vector error calculation part 5 calculates a visual attention concentration degree Ps in the image on the basis of the visual saliency map and the ideal visual line. A determination part 6 determines the possibility of occurrence of a safety problem such as a near-miss during running of the moving body on the basis of a temporal change amount in the visual attention concentration degree Ps.SELECTED DRAWING: Figure 1

Description

The present invention relates to a determination device that performs a predetermined determination process based on an image obtained by capturing an image of the outside from a moving body.

For example, in a drive recorder, the occurrence of an accident or a hilarious hat is detected based on the acceleration of the vehicle, and images before and after the occurrence are recorded.

Patent Document 1 describes an invention of a drive recorder device capable of detecting an event such as an accident or a hilarious hat with high accuracy and identifying the cause of the occurrence of the event. In the invention described in Patent Document 1, when the amount of change in acceleration occurs on the acceleration side, is the amount of change in accelerator opening of the control unit 21 equal to or less than a certain value determined to be operated by the driver's judgment? Whether or not it is determined, and if the amount of change in the accelerator opening is equal to or less than a certain value, it is determined that the event is due to external energy. On the other hand, when the amount of change in the accelerator opening exceeds a certain value, the control unit 21 determines that the event is determined by the driver. In this case, the control unit 21 records the event information, the data collected by various sensors, and the video information in the recording unit 26 as an example of the hearing hat.

Japanese Unexamined Patent Publication No. 2012-164131

If it is determined that an accident or a hiyari hat is based only on acceleration, it is not possible to detect a hiyari hat that is not accompanied by sudden braking, for example, due to a sudden intrusion from the side of a vehicle, or a hiyari hat in a loose state.

In the invention described in Patent Document 1, since the opening degree of the accelerator is also a criterion for determination, it is possible to detect a hilarious hat caused by a sudden intrusion of a vehicle from the side without sudden braking, or a hilarious hat in a loose state. Can't.

As an example of the problem to be solved by the present invention, it is possible to determine the suspicion that a safety problem such as an accident or a hiyari hat caused by a psychological burden has occurred.

In order to solve the above problem, the invention according to claim 1 is based on an image obtained by capturing an image of the outside from a moving body, and the visual saliency distribution information obtained by estimating the level of visual saliency in the image. The degree of concentration of visual attention in the image based on the acquisition unit for acquiring the image, the line-of-sight position setting unit for setting the reference line-of-sight position in the image according to a predetermined rule, and the visual saliency distribution information and the line-of-sight position. Judgment that there is a suspicion that a safety problem has occurred while the moving object is running, based on the visual attention concentration calculation unit that calculates It is characterized by having a part and.

The invention according to claim 5 is a determination method executed by a determination device that performs a predetermined determination process based on an image obtained by capturing an image of the outside from a moving body, and is visually remarkable in the image based on the image. An acquisition step of acquiring visual saliency distribution information obtained by estimating the level of sexuality, a line-of-sight position setting step of setting a reference line-of-sight position in the image according to a predetermined rule, the visual saliency distribution information and the above. The moving body is traveling based on the visual attention concentration calculation step of calculating the concentration of visual attention in the image based on the line-of-sight position and the temporal change amount of the concentration of visual attention. It is characterized by including a determination step of determining that there is a suspicion that a safety problem has occurred.

The invention according to claim 6 is characterized in that the determination method according to claim 5 is executed by a computer.

The invention according to claim 7 is characterized in that the determination program according to claim 6 is stored.

It is a schematic block diagram of the system which has the determination apparatus which concerns on 1st Embodiment of this invention. It is a functional block diagram of the determination device shown in FIG. It is a block diagram which illustrates the structure of the visual saliency calculation part shown in FIG. (A) is a diagram exemplifying an image to be input to the determination device, and (b) is a diagram exemplifying a visual saliency map estimated with respect to (a). It is a flowchart which illustrates the processing method of the visual saliency calculation part shown in FIG. It is a figure which exemplifies the structure of the nonlinear mapping part in detail. It is a figure which illustrates the structure of the intermediate layer. (A) and (b) are diagrams showing an example of a convolution process performed by a filter, respectively. (A) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the second pooling unit. It is a figure for demonstrating the process of. It is explanatory drawing of a vector error. This is an example of an image input to the image input unit shown in FIG. 1 and a visual saliency map acquired from the image. It is a graph which showed the example of the temporal change of the degree of visual attention concentration. It is a flowchart of the operation of the determination device shown in FIG. This is a screen example of the output of the judgment device.

Hereinafter, a determination device according to an embodiment of the present invention will be described. In the determination device according to the embodiment of the present invention, the acquisition unit estimates the visual saliency distribution information in the image based on the image obtained by capturing the outside from the moving body, and obtains the visual saliency distribution information. Acquired, the line-of-sight position setting unit sets the reference line-of-sight position in the image according to a predetermined rule. Then, the visual attention concentration calculation unit calculates the concentration of visual attention in the image based on the visual saliency distribution information and the line-of-sight position. Then, the determination unit determines that there is a suspicion that a safety problem has occurred while the moving body is traveling, based on the amount of change in the degree of concentration of visual attention over time. By doing so, since the visual saliency distribution information is used, it is based on the amount of temporal change in the contextual attention state that the line of sight tends to be unconsciously focused on an object such as a sign or a pedestrian included in the image. It can be determined that a safety problem has occurred. Therefore, it is possible to determine the suspicion that a safety problem such as an accident or a hiyari hat caused by a psychological burden is caused only by the image.

In addition, the visual attention concentration calculation unit concentrates visual attention based on the value of each pixel constituting the visual saliency distribution information and the vector error between the position of each pixel and the coordinate position of the reference line-of-sight position. The degree may be calculated. By doing so, a value corresponding to the difference between the position where the visual prominence is high and the reference line-of-sight position is calculated as the degree of concentration of visual attention. Therefore, for example, the value of the degree of concentration of visual attention can be changed according to the distance between the position where the visual prominence is high and the reference line-of-sight position.

Further, an output unit that outputs information regarding the determination result of the determination unit may be provided. By doing so, it is possible to display and transmit the determination result and the information based on the determination result to the outside.

Further, the image may be obtained by detecting the sudden braking acceleration of the moving body by the sensor provided in the moving body. By doing so, for example, in a drive recorder or the like, the image extracted based on the detection of sudden braking can be further determined to be a hilarious hat or the like, so that manual labor can be saved.

In addition, the acquisition unit generates an input unit that converts an image into intermediate data that can be mapped, a non-linear mapping unit that converts intermediate data into mapping data, and saliency estimation information that shows a saliency distribution based on the mapping data. The non-linear mapping unit may include an output unit, a feature extraction unit that extracts features from intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit. By doing so, the visual prominence can be estimated at a small calculation cost. Moreover, the visual saliency estimated in this way reflects the contextual attention state.

Further, the information processing method according to the embodiment of the present invention is obtained by estimating the high and low visual saliency in the image based on the image obtained by capturing the outside from the moving body in the acquisition step. Distribution information is acquired, and in the line-of-sight position setting step, the reference line-of-sight position in the image is set according to a predetermined rule. Then, in the visual attention concentration calculation step, the concentration of visual attention in the image is calculated based on the visual saliency distribution information and the line-of-sight position. Then, in the determination step, it is determined that there is a suspicion that a safety problem has occurred while the moving body is traveling, based on the amount of change in the degree of concentration of visual attention over time. By doing so, since the visual saliency distribution information is used, it is based on the amount of temporal change in the contextual attention state that the line of sight tends to be unconsciously focused on an object such as a sign or a pedestrian included in the image. It can be determined that a safety problem has occurred. Therefore, it is possible to determine the suspicion that a safety problem such as an accident or a hiyari hat caused by a psychological burden is caused only by the image.

Further, the above-mentioned information processing method is executed by a computer. By doing so, it is possible to determine that there is a suspicion that a safety problem has occurred using a computer. Therefore, it is possible to accurately determine the suspicion that a safety problem such as an accident or a hiyari hat has occurred using only the image.

Further, the above-mentioned information processing program may be stored in a computer-readable storage medium. By doing so, the program can be distributed as a single unit in addition to being incorporated in the device, and version upgrades and the like can be easily performed.

A determination device according to an embodiment of the present invention will be described with reference to FIGS. 1 to 14. The determination device according to this embodiment is not limited to being installed in a moving body such as an automobile, for example, and may be configured by a server device or the like installed in a business establishment or the like (see FIG. 1). That is, it is not necessary to analyze in real time, and the analysis may be performed after running or the like.

FIG. 1 is an example in which the determination device is configured as a server device. In FIG. 1, when a drive recorder 10 mounted on a vehicle V detects a large acceleration due to sudden braking, sudden acceleration, or other impact by a vehicle behavior detection unit 11 such as an acceleration sensor, predetermined values before and after the detection are detected. The image (moving image) of the period is transmitted to the determination device 1 via the network N such as the Internet. The vehicle behavior detection unit 11 shown in FIG. 2 is not limited to the acceleration sensor, but may be an ABS (Anti-lock Braking System) mounted on the vehicle V, an electronic stability control system, or the like. Images for a predetermined period before and after may be transmitted or stored with the activation of these devices (sensors) as a trigger. By targeting an image in which such sudden braking is detected as a processing target described later, it is possible to execute processing on an image narrowed down to some extent and reduce the processing time (processing amount). Can be done. The image is not limited to the form of transmission by communication as shown in FIG. 1, and may be an image read from a recording medium such as a hard disk drive or a memory card connected to the server device 1. Further, the determination device is not limited to the server device, but may be an in-vehicle device incorporating a determination means or the like, or a PC terminal at home or office, and is configured to distribute processing between these devices and the server. It may have been done.

As shown in FIG. 2, the determination device 1 includes an image input unit 2, a visual saliency calculation unit 3, a line-of-sight coordinate setting unit 4, a vector error calculation unit 5, and a determination unit 6. ..

The image input unit 2 inputs, for example, an image (for example, a moving image) captured by a camera such as the drive recorder described above, and outputs the image as image data. The input moving image is output as image data decomposed in time series such as for each frame. A still image may be input as an image to be input to the image input unit 2, but it is preferable to input as an image group composed of a plurality of still images in chronological order.

Examples of the image input to the image input unit 2 include an image in which the traveling direction of the vehicle is captured. That is, the image is obtained by continuously capturing the outside from the moving body. This image may be an image including a so-called panoramic image, an image acquired by using a plurality of cameras, or the like, which includes a direction other than the traveling direction such as 180 ° or 360 ° in the horizontal direction. Further, what is input to the image input unit 2 is not limited to the image captured by the camera, but may be an image read from a recording medium such as a hard disk drive or a memory card as described above.

The visual saliency calculation unit 3 receives image data from the image input unit 2 and outputs a visual saliency map as visual saliency estimation information described later. That is, the visual saliency calculation unit 3 is a acquisition unit that acquires a visual saliency map (visual saliency distribution information) obtained by estimating the height of the visual saliency based on an image obtained by capturing the outside from a moving body. Function.

FIG. 3 is a block diagram illustrating the configuration of the visual saliency calculation unit 3. The visual saliency calculation unit 3 according to this embodiment includes an input unit 310, a non-linear mapping unit 320, an output unit 330, and a storage unit 390. The input unit 310 converts the image into intermediate data that can be mapped. The non-linear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information showing a saliency distribution based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. The storage unit 390 holds the image data input from the image input unit 2, the coefficient of the filter described later, and the like. This will be described in detail below.

FIG. 4 (a) is a diagram illustrating an image to be input to the visual saliency calculation unit 3, and FIG. 4 (b) is an image showing a visual saliency distribution estimated with respect to FIG. 4 (a). It is a figure which exemplifies. The visual saliency calculation unit 3 according to the present embodiment is a device that estimates the visual saliency of each part in the image. Visual prominence means, for example, the ease of conspicuousness and the ease of gathering eyes. Specifically, the visual prominence is indicated by a probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person who sees the image points to the position.

The positions of FIGS. 4 (a) and 4 (b) correspond to each other. Then, in FIG. 4A, the higher the visual prominence is, the higher the brightness is displayed in FIG. 4B. The image showing the visual saliency distribution as shown in FIG. 4B is an example of the visual saliency map output by the output unit 330. In the example of this figure, the visual saliency is visualized by the luminance value of 256 gradations. An example of the visual saliency map output by the output unit 330 will be described in detail later.

FIG. 5 is a flowchart illustrating the operation of the visual saliency calculation unit 3 according to the present embodiment. The flowchart shown in FIG. 5 is part of a determination method performed by a computer and includes an input step S110, a non-linear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapping data. In the output step S130, visual saliency estimation information (visual saliency distribution information) showing a saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 for extracting features from the intermediate data and an upsampling step S122 for upsampling the data generated in the feature extraction step S121.

Returning to FIG. 3, each component of the visual saliency calculation unit 3 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires image data from the image input unit 2. Then, the input unit 310 converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, but is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a slope of the brightness. In the input step S110, the input unit 310 may further perform image noise removal, resolution conversion, and the like.

In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. Then, the non-linear mapping unit 320 converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by a parameter or the like, and is preferably a function, a functional, or a neural network process.

FIG. 6 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 7 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsample step S122 is performed in the upsample unit 322. Further, in the example of this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, a plurality of intermediate layers 323 are connected.

In particular, the neural network is preferably a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolution layer 324, the input data is convolved by the plurality of filters 325, and the outputs of the plurality of filters 325 are activated.

In the example of FIG. 6, the feature extraction unit 321 is configured to include a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323, and an amplifiering unit 328 is provided between the plurality of intermediate layers 323. Further, the feature extraction unit 321 and the upsampling unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

In the example of this figure, each intermediate layer 323 is composed of two or more convolutional layers 324. However, at least a part of the intermediate layer 323 may consist of only one convolution layer 324. The intermediate layers 323 adjacent to each other are separated by one of a first pooling portion 326, a second pooling portion 327, and an amplifiering portion 328. Here, when the intermediate layer 323 includes two or more convolution layers 324, it is preferable that the number of filters 325 in the convolution layers 324 is equal to each other.

In this figure, the intermediate layer 323 marked "AxB" is composed of B convolution layers 324, meaning that each convolution layer 324 includes A convolution filters for each channel. .. Such an intermediate layer 323 will also be referred to as an "A × B intermediate layer" below. For example, the 64 × 2 intermediate layer 323 consists of two convolution layers 324, meaning that each convolution layer 324 includes 64 convolution filters for each channel.

In the example of this figure, the feature extraction unit 321 includes a 64 × 2 intermediate layer 323, a 128 × 2 intermediate layer 323, a 256 × 3 intermediate layer 323, and a 512 × 3 intermediate layer 323 in this order. Further, the upsampling unit 322 includes 512 × 3 intermediate layer 323, 256 × 3 intermediate layer 323, 128 × 2 intermediate layer 323, and 64 × 2 intermediate layer 323 in this order. Further, the second pooling portion 327 connects two 512 × 3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined according to, for example, the number of pixels of the image data.

Note that this figure is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have another configuration. For example, the 64 × 1 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of convolution layers 324 included in the intermediate layer 323, the calculation cost may be further reduced. Further, for example, the 32 × 2 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of channels in the intermediate layer 323, the calculation cost may be further reduced. Further, both the number of convolution layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling portion 326, and the second intermediate layer is behind the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolution layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is a convolution layer in which the number of filters 325 for each channel is N2. It is composed of layers 324. At this time, it is preferable that N2> N1 holds. Further, it is more preferable that N2 = N1 × 2 holds.

Further, in the plurality of intermediate layers 323 included in the upsample unit 322, it is preferable that the number of filters 325 decreases each time the amplifier ring unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the amplifiering portion 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolution layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is a convolution layer in which the number of filters 325 for each channel is N4. It is composed of layers 324. At this time, it is preferable that N4 <N3 holds. Further, it is more preferable that N3 = N4 × 2 holds.

The feature extraction unit 321 extracts image features having a plurality of abstractions such as gradients and shapes from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 7 shows 64 × 2.
The configuration of the intermediate layer 323 is illustrated. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolution layer 324a and a second convolution layer 324b, and each convolution layer 324 includes 64 filters 325. In the first convolution layer 324a, each channel of the data input to the intermediate layer 323 is subjected to a convolution process using the filter 325. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the ^{three channels h 0} _{i (i = 1.3).} Further, in the example of this figure, the filter 325 is 64 types of 3 × 3 filters, that is, a total of 64 × 3 types of filters. As a result of the convolution process, 64 results h ⁰ _{i, j} (i = 1..3, j = 1...64) are obtained for each channel i.

Next, the activation process is performed on the output of the plurality of filters 325 in the activation unit 329. Specifically, for the corresponding result j of all channels, the activation process is applied to the sum of the corresponding elements. This activation treatment resulted in 64 channels h ¹ _i (i = 1..64).
), That is, the output of the first convolution layer 324a is obtained as an image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

Further, the output data of the first convolution layer 324a is used as the input data of the second convolution layer 324b, and the second convolution layer 324b performs the same processing as that of the first convolution layer 324a, resulting in 64 channels. ^{The output of 2} _i (i = 1..64), i.e., the second convolution layer 324b, is obtained as an image feature. The output of the second convolution layer 324b becomes the output data of the 64 × 2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but a 3 × 3 two-dimensional filter is preferable. Further, the coefficient of each filter 325 can be set independently. In this embodiment, the coefficient of each filter 325 is stored in the storage unit 390, and the nonlinear mapping unit 320 can read it out and use it for processing. Here, the coefficients of the plurality of filters 325 may be determined based on the correction information generated and corrected by using machine learning. For example, the correction information includes the coefficients of the plurality of filters 325 as a plurality of correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert intermediate data into mapping data. The storage unit 390 may be provided in the visual saliency calculation unit 3 or may be provided outside the visual saliency calculation unit 3. Further, the nonlinear mapping unit 320 may acquire the correction information from the outside via the communication network.

8 (a) and 8 (b) are diagrams showing an example of the convolution process performed by the filter 325, respectively. In both FIGS. 8 (a) and 8 (b), an example of 3 × 3 convolution is shown. The example of FIG. 8A is a convolution process using the closest element. The example of FIG. 8B is a convolution process using proximity elements having a distance of two or more. It should be noted that a convolution process using proximity elements having a distance of three or more is also possible. The filter 325 preferably performs a convolution process using proximity elements having a distance of two or more. This is because a wider range of features can be extracted and the accuracy of estimating visual saliency can be further improved.

The operation of the 64 × 2 intermediate layer 323 has been described above. The operation of the other intermediate layers 323 (128 × 2 intermediate layer 323, 256 × 3 intermediate layer 323, 512 × 3 intermediate layer 323, etc.) is also 64 ×, excluding the number of convolution layers 324 and the number of channels. 2 The operation is the same as that of the intermediate layer 323. Further, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsampling unit 322 are the same as described above.

9 (a) is a diagram for explaining the processing of the first pooling unit 326, and FIG. 9 (b) is a diagram for explaining the processing of the second pooling unit 327, and FIG. 9 (b) is a diagram for explaining the processing of the second pooling unit 327. (C) is a diagram for explaining the processing of the amplifier ring unit 328.

The data output from the intermediate layer 323 in the feature extraction unit 321 is input to the next intermediate layer 323 after the pooling process is performed for each channel in the first pooling unit 326. In the first pooling unit 326, for example, a non-overlapping pooling process is performed. FIG. 9A shows a process of associating four 2 × 2 elements 30 with one element 30 for an element group included in each channel. In the first pooling unit 326, such a correspondence is made for all the elements 30. Here, the four elements 30 of 2 × 2 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced to a quarter. As long as the number of elements in the first pooling unit 326 is reduced, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is applied to the output data from the feature extraction unit 321. FIG. 9B shows a process of associating four 2 × 2 elements 30 with one element 30 while overlapping some elements 30. That is, in the repeated association, a part of the 2 × 2 four elements 30 in one association is also included in the 2 × 2 four elements 30 in the next association. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. The number of elements 30 before and after being associated with the second pooling unit 327 is not particularly limited.

The method of each processing performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, a mapping (max pooling) in which the maximum value of the four elements 30 is set as one element 30 or 4 An association (average pooling) in which the average value of one element 30 is set as one element 30 can be mentioned.

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsampling unit 322. Then, the output data from the intermediate layer 323 of the upsample unit 322 is input to the next intermediate layer 323 after the amplifiering process is performed for each channel in the amplifiering unit 328. FIG. 9C shows a process of expanding one element 30 to a plurality of elements 30. The method of enlargement is not particularly limited, and an example is a method of duplicating one element 30 into four 2 × 2 elements 30.

The output data of the last intermediate layer 323 of the upsampling unit 322 is output from the nonlinear mapping unit 320 as mapping data and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs a visual saliency map by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The visual saliency map is, for example, an image (image data) in which the visual saliency as illustrated in FIG. 4B is visualized by a luminance value. Further, the visual saliency map may be an image color-coded according to the visual saliency, such as a heat map, or a visual saliency region having a visual saliency higher than a predetermined reference can be set at other positions. May be an image marked so as to be identifiable. Further, the visual prominence estimation information is not limited to the map information shown as an image or the like, and may be a table or the like listing information indicating the visually prominent region.

The line-of-sight coordinate setting unit 4 sets an ideal line-of-sight, which will be described later, on the visual saliency map. The ideal line of sight is the line of sight that the driver of a vehicle directs along the direction of travel in an ideal traffic environment where there are no obstacles or traffic participants other than himself / herself. It is treated as (x, y) coordinates on image data and visual saliency maps. In this embodiment, the ideal line of sight is a fixed value, but it may be treated as a function of the speed affecting the stopping distance of the moving body or the friction coefficient of the road, or it is determined by using the set route information. May be good. Further, as a method of calculating the ideal viewpoint, the vanishing point corresponding to the current driving road may be used. At that time, the speed of the own vehicle may be detected and the ideal viewpoint may be set 2 seconds or 3 seconds after the vanishing point and the position of the own vehicle. That is, the line-of-sight coordinate setting unit 4 functions as a line-of-sight position setting unit that sets an ideal line-of-sight (reference line-of-sight position) in an image according to a predetermined rule.

The vector error calculation unit 5 calculates the vector error based on the visual saliency map output by the visual saliency calculation unit 3 and the ideal line of sight set by the line-of-sight coordinate setting unit 4 for the visual saliency map and the image. Based on the vector error, the visual attention concentration Ps described later, which indicates the concentration of visual attention, is calculated. That is, the vector error calculation unit 5 functions as a visual attention concentration calculation unit that calculates the concentration of visual attention in the image based on the visual saliency distribution information and the line-of-sight position.

Here, the vector error in this embodiment will be described with reference to FIG. FIG. 10 shows an example of a visual saliency map. This visual saliency map is shown by the brightness values of 256 gradations of H pixels × V pixels, and as in FIG. 4, the higher the visual saliency of the pixels, the higher the brightness is displayed. In FIG. 10, when the coordinates of the ideal line of sight (x, y) = (x _im , y _im ), the vector error with the pixels of arbitrary coordinates (k, m) in the visual saliency map is calculated. When the coordinates with high brightness and the coordinates of the ideal line of sight are separated from each other in the visual saliency map, it means that the position to be gazed at and the position where it is actually easy to gaze are separated, and the image tends to distract the visual attention. It can be said that. On the other hand, when the coordinates with high brightness and the coordinates of the ideal line of sight are close to each other, it means that the position to be watched is close to the position where it is easy to actually watch, and the image is such that the visual attention is easily focused on the position to be watched. I can say.

Next, a method of calculating the visual attention concentration Ps in the vector error calculation unit 5 will be described. In this embodiment, the visual attention concentration Ps is calculated by the following equation (1).

In equation (1), V _vc is the pixel depth (luminance value), f _w is the weighting function, and _derr is the vector error. This weighting function is, for example, a function in which weights are set based on the distance from the pixel indicating the value of _{Vvc to the coordinates of the ideal line of sight.} α is a coefficient such that the visual attention concentration Ps becomes 1 when the coordinates of the bright spot and the coordinates of the ideal line of sight match in the visual saliency map (reference heat map) of one bright spot.

That is, the vector error calculation unit 5 (visual attention concentration calculation unit) includes the value of each pixel constituting the visual saliency map (visual saliency distribution information), the position of each pixel, and the ideal line of sight (reference line of sight position). The degree of concentration of visual attention is calculated based on the vector error with the coordinate position of.

The visual attention concentration Ps obtained in this way is the sum of the weighted relationship between the vector error of the coordinates of all the pixels and the brightness value from the coordinates of the ideal line of sight set on the visual saliency map. It is the reciprocal. A low value of this visual attention concentration Ps is calculated when the distribution with high brightness of the visual saliency map is separated from the coordinates of the ideal line of sight. That is, the visual attention concentration Ps can be said to be the concentration with respect to the ideal line of sight.

FIG. 11 shows an example of an image input to the image input unit 2 and a visual saliency map acquired from the image. FIG. 11A is an input image, and FIG. 11B is a visual saliency map. As described above, in FIG. 11, when the coordinates of the ideal line of sight are set on a road such as a truck traveling ahead, the visual attention concentration Ps in that case is calculated.

In the determination unit 6, the image input from the image input unit 2 based on the temporal change of the visual attention concentration Ps calculated by the vector error calculation unit 5 is for safety reasons during an accident or traveling of a moving object such as a hiyari hat. Determine if you suspect that the problem has occurred. After the judgment, the judgment result and the like are output to the outside.

FIG. 12 shows an example of the temporal change of the visual attention concentration Ps. FIG. 12 shows the change in the visual attention concentration Ps in the moving image for 12 seconds. In FIG. 12, the visual attention concentration Ps changes abruptly between about 6.5 seconds and about 7 seconds. This is, for example, when another vehicle interrupts the front of the own vehicle, and by detecting such a change, it is possible to detect an event that causes a hiatus.

As shown in FIG. 12, an image suspected of being a hilarious hat or the like is extracted by comparing the amount of change such as the rate of change or the change value of the visual attention concentration Ps per short time with a predetermined threshold value or the like. Can be done.

Next, the operation (determination method) in the determination device 1 having the above-described configuration will be described with reference to the flowchart of FIG. Further, the determination program can be obtained by configuring this flowchart as a program executed by a computer functioning as the determination device 1. Further, this determination program is not limited to being stored in the memory or the like of the determination device 1, and may be stored in a storage medium such as a memory card or an optical disk.

First, the image input unit 2 outputs the input image as image data to the visual saliency calculation unit 3 (step S11). In this step, the image data input to the image input unit 2 is decomposed into time series such as an image frame and input to the visual saliency calculation unit 3. In addition, image processing such as noise removal and geometric transformation may be performed in this step.

Next, the visual saliency calculation unit 3 acquires the visual saliency map (step S12). As for the visual saliency map, the visual saliency calculation unit 3 outputs the visual saliency map as shown in FIG. 4B in chronological order by the method described above.

On the other hand, in parallel with step S12, the line-of-sight coordinate setting unit 4 sets the coordinates of the ideal line-of-sight (step S13). As described above, this coordinate is a fixed position such as forward gaze in this embodiment.

Next, the vector error calculation unit 5 calculates the visual attention concentration Ps from the visual saliency map and the ideal line of sight (step S14). That is, as described above, the vector error between the coordinates of the ideal line of sight and the coordinates of the visual saliency map is calculated, and the visual attention is focused by the equation (1) based on the vector error and the value of each pixel. Calculate the degree Ps.

Next, the determination unit 6 causes a safety problem while the moving body is traveling with the image input from the image input unit 2 based on the temporal change of the visual attention concentration Ps calculated by the vector error calculation unit 5. Is suspected to have occurred (step S15).

Next, the determination unit 6 outputs the determination result of step S16 (step S16). In this step, the determination result is not only output, but the result is displayed on a display device or the like, and a label corresponding to the determination result is added to the image input from the image input unit 2. May be good. Alternatively, a process may be performed such that only the images input from the image input unit 2 that are determined to have a safety problem are stored in a specific storage device (specific storage area). .. That is, the determination unit 6 functions as an output unit that outputs information regarding the determination result.

As is clear from the above description, step S12 functions as an acquisition step, step S13 a line-of-sight position setting step, step S14 a visual attention concentration calculation step, and step S15 a determination step.

Here, an example of an image displaying the determination result in the determination device 1 described above will be described with reference to FIG. The image shown in FIG. 14 is generated by the determination unit 6 and displayed on a predetermined display device. The image 50 shown in FIG. 14 includes a traveling image display area 51 and a visual attention concentration display area 52.

In the traveling image display area 51, a traveling image of the vehicle captured by a drive recorder or the like is displayed. In the traveling image display area 51, the vanishing point VP, the visual saliency map VM, the line-of-sight estimation position GE, the detection object frame OF, the horizon and depth distance estimation line HD, the white line estimation WL, and the hiyari hat determination frame HF And can be displayed.

The vanishing point VP may be estimated from the white line estimation WL or the like described later, or may be estimated by using an optical flow or the like. The visual saliency map VM displays a visual saliency map (heat map) of the image displayed in the traveling image display area 51 overlaid on the image. In FIG. 14, only the high-luminance portion on the heat map can be visually recognized, but in reality, the low-luminance portion is also superimposed on the traveling image. That is, the portion of the traveling image in which the line of sight is easily directed is displayed.

In this embodiment, the line-of-sight estimated position GE estimates the position with the highest brightness on the heat map as the line-of-sight position. The detected object frame OF is displayed as a frame surrounding an object detected as a result of object detection by a well-known algorithm in a traveling image. In the object detection in this embodiment, the type of the object to be detected (vehicle, human, etc.) is specified, and only the object belonging to the specified type is detected. The object detection process may be performed by the determination unit 6 or may be performed by another block (not shown in FIG. 1).

The horizon and the depth distance estimation line HD indicate the horizon and the depth distance in the traveling image. The white line estimation WL recognizes and indicates a division line such as a white line in the traveling image. The hiyari hat determination frame HF is formed in a frame shape along the four sides of the traveling image display area 51, and is displayed when it is determined by the determination unit 6 that a safety problem such as a hiyari hat has occurred. Alternatively, the hiyari hat judgment frame HF is always displayed as a frame such as blue, and when it is judged that a safety problem has occurred, it is displayed in red etc., or the display color is changed or blinked. May be good.

The visual attention concentration display area 52 is provided on the right side of the traveling image display area 51. The visual attention concentration display area 52 displays the visual attention concentration Ps calculated by the vector error calculation unit 5 in a bar graph shape. In FIG. 14, reference numeral 52a is a bar indicating the degree of visual attention concentration Ps. When the visual attention concentration Ps shows a large value, the bar 52a becomes high, and when the visual attention concentration Ps shows a small value, the bar 52a becomes low. That is, it can be said that the position where the bar 52a is high tends to have a high degree of concentration (concentration), and the position where the bar 52a is low tends to have a low degree of concentration (dispersion).

The height of the bar 52a of the visual attention concentration display area 52 changes with the transition of the reproduction time of the image because the visual saliency map is acquired in frame units. Therefore, for example, a scene in which the height of the bar 52a suddenly increases indicates that the degree of concentration of visual attention has changed rapidly from dispersion to concentration, and this bar 52a also indicates that the determination unit 6 is safe. It is possible to visually indicate that there is a suspicion that a problem has occurred.

According to the present embodiment, the determination device 1 is obtained by the visual saliency calculation unit 3 estimating the level of visual saliency in the image based on the image obtained by capturing the outside from the moving body. The sex map is acquired, and the line-of-sight coordinate setting unit 4 sets the coordinates of the ideal line-of-sight. Then, the vector error calculation unit 5 calculates the visual attention concentration Ps in the image based on the visual saliency map and the ideal line of sight. Based on the amount of change over time in the visual attention concentration Ps, the determination unit 6 determines that there is a suspicion that a safety problem has occurred while the moving object such as a hiyari hat is running. By doing so, since the visual saliency map is used, the hiyari hat is based on the amount of temporal change in the contextual attention state that the line of sight tends to be unconsciously focused on objects such as signs and pedestrians contained in the image. It can be determined that there is a suspicion that a safety problem such as the above has occurred. Therefore, it is possible to determine the suspicion that a safety problem such as an accident or a hiyari hat caused by a psychological burden is caused only by the image.

Further, the vector error calculation unit 5 calculates the visual attention concentration Ps based on the value of each pixel constituting the visual saliency map and the vector error between the position of each pixel and the coordinate position of the ideal line of sight. ing. By doing so, a value corresponding to the difference between the position where the visual prominence is high and the ideal line of sight is calculated as the visual attention concentration Ps. Therefore, for example, the value of the visual attention concentration Ps can be changed according to the distance between the position where the visual prominence is high and the ideal line of sight.

Further, the determination unit 6 outputs information regarding the determination result. By doing so, it is possible to display and transmit the determination result and the information based on the determination result to the outside.

Further, the image input to the image input unit 2 may be obtained by detecting sudden braking by a sensor included in the moving body, for example, when the acceleration of the moving body is equal to or higher than a reference value. By doing so, for example, in a drive recorder or the like, it is possible to further determine a hiyari hat or the like with respect to the image extracted based on the acceleration. It is possible to further save the trouble of determining whether the cause is a factor other than the above-mentioned or the hiyari hat (for example, the cause of the moving body crossing a step or the cause of rough driving). Further, in addition to the case of sudden braking, the case of sudden driving operation (in other words, dangerous behavior) may be included. For example, the image may be extracted on the assumption that a hilarious hat may have occurred related to an operation of avoiding something from the lateral acceleration, an operation of noticing the running out of the white line and returning the image, and the like. It may also include further rapid acceleration. For example, the image may be extracted on the assumption that the vehicle may have noticed a decrease in speed due to casual driving on a highway and suddenly accelerated.

Further, the visual saliency calculation unit 3 includes an input unit 310 that converts an image into intermediate data that can be mapped, a nonlinear mapping unit 320 that converts the intermediate data into mapping data, and a remarkableness that shows a saliency distribution based on the mapping data. The non-linear mapping unit 320 includes an output unit 330 for generating sex estimation information, and the non-linear mapping unit 320 up-samples the feature extraction unit 321 that extracts features from the intermediate data and the data generated by the feature extraction unit 321. A sample unit 322 and a sample unit 322 are provided. By doing so, the visual prominence can be estimated at a small calculation cost. Moreover, the visual saliency estimated in this way reflects the contextual attention state.

Further, the present invention is not limited to the above examples. That is, those skilled in the art can carry out various modifications according to conventionally known knowledge within a range that does not deviate from the gist of the present invention. Even with such a modification, as long as the determination device of the present invention is still provided, it is, of course, included in the category of the present invention.

1 Judgment device 2 Image input unit 3 Visual prominence calculation unit (acquisition unit)
4 Line-of-sight coordinate setting unit (line-of-sight position setting unit)
5 Vector error calculation unit (visual attention concentration calculation unit)
6 Judgment unit

Claims (8)

An acquisition unit that acquires visual saliency distribution information obtained by estimating the level of visual saliency in the image based on an image of the outside taken from a moving body.
A line-of-sight position setting unit that sets a reference line-of-sight position in the image according to a predetermined rule,
A visual attention concentration calculation unit that calculates the concentration of visual attention in the image based on the visual saliency distribution information and the line-of-sight position.
Based on the amount of change in the degree of concentration of visual attention over time, a determination unit that determines that a safety problem may have occurred while the moving body is running, and a determination unit.
A determination device comprising. The visual attention concentration calculation unit is based on the value of each pixel constituting the visual saliency distribution information and the vector error between the position of each pixel and the coordinate position of the reference line-of-sight position. The determination device according to claim 1, wherein the degree of concentration of the above is calculated. The determination device according to claim 1 or 2, further comprising an output unit that outputs information regarding the determination result of the determination unit. The determination device according to any one of claims 1 to 3, wherein the image is obtained by detecting sudden braking of the moving body by a sensor included in the moving body. .. The acquisition unit
An input unit that converts the image into intermediate data that can be mapped,
A non-linear mapping unit that converts the intermediate data into mapping data,
It is provided with an output unit that generates saliency estimation information showing a saliency distribution based on the mapping data.
The nonlinear mapping unit includes a feature extraction unit that extracts features from the intermediate data, and an upsample unit that upsamples the data generated by the feature extraction unit.
The determination device according to any one of claims 1 to 4, wherein the determination device is characterized by the above. It is a judgment method executed by a judgment device that performs a predetermined judgment process based on an image obtained by capturing an image of the outside from a moving body.
An acquisition step of acquiring visual saliency distribution information obtained by estimating the level of visual saliency in the image based on the image, and
A line-of-sight position setting step of setting a reference line-of-sight position in the image according to a predetermined rule, and
A visual attention concentration calculation step for calculating the concentration of visual attention in the image based on the visual saliency distribution information and the line-of-sight position.
Based on the amount of change in the degree of concentration of visual attention over time, a determination step of determining that there is a suspicion that a safety problem has occurred while the moving body is running, and a determination step.
A determination method characterized by including. A determination program, characterized in that the determination method according to claim 6 is executed by a computer. A computer-readable storage medium comprising storing the determination program according to claim 7.

Images