WO2023286835A1

WO2023286835A1 - Gaze guidance device, gaze guidance method, gaze guidance program, and storage medium

Info

Publication number: WO2023286835A1
Application number: PCT/JP2022/027707
Authority: WO
Inventors: 俊明井上
Original assignee: パイオニア株式会社
Priority date: 2021-07-14
Filing date: 2022-07-14
Publication date: 2023-01-19
Also published as: JPWO2023286835A1

Abstract

A calculation unit (131) of this gaze guidance device (10) calculates a first order in which a plurality of areas in an image, determined on the basis of the degree of ease in focusing on a gaze, are arranged in order from the largest peak value of the degree of ease in focusing on the gaze. A determination unit (132) determines whether a designated second order and the first order are the same. A guidance unit (133) guides a gaze that is based on the second order when the determination unit (132) determines that the first order and the second order differ from each other.

Description

Gaze Guidance Device, Gaze Guidance Method, Gaze Guidance Program, and Storage Medium

The present invention relates to a gaze guidance device, a gaze guidance method, a gaze guidance program, and a storage medium.

Conventionally, there is known a technique for estimating information related to movement of the gaze point based on visual saliency calculated from images.

Japanese Unexamined Patent Application Publication No. 2021-77248

However, conventional technology has the problem that it is sometimes difficult to guide the line of sight.

For example, the line-of-sight movement route estimated from an image of the front of the vehicle may not match the line-of-sight movement route for driving the vehicle safely.

For example, in order for a vehicle driver to drive safely, it may be desirable to give priority to inconspicuous pedestrians over buildings with eye-catching colors and shapes.

The present invention has been made in view of the above, and aims to provide a line-of-sight guidance device, a line-of-sight guidance method, a line-of-sight guidance program, and a storage medium that can guide the line of sight.

The line-of-sight guidance device according to claim 1 is a first region in which a plurality of regions in an image determined based on the degree of ease of gathering of the line of sight are arranged in descending order of the peak value of the degree of ease of gathering of the line of sight. a calculation unit for calculating the order of the , a determination unit for determining whether or not the designated second order and the first order are the same, and the determination unit determines the first order and the second and a guiding unit that guides the line of sight according to the second order when it is determined that the order of the two is different.

A visual guidance method according to claim 6 is a visual guidance method executed by a computer, wherein a plurality of areas in an image determined based on the degree of ease of gathering of the line of sight is determined by the ease of gathering of the line of sight. a calculating step of calculating a first order arranged in descending order of the peak value of the degree of; a determining step of determining whether the specified second order and the first order are the same; and a guidance step of guiding a line of sight according to the second order when the determination step determines that the first order and the second order are different.

According to a seventh aspect of the present invention, there is provided a line-of-sight guidance program, wherein a plurality of regions in an image, which are determined based on the degree of ease of line-of-sight gathering, are arranged in descending order of the peak value of the degree of ease of line-of-sight gathering. a calculating step of calculating the order of; a determining step of determining whether or not the specified second order and the first order are the same; and the first order and the second If it is determined that the order of is different, the guidance step of guiding the line of sight according to the second order is executed by the computer.

In the storage medium according to claim 8, the plurality of regions in the image determined based on the degree of ease of line-of-sight gathering are arranged in descending order of the peak value of the degree of ease of line-of-sight gathering. a calculating step of calculating an order; a determining step of determining whether the designated second order and the first order are the same; and a guidance step of guiding the line of sight in accordance with the second order when the order is determined to be different.

FIG. 1 is a diagram showing an overview of the line-of-sight guidance process. FIG. 2 is a diagram illustrating a configuration example of a visual guidance device. FIG. 3 is a diagram illustrating visual salience. FIG. 4 is a diagram illustrating an example of calculation of the order of line of sight. FIG. 5 is a diagram showing an example of a line-of-sight guidance method. FIG. 6 is a diagram showing an example of a line-of-sight guidance method. FIG. 7 is a diagram showing an example of a line-of-sight guidance method. FIG. 8 is a flow chart showing the processing flow of the visual guidance device.

Below, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. In addition, this invention is not limited by embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[First embodiment]
The visual line guidance device according to the first embodiment calculates the movement order of the line of sight based on the image, and performs guidance so that the movement order becomes the designated order.

FIG. 1 is a diagram showing an overview of the line-of-sight guidance process. As shown in FIG. 1, the visual guidance device 10 is provided in a vehicle V. As shown in FIG.

The line-of-sight guidance device 10 may be an in-vehicle device such as a drive recorder and a car navigation system, or an information processing device such as a personal computer and a server device. In addition, each block may be appropriately distributed, such as the calculation unit 131 and the determination unit 132 existing on the server, and the guidance unit 133 existing on the terminal of the vehicle. is not limited to

As shown in FIG. 1, the visual guidance device 10 first captures an image in front of the vehicle V (step S1).

Next, the gaze guidance device 10 calculates the order of gaze based on visual salience from the image (step S2). A method of calculating the order of sight lines will be described later.

Then, if the order of the line of sight calculated in step S2 is different from the designated order, the visual line guidance device 10 guides the line of sight according to the designated order (step S3).

For example, the line-of-sight guidance device 10 can guide the line of sight by outputting an image or sound.

FIG. 2 is a diagram showing a configuration example of a visual guidance device. As shown in FIG. 2 , the visual guidance device 10 has an interface section 11 , a storage section 12 and a control section 13 .

The interface unit 11 is an interface for inputting and outputting data. Also, the interface unit 11 may be a communication module capable of data communication with another device via a communication network such as the Internet.

In the example of FIG. 2, the interface unit 11 is connected to the camera 20 and the output device 30. The camera 20 is provided in the vehicle V and photographs the surroundings or inside of the vehicle V. As shown in FIG. The output device 30 is a display that displays images or a speaker that outputs audio.

Also, the output device 30 may be a see-through projection with a transparent screen and a projector. The transparent screen may be the front window of the vehicle. Also, the output device 30 may be AR (Augmented Reality) goggles that allow viewing of real scenery and CG at the same time. Also, the output device 30 may be any device capable of displaying an object superimposed on the scenery, such as a head-up display, an MR (Mixed Reality) device, or a mirror device having a transmissive display.

The storage unit 12 stores various programs executed by the visual guidance device 10, data necessary for execution of processing, and the like.

The storage unit 12 stores model information 121. The model information 121 is parameters such as weights for constructing a neural network for calculating visual saliency.

The control unit 13 is realized by executing various programs stored in the storage unit 12 by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), and controls the operation of the entire visual guidance device 10. do.

Note that the control unit 13 is not limited to a CPU or MPU, and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 13 has a calculation unit 131 , a determination unit 132 and a guidance unit 133 .

The calculation unit 131 calculates the order in which the plurality of regions in the image, which are determined based on the degree of ease of line-of-sight gathering, are arranged in descending order of the peak value of the degree of ease of line-of-sight gathering.

Here, the order calculated by the calculation unit 131 is an example of the first order. Also, the degree of easiness of attracting lines of sight is, for example, visual saliency.

Visual salience will be explained using FIG. FIG. 3 is a diagram illustrating visual salience. As shown in FIG. 3, visual saliency is an index obtained by estimating the position of the line of sight of the driver for an image showing the front of the vehicle (see, for example, Patent Document 1).

Visual salience may be calculated by inputting an image into a neural network. For example, the neural network is trained on a large number of images taken in a wide range of fields and the gaze information of multiple subjects who actually saw them.

Visual salience is, for example, an 8-bit (0 to 255) value given to each pixel of an image, and is expressed as a value that increases as the probability of being the position of the driver's line of sight increases.

Therefore, as shown in Fig. 3, a saliency map can be obtained by arranging (mapping) the visual saliency to the corresponding pixels of the image. A saliency map can be said to be data obtained by mapping the degree of easiness of eye-gazing for each pixel in an image.

Here, multiple regions (clusters) can be defined by dividing the saliency map, which is a plane corresponding to the image. At this time, each region has a point at which the visual salience peaks. Although the method of division is not particularly limited, it may be based on object regions detected using a single shot multiple detector (SSD), or may be based on a clustering method such as k-means. may Alternatively, the point having the peak value may be directly detected by scanning and comparing the saliency map pixel by pixel without segmentation.

Here, it is known that the line of sight tends to move in order from a region with a large peak value to a region with a small peak value in units of calculated visual saliency regions (see, for example, Reference 1). .
Reference 1: Adachi et al., ``Comparison of top-down and bottom-up factors in saliency maps using gaze measurement'', MIRU2017, PS2-19, 2017.

The peak value and area can be defined by any method. For example, the eye guidance device 10 identifies the point of maximum visual salience in the entire saliency map. Then, the visual guidance device 10 further identifies the point with the maximum visual salience from the area excluding the area within the predetermined range centered on the identified point. The visual guidance device 10 can regard the specified points as points having peak values by repeating this process.

FIG. 4 is a diagram showing an example of calculating the order of line of sight. As shown in FIG. 4, the calculation unit 131 first calculates a saliency map 200a from the image 200. As shown in FIG. Here, a region 201, a region 202, a region 203, and a region 204 are defined in the saliency map 200a.

Each of the

regions

201, 202, 203, and 204 is a region of a predetermined size including the point where the visual saliency takes a peak value.

Here, when arranging the respective regions in the order of the magnitude of the peak value, the region 201, the region 202, the region 203, and the region 204 are obtained. For example, the region 201 is the region containing the points with the maximum peak values in the saliency map 200a.

At this time, the calculation unit 131 calculates the order of the line of sight in the image 200 as (A)→(B)→(C)→(D). (A), (B), (C), and (D) are points corresponding to

regions

201, 202, 203, and 204 when the saliency map 200a is superimposed on the image 200, respectively.

The determination unit 132 determines whether or not the designated designation order and the order calculated by the calculation unit 131 (hereinafter referred to as calculation order) are the same. The specified order is an example of a second order. The designated order is the order of the driver's line of sight that improves safety.

The designated order may be determined manually based on knowledge, or may be determined automatically by image recognition or the like.

FIG. 5 is a diagram showing an example of a line-of-sight guidance method. As shown in FIG. 5, it is assumed that the designation order for the image 200 is (A)→(D)→(B)→(C). At this time, the determination unit 132 determines that the calculation order and the designation order are different.

When the determination unit 132 determines that the calculation order and the designation order are different, the guidance unit 133 guides the line of sight according to the designation order.

For example, in the example of FIG. 5, it is thought that increasing the visual saliency of the region containing (D) can bring the calculation order closer to the designated order. Therefore, the guiding unit 133 edits (D) in the image 200 by combining, transforming, or the like.

In this way, the guidance unit 133 outputs an image edited to increase the visual saliency of the region (region including (D)) ranked higher in the designation order than in the calculation order. to induce

For example, the guidance unit 133 edits the image by changing pixel values or by combining processing. Specifically, the guiding unit 133 locally changes the brightness of the target area, changes the color temperature of the target area compared to the surroundings, synthesizes another image with the target area, and the like. Edit by

Furthermore, the guidance unit 133 causes the output device 30 to display an object for guiding the line of sight to an area with a higher rank in the designation order than the rank in the calculation order, in such a manner that the user can view it simultaneously with the scenery.

In this case, the calculation unit 131 calculates the order of calculation for the landscape image in the user's field of view. Also, the user here is assumed to be the driver of the vehicle V. FIG.

In this case, the output device 30 is, for example, a display device with a transparent display. As shown in FIG. 6 , the guidance unit 133 causes the output device 30 to display the object 311 .

As a result, the user can simultaneously view the scenery 310 and the object 311 displayed by the guidance unit 133 through the output device 30.

Further, the guidance unit 133 may display an image in which the scenery 310 and the object 311 are superimposed. The object 311 is for increasing the line-of-sight rank of the bicycle, which is a traffic participant to which attention should be paid with priority.

As shown in FIG. 7, the output device 30 may be a mirror having an image display function. FIG. 7 is a diagram showing an example of a line-of-sight guidance method.

In the example of FIG. 7, the guidance unit 133 causes the output device 30 to display the object 321. In the example of FIG.

As a result, the user can simultaneously view the scenery 320 and the object 321 displayed by the guidance unit 133 via the output device 30.

The object 321 is for raising the line-of-sight rank of the following vehicle, which is a traffic participant that should be given priority attention.

FIG. 8 is a flowchart showing the processing flow of the visual guidance device. As shown in FIG. 5, first, the camera 20 provided on the vehicle V captures an image (step S101).

Next, the gaze guidance device 10 calculates the gaze order based on visual salience from the image (step S102).

Here, the visual guidance device 10 determines whether or not the calculated order matches the specified order (step S103).

If the calculated order does not match the designated order (step S103, No), the visual guidance device 10 guides the line of sight according to the designated order (step S104).

If the calculated order matches the specified order (step S103, Yes), the visual guidance device 10 outputs the captured image as it is (step S106). In this case, the visual guidance device 10 does not have to output an image.

[Effects of the first embodiment]
As described above, the calculation unit 131 of the visual line guidance device 10 determines a plurality of regions in the image determined based on the degree of ease with which the line of sight gathers. Compute a first order in descending order. The determination unit 132 determines whether the specified second order and the first order are the same. When the determining unit 132 determines that the first order and the second order are different, the guiding unit 133 guides the line of sight according to the second order.

With this, it is possible to guide the user's line of sight. Also, for example, it is possible to guide the line of sight of the driver of the vehicle and improve safety.

The determination unit 132 determines whether or not the second order, which is designated so that the order of the specific object is higher, is the same as the first order. The guidance unit 133 guides the line of sight by outputting an image edited to increase the visual salience of the regions ranked higher in the second order than in the first order.

As a result, for example, the line of sight of the vehicle driver can be guided to a specific traffic participant, improving safety.

The guidance unit 133 edits the image by changing the pixel value or by combining processing. This makes it possible to easily guide the line of sight.

The calculation unit 131 calculates the first order for the landscape image in the user's field of view. The guidance unit 133 displays an object for guiding the line of sight to an area ranked higher in the second order than in the first order on a predetermined display device in a manner that allows the user to view the object simultaneously with the scenery. Let

With this, for example, using a transparent display, you can see the actual scenery and virtual objects for visual guidance at the same time.

The determination unit 132 classifies images into classes corresponding to the output obtained by inputting the saliency map and the feature amount output from the encoder to a CNN having an output layer corresponding to each class.

As a result, it is possible to reduce the impact of errors in saliency map estimation on class classification and improve classification accuracy.

REFERENCE SIGNS LIST 10 visual guidance device 11 interface unit 12 storage unit 13 control unit 20 camera 30 output device 121 model information 131 calculation unit 132 determination unit 133 guidance unit

Claims

a calculation unit that calculates a first order in which a plurality of regions in an image determined based on the degree of ease of line-of-sight gathering are arranged in descending order of the peak value of the degree of ease of line-of-sight gathering;
a determination unit that determines whether the designated second order and the first order are the same;
a guiding unit that guides a line of sight according to the second order when the determining unit determines that the first order and the second order are different;
A line-of-sight guidance device comprising:
2. The method according to claim 1, wherein the determination unit determines whether or not the first order is the same as the second order in which a specific object is ranked higher. line-of-sight device.
The guidance unit guides the line of sight by outputting an image edited to increase the visual salience of the regions ranked higher in the second order than in the first order. The line-of-sight guidance device according to claim 1 or 2.
4. The line of sight guidance device according to claim 3, wherein the guidance unit edits the image by changing a pixel value or by combining processing.
The calculation unit calculates the first order for images of scenery in a user's field of view;
The guidance unit is configured to provide an object for guiding a line of sight to an area ranked higher in the second order than in the first order in a manner that allows the user to see the scenery at the same time. 2. The line-of-sight guidance device according to claim 1, which is displayed on a display device.
A computer-implemented eye guidance method comprising:
a calculating step of calculating a first order in which a plurality of regions in an image determined based on the degree of ease of attracting the line of sight are arranged in descending order of the peak value of the degree of ease of attracting the line of sight;
a determination step of determining whether the designated second order and the first order are the same;
a guiding step of guiding a line of sight according to the second order when the determining step determines that the first order and the second order are different;
A line-of-sight guidance method comprising:
a calculating step of calculating a first order in which a plurality of regions in an image determined based on the degree of ease of attracting the line of sight are arranged in descending order of the peak value of the degree of ease of attracting the line of sight;
a determination step of determining whether the designated second order and the first order are the same;
a guiding step of guiding a line of sight according to the second order when the determining step determines that the first order and the second order are different;
A line-of-sight guidance program for executing on a computer.
a calculating step of calculating a first order in which a plurality of regions in an image determined based on the degree of ease of attracting the line of sight are arranged in descending order of the peak value of the degree of ease of attracting the line of sight;
a determination step of determining whether the designated second order and the first order are the same;
a guiding step of guiding a line of sight according to the second order when the determining step determines that the first order and the second order are different;
A storage medium characterized by storing a line-of-sight guidance program for causing a computer to execute.