JP2020184146A - Saliency estimation device, saliency estimation method and program - Google Patents

Abstract

To provide a saliency estimation device, a saliency estimation method and a program for accurately detecting from an image a region which a moving person looks at and feels to be highly salient.SOLUTION: A saliency estimation device 10 comprises: an input unit 110 which acquires moving image data and outputs each frame image of frame images constituting the acquired moving image data; a modification unit 120 which generates a modified image by modifying the frame image outputted by the input unit 110; and a saliency estimation unit 130 which processes the modified image to generate saliency estimation information showing a saliency distribution within the modified image or the frame image. The modification unit 120 generates brightness information for modifying brightness of at least a part of the frame image, by using speed information about a movement speed of a first viewpoint and a relative position from a reference point to the at least a part in the frame image, and uses this brightness information to modify brightness of the at least a part.SELECTED DRAWING: Figure 1

Description

The present invention relates to a saliency estimation device, a saliency estimation method, and a program.

A technique for automatically detecting a prominent area in an image has been proposed. On the other hand, when a person is moving, the effective field of view of the person becomes narrower as the moving speed of the person increases. Non-Patent Document 1 describes that a prominent region is automatically detected in consideration of an effective field of view. Specifically, Non-Patent Document 1 describes that the resolution and saturation of the target image are reduced according to the distance from the gazing point, and then the prominence is estimated.

Morimoto et al., "Estimation model for moving images considering the response characteristics of the MST field", Technical Report of the Institute of Imaging Media, Vol.38, No.10, pp.57-60, 2014

The present inventor has investigated a method of detecting a region that is felt to be highly prominent when viewed by a moving person from an image with high accuracy. As an example of the problem to be solved by the present invention, it is possible to detect, from an image, a region that is felt to be highly prominent when viewed by a moving person with high accuracy.

The invention according to claim 1 comprises a correction unit that generates a corrected image by correcting an image of a landscape viewed from a first viewpoint.
A saliency estimation unit that generates saliency estimation information indicating a saliency distribution in the corrected image or in the image by processing the corrected image, and a saliency estimation unit.
With
The correction unit
Brightness information indicating a change in the brightness of at least a part of the image is generated by using the speed information regarding the moving speed of the first viewpoint and the relative position from the reference point in the image to the at least a part.
It is a saliency estimation device that corrects at least a part of the brightness by using the brightness information.

The invention according to claim 8 is a computer.
A corrected image is generated by correcting the image of the scenery seen from the first viewpoint.
By processing the corrected image, saliency estimation information indicating the saliency distribution in the corrected image or in the image is generated.
Furthermore, the computer
Brightness information indicating a change in the brightness of at least a part of the image is generated by using the speed information regarding the moving speed of the first viewpoint and the relative position from the reference point in the image to the at least a part.
This is a saliency estimation method for correcting at least a part of the brightness using the brightness information.

The invention according to claim 9 is applied to a computer.
A correction function that generates a corrected image by correcting the image of the scenery seen from the first viewpoint,
An estimation function that generates saliency estimation information indicating a saliency distribution in the corrected image or in the image by processing the corrected image, and an estimation function.
To have
Further, as at least a part of the correction function
A function to generate brightness information indicating a change in brightness of at least a part of the image by using speed information regarding the moving speed of the first viewpoint and a relative position from a reference point in the image to at least a part of the image. When,
A function to correct at least a part of the brightness using the brightness information, and
It is a program to have.

It is a figure which shows the functional structure of the saliency estimation apparatus which concerns on 1st Embodiment. It is a figure for demonstrating the setting method of the viewing angle information. It is a figure for demonstrating an example of a viewing angle information. It is a figure for demonstrating the brightness information. It is a figure for demonstrating the resolution information. It is a figure for demonstrating the saturation information. It is a figure which shows an example of the functional structure of a correction processing part. It is a block diagram which illustrates the structural example of the saliency estimation part. (A) is a diagram exemplifying an image to be input to the saliency estimation unit, and (b) is a diagram exemplifying an image showing a saliency distribution estimated with respect to (a). It is a flowchart which illustrates the processing method which concerns on the 1st configuration example. It is a figure which illustrates the structure of the nonlinear mapping part in detail. It is a figure which illustrates the structure of the intermediate layer. (A) and (b) are diagrams showing an example of a convolution process performed by a filter, respectively. (A) is a diagram for explaining the processing of the first pooling unit, (b) is a diagram for explaining the processing of the second pooling unit, and (c) is a diagram for explaining the processing of the second pooling unit. It is a figure for demonstrating the process of. It is a block diagram which illustrates the hardware configuration of the saliency estimation device. It is a figure which shows the functional structure of the saliency estimation apparatus which concerns on 2nd Embodiment. It is a figure which shows the functional structure of the saliency estimation apparatus which concerns on 3rd Embodiment. It is a figure for demonstrating the operation example of the reference point setting part. It is a figure which shows the functional structure of the saliency estimation apparatus which concerns on 4th Embodiment. It is a figure which illustrates the structure of the saliency estimation part which concerns on 5th Embodiment. It is a flowchart which illustrates the learning operation which concerns on 5th Embodiment. It is a figure which illustrates the structure and use environment of the arithmetic unit which concerns on 6th Embodiment. It is a figure which illustrates the structure of the saliency estimation part which concerns on 7th Embodiment. It is a figure which illustrates the image which the synthesis information generated by the synthesis part shows. It is a figure which illustrates the structure of the saliency estimation part which concerns on 8th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by the same reference numerals, and description thereof will be omitted as appropriate.

(First Embodiment)
FIG. 1 is a diagram showing a functional configuration of the saliency estimation device 10 according to the first embodiment. The saliency estimation device 10 shown in this figure includes an input unit 110, a correction unit 120, and a saliency estimation unit 130. The input unit 110 acquires the moving image data, and outputs each of the frame images constituting the acquired moving image data to the correction unit 120. These frame images are images of the scenery seen from the first viewpoint. The first viewpoint is, for example, the position where the camera that generated the moving image data was arranged. The correction unit 120 generates a corrected image by correcting the frame image output by the input unit 110. The saliency estimation unit 130 generates saliency estimation information by processing the corrected image. The saliency estimation information shows the saliency distribution in the corrected image or the frame image. Here, the correction unit 120 provides the brightness information indicating the change in the brightness of at least a part of the frame image from the speed information regarding the moving speed of the first viewpoint and the reference point in the frame image to at least a part described above. Generate using relative position. Then, the correction unit 120 corrects at least a part of the above-mentioned brightness by using this brightness information. Hereinafter, the saliency estimation device 10 will be described in detail.

As described above, moving image data is input to the input unit 110. The input unit 110 outputs each of the plurality of frame images included in the moving image data to the correction unit 120.

The correction unit 120 corrects the frame image input from the input unit 110. Specifically, the correction unit 120 includes a field of view setting unit 122, a correction information generation unit 124, and a correction processing unit 126.

The field of view setting unit 122 receives speed information and reference point information from the outside. The speed information indicates the moving speed. This moving speed is, for example, the speed of the camera when the moving image data acquired by the input unit 110 is captured, but is not limited to this. The reference point information specifies a position to be a gazing point in the frame image acquired by the correction unit 120. The visual field setting unit 122 uses the speed information to generate viewing angle information indicating the viewing angle of the person when the person moves at that speed. Then, the visual field setting unit 122 outputs the speed information, the reference point information, and the visual field angle information to the correction information generation unit 124.

FIG. 2 is a diagram for explaining a method of setting the viewing angle information. The viewing angle of a moving person becomes narrower as the moving speed increases. The visual field setting unit 122 stores data showing the relationship between the speed and the viewing angle, as shown in FIG. 2, for example, and uses this data to specify and specify the viewing angle corresponding to the input moving speed. Generates viewing angle information indicating the viewing angle.

FIG. 3 is a diagram for explaining an example of viewing angle information. In the example shown in this figure, the viewing angle information is information that divides the image into a plurality of regions based on the distance from the reference point. Specifically, the 0th region including the reference point is the region where the image should be clear. Then, the correction information generation unit 124 and the correction processing unit 126 become the first region located outside the 0th region, the second region located outside the first region, and so on. Correct the image so that is gradually obscured and darkened.

The field of view setting unit 122 determines the size of at least the 0th region by using the speed information. For example, when the speed indicated by the speed information is low, the 0th region is increased and the other regions are narrowed. In addition to the size of each region, the field of view setting unit 122 may set the number of regions to be set by using the speed information. In this case, the field of view setting unit 122 increases the number of areas to be set as the speed increases.

In the example shown in FIG. 3, the outline of each region is rectangular. However, this outline may have other shapes (for example, circular or elliptical).

Return to FIG. The correction information generation unit 124 generates correction information using the reference point information acquired from the field of view setting unit 122. The correction information is information for specifying the correction amount of the value of each pixel of the frame image. The viewing angle information is generated using the velocity information and the distance from the reference point as described above. Therefore, the field of view setting unit 122 substantially uses the relative position and velocity information from the reference point to generate the correction information.

Specifically, the correction information includes brightness information, resolution information, and saturation information. The brightness information indicates a change in the brightness of at least a part of the image, the resolution information indicates a change in the resolution of at least a part of the image, and the saturation information indicates a change in the saturation of at least a part of the image. Indicates a change. Then, the correction information generation unit 124 outputs the viewing angle information, the reference point information, and the correction information to the correction processing unit 126.

FIG. 4 is a diagram for explaining brightness information, FIG. 5 is a diagram for explaining resolution information, and FIG. 6 is a diagram for explaining saturation information. As shown in these figures, the correction information generated by the correction information generation unit 124 indicates that the brightness, the resolution, and the saturation decrease as the distance from the reference point increases. Specifically, for any of the brightness, the resolution, and the saturation, the correction amount is set for each k value in the “kth region” shown in FIG. And as the value of k increases, all of the brightness, resolution, and saturation decrease.

Return to FIG. The correction processing unit 126 acquires a frame image from the input unit 110, and corrects the frame image using the viewing angle information, the reference point information, and the correction information. Specifically, the correction processing unit 126 defines a reference point in the frame image using the reference point information. Then, the correction processing unit 126 divides the frame image into each region shown in FIG. 3 by using the reference point and the viewing angle information. Then, the correction processing unit 126 corrects each area according to the correction information.

FIG. 7 is a diagram showing an example of the functional configuration of the correction processing unit 126. The correction processing unit 126 includes a resolution correction unit 202, a saturation correction unit 204, and a brightness correction unit 206. The resolution correction unit 202 corrects the resolution of the frame image for each area by using the resolution information included in the correction information. The saturation correction unit 204 corrects the saturation of the frame image for each region by using the information included in the correction information again. The brightness correction unit 206 corrects the brightness of the frame image for each region by using the brightness information included in the correction information. In the example shown in this figure, the resolution correction unit 202, the saturation correction unit 204, and the brightness correction unit 206 are arranged in series in this order, but the arrangement order thereof is not limited to the example shown in FIG.

Then, the correction unit 120 outputs the corrected frame image (corrected image) to the saliency estimation unit 130.

<Structure example of saliency estimation unit 130>
FIG. 8 is a block diagram illustrating a configuration example of the saliency estimation unit 130. The saliency estimation unit 130 generates saliency estimation information by inputting a corrected frame image into the model generated by machine learning. Specifically, the saliency estimation unit 130 includes an input unit 310, a non-linear mapping unit 320, and an output unit 330. The input unit 310 converts the input frame image (hereinafter, referred to as an image in the description regarding the saliency estimation unit 130) into intermediate data that can be mapped. The non-linear mapping unit 320 converts the intermediate data into mapping data. The output unit 330 generates saliency estimation information based on the mapping data. The nonlinear mapping unit 320 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. This will be described in detail below.

FIG. 9A is a diagram illustrating an image to be input to the saliency estimation unit 130, and FIG. 9B exemplifies an image showing a saliency distribution estimated with respect to FIG. 9A. It is a figure. For the sake of explanation, these figures show the frame image before being corrected by the correction unit 120. The saliency estimation unit 130 according to this configuration example estimates the saliency of each part in the image. Severity means, for example, the ease of conspicuousness and the ease of gathering eyes. Specifically, the prominence is indicated by a probability or the like. Here, the magnitude of the probability corresponds to, for example, the magnitude of the probability that the line of sight of the person who sees the image points to the position.

The positions of FIGS. 9 (a) and 9 (b) correspond to each other. Then, in FIG. 9A, the higher the prominence is, the higher the brightness is displayed in FIG. 9B. The image showing the saliency distribution as shown in FIG. 9B is an example of the saliency estimation information output by the output unit 330. In the example of this figure, the prominence is visualized by the luminance value of 256 gradations. An example of the saliency estimation information output by the output unit 330 will be described in detail later.

The estimation results of the saliency distribution are, for example, prediction of the line of sight of traffic participants such as drivers and pedestrians, prevention of oversight of traffic participants, evaluation of the appearance of contents such as advertising media, line of sight guidance, athletes and skilled experts. It can be used in various fields such as digitization of know-how and understanding of visual recognition of living organisms. Further, the prominence estimation unit 130 and the processing method according to this configuration example include mobility fields such as automatic driving, advanced driver assistance system (ADAS), and road traffic system, virtual reality (VR), augmented reality (AR), games, and the like. It can be applied to the entertainment field, document, video content, content field such as signage, medical field such as diagnostic imaging, surgical support, and nursing care service.

FIG. 10 is a flowchart illustrating a processing method according to the first configuration example. The processing method according to this configuration example is a processing method executed by a computer and includes an input step S110, a nonlinear mapping step S120, and an output step S130. In the input step S110, the image is converted into intermediate data that can be mapped. In the nonlinear mapping step S120, the intermediate data is converted into mapping data. In the output step S130, the saliency estimation information showing the saliency distribution is generated based on the mapping data. Here, the nonlinear mapping step S120 includes a feature extraction step S121 for extracting features from the intermediate data and an upsampling step S122 for upsampling the data generated in the feature extraction step S121. The processing method according to this configuration example is realized by the prominence estimation unit 130 according to the configuration example.

Returning to FIG. 8, each component of the saliency estimation unit 130 will be described. In the input step S110, the input unit 310 acquires an image and converts it into intermediate data. The input unit 310 acquires an image from the correction unit 120. Then, the input unit 310 converts the acquired image into intermediate data. The intermediate data is not particularly limited as long as it is data that can be accepted by the nonlinear mapping unit 320, but is, for example, a high-dimensional tensor. Further, the intermediate data is, for example, data in which the brightness of the acquired image is normalized, or data in which each pixel of the acquired image is converted into a slope of the brightness. In the input step S110, the input unit 310 may further perform image noise removal, resolution conversion, and the like.

In the nonlinear mapping step S120, the nonlinear mapping unit 320 acquires intermediate data from the input unit 310. Then, the non-linear mapping unit 320 converts the intermediate data into mapping data. Here, the mapping data is, for example, a high-dimensional tensor. The mapping process applied to the intermediate data by the nonlinear mapping unit 320 is, for example, a mapping process that can be controlled by parameters or the like, and is preferably a function, a functional, or a neural network process.

FIG. 11 is a diagram illustrating the configuration of the nonlinear mapping unit 320 in detail, and FIG. 12 is a diagram illustrating the configuration of the intermediate layer 323. As described above, the nonlinear mapping unit 320 includes a feature extraction unit 321 and an upsampling unit 322. The feature extraction step S121 is performed in the feature extraction unit 321, and the upsample step S122 is performed in the upsample unit 322. Further, in the example of this figure, at least one of the feature extraction unit 321 and the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323. In the neural network, a plurality of intermediate layers 323 are connected.

In particular, the neural network is preferably a convolutional neural network. Specifically, each of the plurality of intermediate layers 323 includes one or more convolutional layers 324. Then, in the convolutional layer 324, the input data is convolved by the plurality of filters 325, and the outputs of the plurality of filters 325 are activated.

In the example of FIG. 11, the feature extraction unit 321 is configured to include a neural network including a plurality of intermediate layers 323, and includes a first pooling unit 326 between the plurality of intermediate layers 323. Further, the upsampling unit 322 is configured to include a neural network including a plurality of intermediate layers 323, and an amplifiering unit 328 is provided between the plurality of intermediate layers 323. Further, the feature extraction unit 321 and the upsample unit 322 are connected to each other via a second pooling unit 327 that performs overlap pooling.

In the example of this figure, each intermediate layer 323 is composed of two or more convolutional layers 324. However, at least a part of the intermediate layer 323 may consist of only one convolutional layer 324. The intermediate layers 323 adjacent to each other are separated by one of a first pooling portion 326, a second pooling portion 327, and an amplifiering portion 328. Here, when the intermediate layer 323 includes two or more convolutional layers 324, it is preferable that the number of filters 325 in those convolutional layers 324 is equal to each other.

In this figure, the intermediate layer 323 marked "AxB" is composed of B convolutional layers 324, which means that each convolutional layer 324 includes A convolutional filters for each channel. .. Such an intermediate layer 323 will also be referred to as an "A × B intermediate layer" below. For example, the 64 × 2 intermediate layer 323 consists of two convolutional layers 324, which means that each convolutional layer 324 includes 64 convolutional filters for each channel.

In the example of this figure, the feature extraction unit 321 includes a 64 × 2 intermediate layer 323, a 128 × 2 intermediate layer 323, a 256 × 3 intermediate layer 323, and a 512 × 3 intermediate layer 323 in this order. Further, the upsampling unit 322 includes 512 × 3 intermediate layer 323, 256 × 3 intermediate layer 323, 128 × 2 intermediate layer 323, and 64 × 2 intermediate layer 323 in this order. Further, the second pooling unit 327 connects two 512 × 3 intermediate layers 323 to each other. The number of intermediate layers 323 constituting the nonlinear mapping unit 320 is not particularly limited, and can be determined, for example, according to the number of pixels of the image data.

Note that this figure is an example of the configuration of the nonlinear mapping unit 320, and the nonlinear mapping unit 320 may have another configuration. For example, a 64x1 intermediate layer 323 may be included instead of the 64x2 intermediate layer 323. By reducing the number of convolutional layers 324 included in the intermediate layer 323, the calculation cost may be further reduced. Further, for example, a 32 × 2 intermediate layer 323 may be included instead of the 64 × 2 intermediate layer 323. By reducing the number of channels in the intermediate layer 323, the calculation cost may be further reduced. Further, both the number of convolutional layers 324 and the number of channels in the intermediate layer 323 may be reduced.

Here, in the plurality of intermediate layers 323 included in the feature extraction unit 321, it is preferable that the number of filters 325 increases each time the first pooling unit 326 is passed. Specifically, the first intermediate layer 323a and the second intermediate layer 323b are continuous with each other via the first pooling portion 326, and the second intermediate layer is behind the first intermediate layer 323a. 323b is located. The first intermediate layer 323a is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N1, and the second intermediate layer 323b is a convolutional layer in which the number of filters 325 for each channel is N2. It is composed of layers 324. At this time, it is preferable that N2> N1 holds. Further, it is more preferable that N2 = N1 × 2 holds.

Further, in the plurality of intermediate layers 323 included in the upsample unit 322, it is preferable that the number of filters 325 decreases each time the amplifier ring unit 328 is passed through. Specifically, the third intermediate layer 323c and the fourth intermediate layer 323d are continuous with each other via the amplifiering portion 328, and the fourth intermediate layer 323d is located after the third intermediate layer 323c. To position. The third intermediate layer 323c is composed of a convolutional layer 324 in which the number of filters 325 for each channel is N3, and the fourth intermediate layer 323d is a convolutional layer 323d in which the number of filters 325 for each channel is N4. It is composed of layers 324. At this time, it is preferable that N4 <N3 holds. Further, it is more preferable that N3 = N4 × 2 holds.

The feature extraction unit 321 extracts image features having a plurality of abstractions such as gradients and shapes from the intermediate data acquired from the input unit 310 as channels of the intermediate layer 323. FIG. 12 illustrates the configuration of the 64 × 2 intermediate layer 323. The processing in the intermediate layer 323 will be described with reference to this figure. In the example of this figure, the intermediate layer 323 is composed of a first convolutional layer 324a and a second convolutional layer 324b, and each convolutional layer 324 includes 64 filters 325. In the first convolutional layer 324a, convolutional processing using the filter 325 is performed on each channel of the data input to the intermediate layer 323. For example, when the image input to the input unit 310 is an RGB image, processing is performed on each of the three channels h0i (i = 1.3). Further, in the example of this figure, the filter 325 is 64 types of 3 × 3 filters, that is, a total of 64 × 3 types of filters. As a result of the convolution process, 64 results h0i, j (i = 1..3, j = 1..64) are obtained for each channel i.

Next, the activation process is performed on the output of the plurality of filters 325 in the activation unit 329. Specifically, for the corresponding result j of all channels, the activation process is applied to the sum of the corresponding elements. By this activation treatment, the result h1i (i = 1..64) of 64 channels, that is, the output of the first convolutional layer 324a is obtained as an image feature. The activation process is not particularly limited, but a process using at least one of a hyperbolic function, a sigmoid function, and a rectified linear function is preferable.

Further, the output data of the first convolutional layer 324a is used as the input data of the second convolutional layer 324b, and the same processing as that of the first convolutional layer 324a is performed on the second convolutional layer 324b, resulting in 64 channels h2i. (I = 1..64), that is, the output of the second convolutional layer 324b is obtained as an image feature. The output of the second convolutional layer 324b becomes the output data of the 64 × 2 intermediate layer 323.

Here, the structure of the filter 325 is not particularly limited, but a 3 × 3 two-dimensional filter is preferable. Further, the coefficient of each filter 325 can be set independently. In this configuration example, the coefficient of each filter 325 is stored in the storage unit 390, and the nonlinear mapping unit 320 can read it out and use it for processing. Here, the coefficients of the plurality of filters 325 may be determined based on the correction information generated and corrected by using machine learning. For example, the correction information includes the coefficients of the plurality of filters 325 as a plurality of correction parameters. The nonlinear mapping unit 320 can further use this correction information to convert the intermediate data into mapping data. The storage unit 390 may be provided in the saliency estimation unit 130, or may be provided outside the saliency estimation unit 130. Further, the nonlinear mapping unit 320 may acquire correction information from the outside via a communication network.

13 (a) and 13 (b) are diagrams showing an example of the convolution process performed by the filter 325, respectively. In both FIGS. 13 (a) and 13 (b), an example of 3 × 3 convolution is shown. The example of FIG. 13A is a convolution process using the closest element. The example of FIG. 13B is a convolution process using proximity elements having a distance of two or more. It should be noted that a convolution process using proximity elements having a distance of three or more is also possible. The filter 325 preferably performs a convolution process using proximity elements having a distance of two or more. This is because a wider range of features can be extracted and the estimation accuracy of saliency can be further improved.

The operation of the 64 × 2 intermediate layer 323 has been described above. The operation of the other intermediate layers 323 (128 × 2 intermediate layer 323, 256 × 3 intermediate layer 323, 512 × 3 intermediate layer 323, etc.) is also 64 ×, excluding the number of convolutional layers 324 and the number of channels. 2 The operation is the same as that of the intermediate layer 323. Further, the operation of the intermediate layer 323 in the feature extraction unit 321 and the operation of the intermediate layer 323 in the upsample unit 322 are the same as described above.

14 (a) is a diagram for explaining the processing of the first pooling unit 326, and FIG. 14 (b) is a diagram for explaining the processing of the second pooling unit 327. (C) is a diagram for explaining the processing of the amplifiering unit 328.

The data output from the intermediate layer 323 in the feature extraction unit 321 is input to the next intermediate layer 323 after the pooling process is performed for each channel in the first pooling unit 326. In the first pooling unit 326, for example, a non-overlapping pooling process is performed. FIG. 14A shows a process of associating four 2 × 2 elements 30 with one element 30 for an element group included in each channel. In the first pooling unit 326, such a correspondence is made for all the elements 30. Here, the four elements 30 of 2 × 2 are selected so as not to overlap each other. In this example, the number of elements in each channel is reduced to a quarter. As long as the number of elements in the first pooling unit 326 is reduced, the number of elements 30 before and after the association is not particularly limited.

The data output from the feature extraction unit 321 is input to the upsampling unit 322 via the second pooling unit 327. In the second pooling unit 327, overlap pooling is applied to the output data from the feature extraction unit 321. FIG. 14B shows a process of associating four 2 × 2 elements 30 with one element 30 while overlapping some elements 30. That is, in the repeated association, a part of the 2 × 2 four elements 30 in one association is also included in the 2 × 2 four elements 30 in the next association. The number of elements is not reduced in the second pooling unit 327 as shown in this figure. The number of elements 30 before and after being associated with the second pooling unit 327 is not particularly limited.

The method of each processing performed by the first pooling unit 326 and the second pooling unit 327 is not particularly limited, but for example, a mapping (max pooling) in which the maximum value of the four elements 30 is set as one element 30 or 4 An association (average pooling) in which the average value of one element 30 is set as one element 30 can be mentioned.

The data output from the second pooling unit 327 is input to the intermediate layer 323 in the upsample unit 322. Then, the output data from the intermediate layer 323 of the upsample unit 322 is input to the next intermediate layer 323 after the amplifiering process is performed for each channel in the amplifiering unit 328. FIG. 14C shows a process of expanding one element 30 to a plurality of elements 30. The method of enlargement is not particularly limited, and an example is a method of duplicating one element 30 into four 2 × 2 elements 30.

The output data of the last intermediate layer 323 of the upsample unit 322 is output as mapping data from the nonlinear mapping unit 320 and input to the output unit 330. In the output step S130, the output unit 330 generates and outputs saliency estimation information by performing, for example, normalization or resolution conversion on the data acquired from the nonlinear mapping unit 320. The saliency estimation information is, for example, an image (image data) in which the saliency as illustrated in FIG. 9B is visualized by a luminance value. Further, the saliency estimation information may be, for example, an image color-coded according to saliency such as a heat map, and a saliency region having a saliency higher than a predetermined standard is distinguished from other positions. The image may be marked as possible. Further, the prominence estimation information is not limited to the image, and may be a table or the like in which information indicating a prominent region is listed.

The saliency estimation information output from the output unit 330 is subjected to various computer vision processing such as image division, object recognition, and image classification inside the saliency estimation unit 130 or outside the saliency estimation unit 130. You may.

<Hardware configuration example>
FIG. 15 is a block diagram illustrating the hardware configuration of the saliency estimation device 10 shown in FIG. The saliency estimation device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission line for the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processors 1020 and the like to each other is not limited to the bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like. The storage device 1040 stores a program module that realizes each function of the saliency estimation device 10. When the processor 1020 reads each of these program modules into the memory 1030 and executes them, each function corresponding to the program module is realized.

The input / output interface 1050 is an interface for connecting the saliency estimation device 10 and various input / output devices.

The network interface 1060 is an interface for connecting the saliency estimation device 10 to the network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method of connecting the network interface 1060 to the network may be a wireless connection or a wired connection.

As described above, according to the present embodiment, the brightness of the frame image is changed by using the velocity information before the saliency estimation process. Therefore, the saliency estimation unit 130 can detect a region that is perceived as having high saliency when viewed by a moving person with high accuracy from each frame image constituting the moving image.

Further, the saliency estimation unit 130 includes a feature extraction unit 321 that extracts features from the intermediate data, and an upsampling unit 322 that upsamples the data generated by the feature extraction unit 321. Therefore, the prominence can be estimated with a small calculation cost.

The saliency estimation device 10 can also process a still image. Even in this case, the above-mentioned effect can be obtained.

(Second Embodiment)
FIG. 16 is a diagram showing a functional configuration of the saliency estimation device 10 according to the second embodiment. The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to the first embodiment, except that the speed estimation unit 140 is provided.

The speed estimation unit 140 estimates the moving speed by processing the moving image data input to the input unit 110. As the estimation algorithm of this movement speed, an existing algorithm such as using optical flow estimation can be used. Then, the speed estimation unit 140 outputs the estimated speed as speed information to the visual field setting unit 122.

The same effect as that of the first embodiment can be obtained by this embodiment as well. Further, since the speed estimation unit 140 generates the speed information, it is not necessary to input the speed information from the outside.

(Third Embodiment)
FIG. 17 is a diagram showing a functional configuration of the saliency estimation device 10 according to the third embodiment. The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to the second embodiment, except that the reference point setting unit 150 is provided.

The reference point setting unit 150 generates reference point information by processing at least one frame image of the moving image data input to the input unit 110.

For example, when the frame image includes a predetermined object, for example, an object that is easily noticed by a person such as a specific traffic sign, the reference point setting unit 150 sets the position of the object as the reference point. The detection of this object is performed, for example, by a feature matching process. The feature amount used here is stored in advance in the saliency estimation device 10.

Further, when the frame image includes a road and the length of the straight line portion of the road is equal to or longer than the reference point, the reference point setting unit 150 sets the vanishing point as the reference point. Existing algorithms such as the Hough transform can be used to detect the vanishing point.

Further, when the landscape indicated by the frame image satisfies a specific condition, the reference point setting unit 150 may set the reference point by performing processing according to the condition. For example, as shown in FIGS. 18A and 18B, when a road is included in the frame image and the road is curved, the road is more than the center (for example, the central separation line) of the road. The part located in the bending direction is set as the reference point.

The same effect as that of the second embodiment can be obtained by this embodiment as well. Further, since the reference point setting unit 150 is provided, it is not necessary to input the reference point information from the outside.

(Fourth Embodiment)
FIG. 19 is a diagram showing a functional configuration of the saliency estimation device 10 according to the fourth embodiment. The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to the first embodiment except that the trimming unit 160 is provided.

The trimming unit 160 acquires the generation conditions of the moving image data acquired by the input unit 110, and trims the frame image using the generation conditions. The correction unit 120 processes the frame image trimmed by the trimming unit 160. The generation condition input to the trimming unit 160 is, for example, the type of camera lens (wide-angle lens or fisheye lens) that generated the moving image data. Depending on the conditions for generating moving image data, the range of the landscape shown in the frame image may be wider than the field of view of a stationary person. The trimming unit 160 trims the frame image in order to match the range of the landscape reflected in the frame image with the field of view of a stationary person. For example, the trimming unit 160 stores in advance the range to be trimmed from the frame image according to the generation conditions.

The same effect as that of the first embodiment can be obtained by this embodiment as well. Further, the trimming unit 160 trims the frame image in order to match the range of the landscape reflected in the frame image with the field of view of a stationary person. Therefore, with even higher accuracy, it is possible to detect from the image a region that is felt to be highly prominent when viewed by a moving person.

The trimming unit 160 according to the present embodiment may be provided in the saliency estimation device 10 shown in the second or third embodiment.

(Fifth Embodiment)
The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to any one of the above-described embodiments except for the functional configuration of the saliency estimation unit 130.

FIG. 20 is a diagram illustrating the configuration of the prominence estimation unit 130 according to the present embodiment. The saliency estimation unit 130 according to the present embodiment is the same as the saliency estimation unit 130 according to the first embodiment except that an error calculation unit 340 and a correction unit 350 are further provided. The error calculation unit 340 uses the saliency estimation information generated for the image and the saliency measurement information indicating the saliency distribution actually measured for the image, and the saliency distribution indicated by the saliency estimation information. And the error from the saliency distribution shown by the saliency measurement information is calculated. Then, the correction unit 350 corrects the correction information based on the calculated error.

The prominence estimation unit 130 according to the present embodiment performs an estimation operation and a learning operation. In the estimation operation, saliency estimation information for the input image is generated and output. The estimated operation is the operation as described in the first embodiment. In particular, in the present embodiment, the nonlinear mapping unit 320 converts the intermediate data into mapping data using the correction information. On the other hand, in the learning operation, machine learning is performed using the teacher image and the saliency actual measurement information for the teacher image, and the correction information is generated or corrected (updated). The correction information is information used by the nonlinear mapping unit 320, and includes, for example, a plurality of correction parameters.

In the present embodiment, the nonlinear mapping unit 320 converts the intermediate data into mapping data using the correction information. Correction information is information that has been generated and modified using machine learning. Specifically, the nonlinear mapping unit 320 includes a plurality of filters 325 as described in the first embodiment, and the coefficients of the plurality of filters 325 are determined based on the correction information. For example, the correction information includes the coefficients of the plurality of filters 325 as a plurality of correction parameters.

FIG. 21 is a flowchart illustrating a learning operation according to the present embodiment. The learning operation will be described in detail below. For the learning operation, a teacher image and saliency measurement information for the teacher image are prepared. For example, the teacher image and the saliency measured information are associated with each other and stored in the storage unit 390. The input unit 310 and the error calculation unit 340 can read and use this information from the storage unit 390.

The teacher image is an arbitrary image such as a photograph. Then, the saliency measurement information is generated, for example, based on the result of actually measuring the line of sight when a person looks at the teacher's image using an eye tracker. The saliency measurement information can have the same form as the saliency estimation information. That is, the saliency actual measurement information may be an image in which the saliency is visualized by a luminance value, and the saliency actual measurement information may be an image color-coded according to the saliency such as a heat map. ..

In the learning operation, the input step S110, the nonlinear mapping step S120, and the output step S130 are performed in the same manner as the input step S110, the nonlinear mapping step S120, and the output step S130 according to the first embodiment. However, the image acquired by the input unit 310 in the input step S110 is a teacher image. Further, in the nonlinear mapping step S120, the nonlinear mapping unit 320 reads the correction information from the storage unit 390. Then, the intermediate data is converted into mapping data using the correction information. The nonlinear mapping unit 320 may directly acquire the correction information from the correction unit 350 instead of reading the correction information from the storage unit 390. Further, in the initial state, the correction parameter included in the correction information can be an arbitrary value.

Next, in the error calculation step S140, the error calculation unit 340 acquires the prominence estimation information from the output unit 330. In addition, the error calculation unit 340 acquires the saliency measurement information associated with the teacher image that is the source of the saliency estimation information. Then, the error calculation unit 340 calculates the error between the acquired saliency estimation information and the saliency measurement information. The method for calculating the error is not particularly limited, but it is preferable to calculate at least one of, for example, L1 distance, L2 distance (Euclidean distance, mean square error), Kullback-Leibler distance, Jensen-Shannon distance, and Pearson correlation coefficient. ..

Specifically, the Euclidean distance is calculated by the following formula (1), the Kullback-Leibler distance is calculated by the following formula (2), and the Jensen-Shannon distance is calculated by the following formula (3). Here, pi indicates an estimation result (value based on saliency estimation information), and qi indicates a true value (value based on saliency measurement information).

Next, in the correction step S150, the correction unit 350 acquires an error from the error calculation unit 340, and corrects the correction parameter so that this error becomes small. Then, the correction parameter held in the storage unit 390 is replaced with the corrected correction parameter. Here, the correction method of the correction parameter is not particularly limited, and for example, at least one of the least squares method, the quadratic programming method, the stochastic gradient descent (SGD), the adaptive moment estimation (ADAM), and the variational method is used. Is preferable.

Here, there are many correction parameters to be corrected, and statistical learning (machine learning) using a large number of teacher data can be used to efficiently determine those values and estimate the prominence with high accuracy. preferable. Therefore, in the learning operation, it is preferable that machine learning is performed by the cooperation of the nonlinear mapping unit 320, the error calculation unit 340, and the correction unit 350.

The correction unit 350 may output the corrected correction parameter directly to the nonlinear mapping unit 320 instead of replacing the correction parameter held in the storage unit 390 with the corrected correction parameter. In the next nonlinear mapping step S120, the nonlinear mapping unit 320 performs processing using the corrected correction parameters.

The saliency actual measurement information associated with one teacher image may be one or a plurality. When a plurality of saliency measurement information is associated with one teacher image, the plurality of saliency measurement information is information based on different measurement results. Then, the error calculation unit 340 calculates the error between the saliency estimation information and each saliency measurement information. Further, the correction unit 350 corrects the correction parameter so that the total of all the errors becomes small, for example.

The learning operation may be performed on a plurality of sets of the teacher image and the measured saliency information. By repeating the learning operation, the estimation accuracy of saliency is further improved.

The timing at which the learning operation is performed is not particularly limited. For example, the saliency estimation unit 130 can accept an operation to start a learning operation by the user. Then, the saliency estimation unit 130 can start the learning operation based on the operation to start the learning operation. Further, the saliency estimation unit 130 can end the learning operation based on a user's end operation or a predetermined end condition. The end condition includes, for example, satisfying a predetermined number of repetitions of the learning operation, or having an error of less than or equal to a predetermined reference value.

As described above, according to the present embodiment, as in the first embodiment, the nonlinear mapping unit 320 has a feature extraction unit 321 that extracts features from the intermediate data and an upsample of the data generated by the feature extraction unit 321. It is provided with an upsampling unit 322 for performing the above. Therefore, the prominence can be estimated with a small calculation cost.

In addition, according to the present embodiment, the prominence estimation unit 130 includes an error calculation unit 340 and a correction unit 350. Therefore, more accurate saliency estimation is realized by using the correction information corrected by the learning operation.

(Sixth Embodiment)
The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to any one of the above-described embodiments except for the functional configuration of the saliency estimation unit 130.

FIG. 22 is a diagram illustrating the configuration and usage environment of the arithmetic unit 40 according to the present embodiment. The arithmetic unit 40 according to the present embodiment is a device that generates correction information used by the saliency estimation unit 130. The arithmetic unit 40 includes an error calculation unit 440 and a correction unit 450. The error calculation unit 440 uses the saliency estimation information generated for the teacher image and the saliency measurement information indicating the saliency distribution actually measured for the teacher image, and the saliency estimation information indicates. Calculate the error between the saliency distribution and the saliency distribution indicated by the saliency measurement information. The correction unit 450 calculates correction information based on the error.

The saliency estimation unit 130 according to the present embodiment is the same as the saliency estimation unit 130 according to the first embodiment. The saliency estimation unit 130 according to the present embodiment includes an input unit 310, a nonlinear mapping unit 320, and an output unit 330. Further, the prominence estimation unit 130 according to the present embodiment does not have to include the error calculation unit 340 and the correction unit 350 described in the fifth embodiment. The input unit 310 according to the present embodiment is the same as the input unit 310 according to at least one of the first and fifth embodiments, and the non-linear mapping unit 320 according to the present embodiment is the first and fifth embodiments. It is the same as the non-linear mapping unit 320 according to at least one of them, and the output unit 330 according to the present embodiment is the same as the output unit 330 according to at least one of the first and fifth embodiments. The operation of the error calculation unit 440 according to the present embodiment is the same as the operation of the error calculation unit 340 according to the fifth embodiment, and the operation of the correction unit 450 according to the present embodiment is the operation of the correction unit 450 according to the fifth embodiment. It is the same as the operation of 350. The saliency estimation unit 130 and the arithmetic unit 40 cooperate with each other to perform the learning operation and the estimation operation described in the fifth embodiment. Further, the saliency estimation unit 130 and the arithmetic unit 40 may be physically separated from each other, and may be connected to each other via, for example, a communication network.

Further, in the learning operation according to the present embodiment, it is preferable that machine learning is performed by the cooperation of the nonlinear mapping unit 320, the error calculation unit 440, and the correction unit 450.

The output unit 330 may temporarily store the generated saliency estimation information in the storage unit 390, and the error calculation unit 440 may read out and use the saliency estimation information stored in the storage unit 390.

In the example of this figure, the storage unit 390 is provided separately from the saliency estimation unit 130 and the arithmetic unit 40, but the present invention is not limited to this, and the storage unit 390 may be provided in the saliency estimation unit 130. , May be provided in the arithmetic unit 40. When the storage unit 390 is provided inside the arithmetic unit 40, for example, the storage unit 390 is realized by using the storage device 1080 of the computer 1000 that realizes the arithmetic unit 40. Further, the storage unit 390 may be formed in collaboration with the storage device 1080 of the computer 1000 that realizes the saliency estimation unit 130 and the storage device 1080 of the computer 1000 that realizes the arithmetic unit 40.

As described above, according to the present embodiment, as in the first embodiment, the nonlinear mapping unit 320 has a feature extraction unit 321 that extracts features from the intermediate data and an upsample of the data generated by the feature extraction unit 321. It is provided with an upsampling unit 322 for performing the above. Therefore, the prominence can be estimated with a small calculation cost.

In addition, according to the present embodiment, the arithmetic unit 40 includes an error calculation unit 440 and a correction unit 450. Therefore, more accurate saliency estimation is realized by using the correction information corrected by the learning operation.

(7th Embodiment)
The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to any one of the above-described embodiments except for the functional configuration of the saliency estimation unit 130.

FIG. 23 is a diagram illustrating the configuration of the prominence estimation unit 130 according to the present embodiment. The saliency estimation unit 130 according to the present embodiment is the same as the saliency estimation unit 130 according to at least one of the first and fifth embodiments, except that the synthesis unit 360 and the display unit 380 are further provided.

The synthesizing unit 360 generates synthesizing information obtained by synthesizing the saliency distribution indicated by the saliency estimation information and the image (input image) input to the input unit 310. Specifically, the synthesis unit 360 acquires the prominence estimation information from the output unit 330, and acquires the input image from, for example, the storage unit 390. Then, the composite information showing the input image and the saliency distribution together is output. The composite information is output to, for example, the display unit 380 provided in the saliency estimation unit 130. Further, the synthesis information output from the synthesis unit 360 may be held in the storage unit 390 or may be acquired by an external device.

FIG. 24 is a diagram illustrating an image indicated by the composite information generated by the composite unit 360. In the example of this figure, the composite information is an image in which the input image and the heat map showing the remarkableness are superimposed. The format of the composite information is not particularly limited. The composite information may be, for example, an image in which a prominent region is surrounded by a circle or a square in the input image. Further, the synthesis method is not particularly limited, and examples thereof include α blend.

The prominence estimation unit 130 according to the present embodiment can be mounted on, for example, a mobile terminal (smartphone, tablet, etc.) equipped with an imaging device such as a camera. By doing so, it is possible to extract an important object with high prominence on the spot and visualize it with good visibility while taking a picture with a mobile terminal.

As described above, according to the present embodiment, as in the first embodiment, the nonlinear mapping unit 320 has a feature extraction unit 321 that extracts features from the intermediate data and an upsample of the data generated by the feature extraction unit 321. It is provided with an upsampling unit 322 for performing the above. Therefore, the prominence can be estimated with a small calculation cost.

In addition, according to the present embodiment, the prominence estimation unit 130 further includes a synthesis unit 360. Therefore, the prominence at each position of the image can be visualized with good visibility.

(8th Embodiment)
The saliency estimation device 10 according to the present embodiment has the same configuration as the saliency estimation device 10 according to any one of the above-described embodiments except for the functional configuration of the saliency estimation unit 130.

FIG. 25 is a diagram illustrating the configuration of the prominence estimation unit 130 according to the present embodiment. The prominence estimation unit 130 according to the present embodiment is at least one of the first, fifth, and seventh embodiments except that the mask image generation unit 370, the region extraction unit 372, and the object detection unit 374 are further provided. It is the same as the saliency estimation unit 130 according to the above.

The mask image generation unit 370 acquires the prominence estimation information from the output unit 330 and generates a mask image. Specifically, in the saliency distribution indicated by the saliency estimation information, the mask image generation unit 370 sets a region whose saliency is lower than a predetermined standard as a mask region, and the saliency is equal to or higher than the predetermined standard. Generate a mask image with the area as a non-mask area. That is, the mask image generation unit 370 binarizes the saliency distribution. Here, the reference is set in advance and is held in the storage unit 390, and the mask image generation unit 370 can read and use it.

The area extraction unit 372 acquires an input image and a mask image. Then, by applying a mask image to the input image, a region having high prominence is extracted from the input image. For example, the area extraction unit 372 can extract a highly prominent region from the input image by performing a logical operation with the input image and the mask image.

Then, the object detection unit 374 detects an object from the region extracted by the region extraction unit 372. The method for detecting an object is not particularly limited, and examples thereof include a method using a Single Shot Multibox Detector (SSD). Since the prominence estimation unit 130 according to the present embodiment extracts a region with high prominence in advance and detects an object only in the extracted region, erroneous detection is suppressed.

The prominence estimation unit 130 according to the present embodiment is mounted on a moving body such as an automobile. Then, the detection result of the object by the object detection unit 374 can be used for automatic driving and driving support.

As described above, according to the present embodiment, as in the first embodiment, the nonlinear mapping unit 320 has a feature extraction unit 321 that extracts features from the intermediate data and an upsample of the data generated by the feature extraction unit 321. It is provided with an upsampling unit 322 for performing the above. Therefore, the prominence can be estimated with a small calculation cost.

In addition, according to the present embodiment, the saliency estimation unit 130 further includes a mask image generation unit 370, a region extraction unit 372, and an object detection unit 374. Therefore, it is possible to detect an object with high accuracy in the input image.

Although the embodiments and examples have been described above with reference to the drawings, these are examples of the present invention, and various configurations other than the above can be adopted.

10 Brightness estimation device 110 Input unit 120 Correction unit 122 Field of view setting unit 124 Correction information generation unit 126 Correction processing unit 130 Randomness estimation unit 140 Speed estimation unit 150 Reference point setting unit 160 Trimming unit 202 Resolution correction unit 204 Saturation correction unit 206 Brightness correction unit

Claims (9)

A correction unit that generates a corrected image by correcting the image of the scenery seen from the first viewpoint,
A saliency estimation unit that generates saliency estimation information indicating a saliency distribution in the corrected image or in the image by processing the corrected image, and a saliency estimation unit.
With
The correction unit
Brightness information indicating a change in the brightness of at least a part of the image is generated by using the speed information regarding the moving speed of the first viewpoint and the relative position from the reference point in the image to the at least a part.
A saliency estimation device that generates the saliency estimation information based on a corrected image in which at least a part of the brightness is corrected using the brightness information. In the saliency estimation device according to claim 1,
The correction unit
Further, the resolution information indicating the change in at least a part of the resolution and the saturation information indicating the change in the saturation of at least a part are generated by using the speed information and the relative position.
A prominence estimation device that corrects the resolution and brightness of at least a part of the image by using the resolution information and the saturation information. In the saliency estimation device according to claim 1 or 2.
The correction unit is a saliency estimation device that generates the brightness information so that the brightness decreases as the distance from the reference point in the image to at least a part of the image increases. In the saliency estimation device according to any one of claims 1 to 3,
The saliency estimation device, wherein the image is a frame image included in a moving image. In the saliency estimation device according to any one of claims 1 to 4.
The correction unit is a saliency estimation device that sets the reference point by processing the image. In the saliency estimation device according to any one of claims 1 to 5.
A trimming unit for trimming the image using the image generation conditions is provided.
The correction unit is a saliency estimation device that processes the trimmed image to generate the corrected image. In the saliency estimation device according to any one of claims 1 to 6.
The saliency estimation unit is a saliency estimation device that generates saliency estimation information by inputting the corrected image into a model generated by machine learning. The computer
A corrected image is generated by correcting the image of the scenery seen from the first viewpoint.
By processing the corrected image, saliency estimation information indicating the saliency distribution in the corrected image or in the image is generated.
Furthermore, the computer
Brightness information indicating a change in the brightness of at least a part of the image is generated by using the speed information regarding the moving speed of the first viewpoint and the relative position from the reference point in the image to the at least a part.
A saliency estimation method that generates the saliency estimation information based on a corrected image in which at least a part of the brightness is corrected using the brightness information. On the computer
A correction function that generates a corrected image by correcting the image of the scenery seen from the first viewpoint,
An estimation function that generates saliency estimation information indicating a saliency distribution in the corrected image or in the image by processing the corrected image, and an estimation function.
To have
Furthermore, as at least a part of the correction function,
A function to generate brightness information indicating a change in brightness of at least a part of the image by using speed information regarding the moving speed of the first viewpoint and a relative position from a reference point in the image to at least a part of the image. When,
A function of generating the saliency estimation information based on a corrected image in which at least a part of the brightness is corrected using the brightness information, and
Program to have.