JP2022177579A - Learning device, estimation device, learning method, estimation method, and program - Google Patents

Abstract

To provide a technique for estimating a focus-of-attention region that is a region in a space and is in a region that does not appear in an image or video.SOLUTION: A learning device includes a control unit that updates a mathematical model indicating a relation between image data and a focus-of-attention region in a space that does not appear in an image indicated by the image data, based on training data that is a pair of image data and correct focus-of-attention region data that is data indicating a focus-of-attention region in a space that does not appear in an image indicated by the image data. The control unit estimates a focus-of-attention region based on image data included in training data by using the mathematical model, and updates the mathematical model so as to make an estimation result closer to the focus-of-attention region indicated by correct focus-of-attention region data included in the training data.SELECTED DRAWING: Figure 6

Description

The present invention relates to a learning device, a guessing device, a learning method, a guessing method, and a program.

Techniques have been developed for predicting a focus of attention (FoA), which is a region within a space such as a scene, to which a human tends to look. A typical example is a technique called saliency estimation that detects a region of interest in an image or video by analyzing the image or video. It has been studied for many years in the technical fields of robot vision and computer vision, and has been used in a wide variety of fields such as object recognition, image or video editing, image coding, image quality evaluation, autonomous driving, and online marketing.

The social implementation of AI (artificial intelligence) technology is progressing, but the ability to perceive or perceive the real world in the same way as humans does is one of the basic requirements for AI technology, and predict the gaze area. Technology is one of the underlying technologies. In particular, AI robots and AI agents, which act in the real world like humans, are expected to perceive the outside world by inputting images or videos from a first-person viewpoint captured by a built-in camera. Therefore, there are high expectations for technology that predicts the region of interest for an image or video from a first-person viewpoint. To simplify the explanation, the technology for predicting the region of interest will be described using an image as an example. However, since a video is a collection of a plurality of images, the following description also applies to videos instead of images.

With the recent development of deep learning, great progress has also been made in saliency estimation based on first-person view images, and excellent prediction performance has been achieved. For example, Non-Patent Document 1 proposed a model based on a three-dimensional convolutional network that can accurately estimate the gaze region in the first-person viewpoint image in an autonomous driving scenario.

In addition, Non-Patent Document 2 focuses on the difference in the gaze area depending on age and uses image conversion technology to convert the result of estimating the gaze area by adults into the result of estimating the gaze area by elderly people. proposed the technique. This technology also targets autonomous driving scenarios and first-person perspective images from the perspective of pedestrians, and enables highly accurate saliency prediction.

Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara, “Predicting the drivers focus of attention: the dr(eye)ve project”, IEEE transactions on pattern analysis and machine intelligence, 2018. Onkar Krishna, Go Irie, Takahito Kawanishi, Kunio Kashino, and Kiyoharu Aizawa, “Translating Adult’s Focus of Attention to Elderly’s”, In Proceedings of International Conference on Pattern Recognition, 2020.

However, these techniques are designed to estimate the region of interest captured in the image, and are techniques for estimating the region of interest captured in the image. Humans who are active in the real world do not always pay attention to observable areas in the scene. Humans sometimes predict or judge an invisible area, such as the back or the other side of a wall, to which they should pay attention, and take preparatory actions or actions to avoid danger.

Therefore, AI systems that work together with humans in the real world are expected to have functions similar to those of humans. Specifically, it is expected that the AI system will have a function of predicting a region of interest in a region not captured in the image.

However, existing techniques for estimating image salience are all implemented as techniques for estimating salient areas only in areas that are visible in the image, and are not techniques for predicting attention areas for areas that are not visible in the image. . As mentioned above, this also applies to video.

SUMMARY OF THE INVENTION In view of the above circumstances, the present invention aims to provide a technology for predicting a region of interest in a region in space that does not appear in an image or video.

One aspect of the present invention is based on training data, which is paired data of image data and correct gaze region data, which is data indicating a gaze region in a space not captured in the image represented by the image data. a control unit that updates a mathematical model that indicates a relationship with a gaze region in a space that is not captured in the image indicated by the image data, wherein the control unit updates the mathematical model based on image data included in training data using the mathematical model; The learning device estimates a gaze area, and updates the mathematical model so that the result of estimation approaches the gaze area indicated by correct gaze area data included in the training data.

One aspect of the present invention is based on training data, which is paired data of image data and correct gaze region data, which is data indicating a gaze region in a space not captured in the image represented by the image data. a control unit that updates a mathematical model that indicates a relationship with a gaze region in a space that is not captured in the image indicated by the image data, wherein the control unit updates the mathematical model based on image data included in training data using the mathematical model; The mathematical model is updated until a predetermined termination condition is satisfied by a learning device that estimates a gaze area and updates the mathematical model so that the result of estimation approaches the gaze area indicated by the correct gaze area data included in the training data. and a control unit for estimating, based on input image data, a gaze region in a space that is not captured in an image represented by the image data, using the updated mathematical model, which is the mathematical model.

One aspect of the present invention is based on training data, which is paired data of image data and correct gaze region data, which is data indicating a gaze region in a space not captured in the image represented by the image data. a control step of updating a mathematical model that indicates a relationship with a region of interest in a space that is not captured in the image indicated by the image data; a gaze area is estimated based on the training data, and the mathematical model is updated so that the result of the estimation approaches the gaze area indicated by the correct gaze area data included in the training data.

One aspect of the present invention is based on training data, which is paired data of image data and correct gaze region data, which is data indicating a gaze region in a space not captured in the image represented by the image data. a control unit that updates a mathematical model that indicates a relationship with a gaze region in a space that is not captured in the image indicated by the image data, wherein the control unit updates the mathematical model based on image data included in training data using the mathematical model; The mathematical model is updated until a predetermined termination condition is satisfied by a learning device that estimates a gaze area and updates the mathematical model so that the result of estimation approaches the gaze area indicated by the correct gaze area data included in the training data. a control step of estimating, based on input image data, a gaze region in a space that does not appear in an image represented by the image data, using the updated mathematical model, which is the mathematical model.

One aspect of the present invention is a program for causing a computer to function as the learning device.

According to the present invention, it is possible to predict a region of interest in a region in space that is not captured in an image or video.

The figure which shows an example of a structure of the prediction system 100 of embodiment. Explanatory drawing explaining the 1st example of the prediction result expression format in embodiment. FIG. 9 is an explanatory diagram for explaining a second example of the prediction result representation format in the embodiment; FIG. 9 is an explanatory diagram for explaining a third example of the prediction result expression format in the embodiment; FIG. 9 is an explanatory diagram for explaining a fourth example of the prediction result expression format in the embodiment; The figure which shows an example of a structure of the predictor 10 in embodiment. 4 is a flowchart showing an example of the flow of processing executed by the predictor 10 in the embodiment; 4 is a flowchart showing an example of the flow of enhanced prediction processing according to the embodiment; 4 is a flowchart showing an example of the flow of reward acquisition processing according to the embodiment; 1 is a diagram showing an example of a hardware configuration of a learning device 1 according to an embodiment; FIG. The figure which shows an example of the functional structure of the control part 11 in embodiment. The figure which shows an example of the hardware constitutions of the prediction apparatus 2 in embodiment. The figure which shows an example of the functional structure of the control part 21 in embodiment.

(embodiment)
Drawing 1 is a figure showing an example of composition of prediction system 100 of an embodiment. The prediction system 100 includes a learning device 1 and a prediction device 2 . For simplicity of explanation, the prediction system 100 will be described below by taking as an example a case where the data input to the learning device 1 and the prediction device 2 are image data. However, since a video is a time series of images, the data input to the learning device 1 and the prediction device 2 may be video data, which is video data, instead of image data. To simplify the explanation, the prediction system 100 will be described with an example in which image data is input to the learning device 1 and the prediction device 2. However, the learning device 1 and the prediction device 2 receive video data instead of image data. may be entered.

The learning device 1 receives input of image data. The learning device 1 updates the prediction model by a machine learning method based on the input image data. The prediction model is a mathematical model that represents the relationship between input image data and a focus of attention (FoA) in space that is not captured in the image represented by the input image data. The gaze area in the space that does not appear in the image indicated by the image data is, in other words, the gaze area in the space outside the space indicated by the image of the image data. A gaze area is an area within a space (hereinafter referred to as "intraspace area") to which a human tends to look.

The prediction device 2 receives input of image data. The prediction device 2 uses the learned prediction model acquired by the learning device 1 and predicts the gaze region based on the input image data. The prediction device 2 includes a predictor control section 211 . Although the details of the predictor control unit 211 will be described later, the predictor control unit 211 controls the operation of a circuit that expresses a learned prediction model.

Note that "learning completed" means that learning has been executed until a predetermined end condition (hereinafter referred to as "learning end condition") is satisfied. Therefore, a trained mathematical model is a mathematical model at the time when the learning termination condition is satisfied. The learning termination condition may be, for example, a condition that learning has been performed a predetermined number of times, and the learning termination condition may be, for example, a condition that the change in the learning model due to learning is less than a predetermined change.

A mathematical model is a set including one or a plurality of processes whose execution conditions and order are predetermined. A process included in the mathematical model is, for example, a process of obtaining a function value by inputting a value into a predetermined function.

Note that learning means updating the mathematical model. Updating the mathematical model means updating the parameter values of the circuit representing the mathematical model. At least part of the processing included in the mathematical model is represented by, for example, a neural network. A neural network is an example of a circuit such as an electronic circuit, an electric circuit, an optical circuit, an integrated circuit, or the like, which expresses at least part of the processing of a mathematical model. Updating the mathematical model by learning means updating the parameter values of the circuit representing the mathematical model. If the part of the circuit representing the mathematical model is a neural network, the parameters of the neural network are preferably adjusted based on predefined quantities. A predefined quantity is, for example, a predefined objective function value (ie loss).

<Example of expression format of prediction result>
A description will be given of an expression format (hereinafter referred to as "prediction result expression format") for expressing the learned prediction model or the prediction result of the prediction model.

FIG. 2 is an explanatory diagram for explaining a first example of the prediction result representation format in the embodiment. FIG. 2 is an example of a prediction result expression format that expresses a gaze area of a prediction result as the position of a point that easily attracts attention (hereinafter referred to as a "gazing point"). FIG. 2 shows an image that is 2H+1 pixels high and 2W+1 pixels wide. In FIG. 2, for example, if the pixel at the center of the image is expressed as (0, 0), the position (that is, coordinates) of any pixel in the image is expressed as (x, y) using horizontal position x and vertical position y. expressed.

The prediction result expression format in FIG. 2 can express the point of interest by coordinates even when the point of interest exists outside the space captured in the image in FIG. That is, even if the x-coordinate of the gaze point is outside the range of −W<x<W and the y-coordinate is outside the range of −H<y<H, the prediction result representation format in FIG. A viewpoint can be represented by coordinates. The x-axis and the y-axis are predetermined coordinate axes orthogonal to each other.

FIG. 2 shows a point located at a position (2W, 0) that is 2W pixels to the right from the center as an example of the gaze point. Since it is 2W to the right of the center, the point located at location (2W, 0) is outside the extent of the image.

FIG. 3 is an explanatory diagram illustrating a second example of the prediction result representation format in the embodiment. FIG. 3 shows an example of a prediction result expression format that expresses the position of the gaze point of the prediction result using the distance r from the center and the angle θ (that is, polar coordinates). In addition, in the prediction result expression format of FIG. 3, the distance r does not necessarily need to be used in expressing the position of the gaze point. In the prediction result expression format of FIG. 3, the position of the gaze point may be expressed only by the angle θ (that is, the gaze direction).

FIG. 4 is an explanatory diagram illustrating a third example of the prediction result representation format in the embodiment. The example of FIG. 4 is an example of a prediction result expression format that discretizes the positions of points and expresses them using the discretization result. The discretization method is, for example, a method of dividing the inner and outer planes of the image into 24 regions in units of W×H, as shown in FIG. Each region after division is hereinafter referred to as a discretized region. Since the inner and outer planes of the image were divided into 24 regions, the size of each discretized region is W×H. Each discretized region is given one of the identifiers of the number of divisions from 1 to the number of divisions, and an arbitrary point is represented by the value of the identifier of each discretized region to which it belongs.

For example, the point A in FIG. 4 belongs to the discretized region with the identifier of 5, so it is output as "5". In a discretization area to which no identifier is assigned, a point is represented by, for example, the identifier of the nearest discretization area among the discretization areas to which identifiers are assigned. In the prediction result representation format, if the discretized region to which point A belongs is a discretized region to which no identifier is assigned, the discretized region to which point A belongs is given an identifier, for example, with the information "no corresponding region". It may be expressed as a discretized region that is not discretized. Note that the size of each discretized region does not necessarily have to be W×H. Also, the number of discretized regions does not necessarily have to be 24. The size of each discretized region and the number of discretized regions may be determined in advance by the user as appropriate according to the scene in which the prediction system 100 is applied.

FIG. 5 is an explanatory diagram illustrating a fourth example of the prediction result representation format in the embodiment. FIG. 5 is an example of the output representing the gaze area as a distribution. FIG. 5 shows an example of output in which the region of interest is represented as an isotropic two-dimensional normal distribution with mean (W, H) and variance ^σ2 . Dispersion need not necessarily be one-way. The dispersion may be different in the x-axis direction and the y-axis direction. Also, the distribution does not necessarily have to be a normal distribution. The distribution may be any probability distribution that can indicate the location and extent of the gaze area. The distribution may for example be a probability distribution.

The prediction result representation format is not limited to the examples shown in FIGS. 2 to 5, and can represent a trained prediction model or a prediction result by a prediction model, and can represent at least a spatial region in a space outside the space shown in the image. Any form of expression may be used.

Returning to the description of FIG. The learning device 1 may update the prediction model by any method as long as the prediction model can be learned. A prediction model may be represented using a neural network, for example. A circuit representing a prediction model is hereinafter referred to as a predictor 10 .

The learning device 1 updates the prediction model by updating the parameters of the predictor 10 . The parameters of the predictor 10 when the learning end condition is satisfied are sent to the prediction device 2 . The prediction device 2 includes a circuit similar to that of the predictor 10 and operates the circuit using the parameters acquired from the learning device 1 . Thereby, the prediction device 2 executes the learned prediction model acquired by the learning device 1 . Therefore, a circuit similar to the predictor 10 included in the prediction device 2 is a circuit to be controlled by the predictor control section 211 . Hereinafter, a circuit provided in the prediction device 2 and similar to the predictor 10 is also referred to as the predictor 10 for the sake of simplicity of explanation.

The details of how the learning device 1 acquires a learned prediction model will be described later, but first, the predictor 10 will be described to explain how the learning device 1 acquires a learned prediction model.

<Description of Predictor 10>
The predictor 10 is a circuit that expresses a prediction model, and is capable of outputting the prediction result of the prediction model in a predetermined prediction result expression format (i.e., the gaze region in space that does not appear in the image indicated by the image data). Any circuit may be used. That is, the predictor 10 predicts a region of interest in a space that does not appear in the image indicated by the image data based on the image data, and outputs the prediction result in a predetermined prediction result expression format. may be The predictor 10 is, for example, a convolutional neural network (CNN), a long short term memory (LSTM) network, and a circuit configured by two fully connected layers (FC) is.

When the predictor 10 is a circuit composed of a CNN, an LSTM, and two fully connected layers, the network structure of the CNN constituting the predictor 10 is such that the predictor 10 can express a prediction model and is predetermined Any network structure may be used as long as the result can be output in a prediction result expression format. The CNN that configures the predictor 10 may be, for example, ResNet described in Reference 1 below. ResNet is suitable for the prediction system 100 because image data can be directly input and acquired.

Reference 1: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

For simplicity of explanation, the prediction system 100 will be described by taking as an example the case where the predictor 10 is a circuit composed of a CNN, an LSTM, and two fully connected layers.

FIG. 6 is a diagram showing an example of the configuration of the predictor 10 in the embodiment. The predictor 10 shown in FIG. 6 comprises CNN 101, LSTM 102, prediction FC 103 and stop decision FC 104. FIG. Image data is input to the CNN 101 . The CNN 101 is a convolutional neural network (CNN), and acquires feature amounts (hereinafter referred to as "image features") of input image data based on the input image data.

The LSTM 102 is a long-term memory network, and acquires area features based on the image features and auxiliary information acquired by the CNN 101 . The area feature is an output of the LSTM 102 and is input to the prediction FC 103 and the stop judgment FC 104 positioned after the LSTM 102 . Region features are therefore a kind of intermediate output. The area feature is information including at least part of the information indicated by the image feature and the auxiliary information, and at least part of the information indicated by the history of past prediction results of the attention area. The type or amount of information included in the area information is updated by learning.

Auxiliary information is information that is input to the LSTM 102 . Auxiliary information indicates a value corresponding to the timing of input to the LSTM 102 . The value indicated by the auxiliary information is a value that follows a predetermined distribution with non-zero variance. The auxiliary information indicates, for example, a vector determined depending on the number of times it is input to the LSTM 102 . The auxiliary information may indicate, for example, area features obtained before the previous timing input. The auxiliary information may indicate, for example, a statistic of the distribution of a plurality of area features obtained before the previous timing input. The auxiliary information may indicate a vector (hereinafter referred to as "auxiliary vector") having the same number of dimensions d as the image feature and whose element values are defined by the following equations (1) and (2).

a(i, k) represents the value of the k-th element of the auxiliary vector at the i-th input. The i-th input time means the i-th timing at which the auxiliary information is input to the LSTM 102 . c is a constant. The value of constant c is 10000, for example.

For simplicity of explanation, the prediction system 100 will be described with an example in which the auxiliary information indicates an auxiliary vector.

<Effects of auxiliary information>
The effect of the auxiliary information will be explained. In order to explain the effect of the auxiliary information, first, the LSTM (that is, the LSTM 102) provided in the predictor 10 will be explained. The LSTM 102 receives as input image features, which are the outputs of the CNN 101, and outputs region features. The number of gaze areas in one input image is not necessarily one. Also, the number may change depending on the input image.

The predictor 10 outputs one or more attention regions by inputting image features to the LSTM 102 multiple times. An LSTM is a type of recursive neural network, and is a neural network that has state variables inside and whose output is determined based on both the input and state variables. The state variables of the LSTM are updated each time an input is received. Therefore, even if the same image feature is input to the LSTM 102, its output may change each time it is input. Therefore, even when image features obtained from the same image data are input multiple times, the predictor 10 using LSTM can predict a region of interest that is not necessarily the same for each input. .

When the auxiliary information is used, even if the input image feature is the same as the image feature at the time of past learning, information different from the information input to the predictor 10 in the past is input to the predictor 10 . As a result, even if the image feature input to the predictor 10 is the same as the image feature at the time of past learning, the frequency at which the predictor 10 outputs a result different from the past result increases. In this way, the auxiliary information has the effect of increasing the frequency of non-identical prediction results.

In this manner, the predictor 10 predicts, for example, a region of interest in a space that does not appear in the image indicated by the image data, based on the image data and the auxiliary information, and outputs the prediction result in a predetermined prediction result expression format. It is a circuit that Hereinafter, a process of predicting a region of interest in a space that does not appear in the image indicated by the image data based at least on the image data will be referred to as a prediction process.

Area features are input to the prediction FC 103 and stop determination FC 104 . The prediction FC 103 outputs a gaze area based on the input area feature. The output format of the prediction FC 103 (that is, the prediction result expression format) is a predetermined format. The prediction result representation format is a format predetermined by the user according to the area feature input to the prediction FC 103, for example.

The prediction FC 103 outputs two-dimensional coordinate values (x, y) when the gaze point is expressed as coordinate values as in the example shown in FIG. The prediction FC 103 outputs the probability that the region of interest exists in each discretized region when the positions of the points are represented by discretization as in the example shown in FIG. 4, for example. The prediction FC 103 outputs parameters of the distribution when the position and range of the attention area are expressed using a distribution as in the example shown in FIG. 5, for example. The distribution parameters output by the prediction FC 103 are, for example, the values of the average x and y and the value of the variance ^σ2 in the example of FIG.

The stop judgment FC 104 judges whether or not to stop the prediction process based on the area feature. More specifically, the stop determination FC 104 outputs stop determination information t, which is a binary value indicating whether or not to stop the prediction process, based on the area feature. A condition for the stop judgment FC 104 to stop the prediction process is obtained by learning. Obtained by learning means obtained by, for example, updating the conditions so that the loss function is minimized.

If the configuration of the predictor 10 is such that the image features are input to the LSTM multiple times to output one or more regions of interest, then one or more regions of interest are output. However, it is not obvious how many times the output should be obtained as described above. Therefore, the stop judgment FC 104 makes a judgment according to the situation based on the region feature. Contextual specifically means based on information generated during the course of the prediction process.

The number of times to obtain the output may be, for example, a predetermined number of times (hereinafter referred to as "maximum predicted number of times") T. In such a case, the stop determination FC 104 determines to stop the prediction process when the number of times the output is obtained reaches the maximum predicted number of times T, and when the number of times the output is obtained is less than the maximum predicted number of times T It is determined that the prediction process is not stopped at this time.

The appropriate number of predictions (that is, the number of gaze regions) differs for each image data input to the predictor 10 . Therefore, it is preferable to determine whether or not to stop the prediction process by the stop determination FC 104 based on the above-described area feature rather than using the predetermined maximum number of predictions T. FIG.

FIG. 7 is a flow chart showing an example of the flow of processing executed by the predictor 10 in the embodiment. Image data is input to CNN 101 (step S101). Image data is, for example, image data of an image captured by a camera. The image data may be, for example, image data of a frame extracted from a video, or may be image data of a frame of a video captured, for example, by an on-board camera of an automobile.

Next, the predictor control unit 211 sets the number i of times the execution of prediction processing is started to 1 (step S102). Setting the number i to 1 means initializing the auxiliary information. Next, based on the image data of the input image, the CNN 101 acquires the image feature f (step S103). Next, the LSTM 102 acquires area features based on the image feature f and the auxiliary information ai (step S104). Next, the prediction FC 103 predicts the region of interest based on the region features (step S105). Next, the prediction FC 103 outputs a prediction result Oi indicating the prediction result (step S106).

Next, the stop judgment FC 104 judges whether or not to stop the prediction process based on the area feature (step S107). Next, the stop judgment FC 104 outputs information indicating the result of the judgment (that is, stop judgment information ti) (step S108). Next, the predictor control unit 211 determines whether or not the prediction end condition is satisfied based on the stop determination information (step S109). The prediction end condition is a condition relating to the end of the prediction process and is based on at least stop determination information. The prediction end condition is, for example, either the condition that the stop determination information ti indicates stop or the condition that the number of times the execution of the prediction process is started is equal to or greater than the maximum prediction number of times T (that is, i<T). condition that it is satisfied.

If the predicted termination condition is met, the process ends. On the other hand, if the prediction end condition is not satisfied, the predictor control unit 211 updates the auxiliary information ai according to a predetermined update rule (step S110). After step S110, the process returns to step S104.

Note that step S106 may be executed at any timing as long as it is executed after step S105 and before step S109 is executed. Note that step S108 may be executed at any timing as long as it is executed after step S107 and before step S109 is executed. Note that the process of step S105 may be performed after the process of step S107 as long as it is after the process of step S104 and before the process of step S109.

Note that the maximum number of predictions T is any positive number, for example, T=5. The maximum number of predictions may be any positive number, but it is desirable that it be a value larger than the maximum number of correct fixation regions indicated by the teacher data used by the learning device 1 when learning the prediction model.

Thus, the predictor 10 performs image feature extraction processing, prediction processing, and stop determination processing. Image feature extraction processing is processing for acquiring image features based on image data. In the example of FIG. 7, this is the processing of step 103 . The prediction process is a process indicated by a series of steps S104, S105 and S106 in the example of FIG. The stop determination process is a process of determining whether or not to stop the prediction process based at least on the image data. The stop determination process is a process indicated by a series of steps S104, S107, and S108 in the example of FIG.

Through such a process, the predictor 10 can obtain a maximum of T prediction results through a maximum of T prediction processes. Hereinafter, the number of prediction results by the predictor 10 is represented as N.

<Method for Acquiring a Learned Prediction Model by the Learning Apparatus 1>
The learning device 1 learns a prediction model using training data. The training data is paired data of image data and data indicating a set of correct gaze areas (hereinafter referred to as "correct gaze area data"). A correct fixation area is a fixation area in a space that is not reflected in the image indicated by the corresponding image data. Therefore, the set of correct fixation regions is the set of fixation regions included in the corresponding image.

More specifically, the training data D is data including image data as learning data (that is, data on the explanatory variable side) and correct gaze region data as teacher data (that is, data on the objective variable side). The image indicated by the image data of the learning data may be an image including only one attention area, or may be an image including a plurality of attention areas. Hereinafter, image data of learning data will be referred to as image data I, and a set of correct fixation regions will be referred to as a set {Sj} (j=1, . . . , M). Therefore, the training data is the set D={(I, {Sj})}. Therefore, the training data will be referred to as training data D hereinafter.

The method of learning the prediction model may be any method as long as it is a method of learning using training data, and may be, for example, a method of reinforcement learning. An example of processing for learning a prediction model by reinforcement learning (hereinafter referred to as “prediction model reinforcement learning processing”) will be described below.

The prediction model reinforcement learning process includes a reinforcement prediction process, a reward acquisition process, a reinforcement prediction learning process, and a stop determination learning process. Before describing each process, an outline of reinforcement learning will be described.

Reinforcement learning is a learning method that learns the optimal action decision policy of an agent that decides actions under a certain situation. Reinforcement learning is a learning method that makes an agent learn by prescribing a reward depending on whether the final situation brought about as a result of a series of actions of the agent is desirable or not.

In the prediction model reinforcement learning process, the predictor 10 is used as an agent in reinforcement learning. In the prediction model reinforcement learning process, prediction of the region of interest and determination of termination of the prediction process are used as actions in reinforcement learning. In the prediction model reinforcement learning process, the image data I, the correct/wrong result of the past prediction, and the correct/wrong result of the stop determination are used as the situation in the reinforcement learning.

As described above, reinforcement learning is trial-and-error search-type learning. More specifically, in reinforcement learning, after obtaining a result using a mathematical model to be updated, a reward is calculated based on the obtained result, and the mathematical model to be updated is updated based on the reward. Therefore, in reinforcement learning, first, processing is performed to obtain a result using a mathematical model to be updated. Reinforcement prediction processing is the processing of obtaining results using a mathematical model to be updated in the prediction model reinforcement learning processing.

Therefore, the enhanced prediction process is a process that uses the predictor 10 to perform the prediction process and the stop determination process.

FIG. 8 is a flowchart illustrating an example of the flow of enhanced prediction processing in the embodiment. The learning control unit 111, which will be described later, acquires the training data D (step S201). Next, the learning control unit 111 sets the number of times i that execution of the reinforcement prediction process is started to 1 (step S202). Setting the number i to 1 means initializing the auxiliary information. Next, based on the image data of the input image, the CNN 101 acquires the image feature f (step S203). Next, the LSTM 102 acquires area features based on the image feature f and the auxiliary information ai (step S204). Next, the prediction FC 103 predicts the region of interest based on the region features (step S205). Next, the prediction FC 103 outputs a prediction result Oi indicating the prediction result. The output prediction result Oi is recorded by the learning control unit 111 in a predetermined storage device such as the storage unit 13 (described later) (step S206).

Next, the stop judgment FC 104 judges whether or not to stop the prediction process based on the area feature (step S207). Next, the stop judgment FC 104 outputs information indicating the result of the judgment (that is, stop judgment information ti). The output stop determination information ti is recorded in a predetermined storage device by the learning control unit 111 (step S208). Next, the learning control unit 111 determines whether or not the learning-time prediction end condition is satisfied based on the stop determination information (step S209). The strengthened prediction end condition is a condition regarding the end of the strengthened prediction process and is based on at least the stop determination information. The strengthened prediction end condition is different from the prediction end condition in that the process for which it is determined whether or not to stop is the strengthened prediction process instead of the prediction process. The strengthened prediction end condition is, for example, either the condition that the stop determination information ti indicates stop or the condition that the number of times the execution of the strengthened prediction process is started is equal to or greater than the maximum number of predictions T (that is, i<T). The condition is that one of them is satisfied.

If the enhanced prediction end condition is met, the process ends. On the other hand, if the reinforcement prediction end condition is not satisfied, the learning control unit 111 updates the auxiliary information ai according to a predetermined update rule (step S210). After step S210, the process returns to step S204.

Note that step S206 may be executed at any timing as long as it is executed after step S205 and before step S209 is executed. Note that step S208 may be executed at any timing as long as it is executed after step S207 and before step S209 is executed. Note that the process of step S205 may be performed after the process of step S207 as long as it is after the process of step S204 and before the process of step S209.

The processing of FIG. 8 is executed for all training data D prepared in advance.

The reward acquisition process includes correctness/incorrectness determination process. The correctness/incorrectness determination process is a process for determining whether or not the set of prediction results {Oi} obtained by the enhanced prediction process correctly predicted the set of normal gaze regions {Sj}. In the reward acquisition process, after execution of the correctness determination process, a reward is acquired based on the result of the correctness determination process. Acquisition of the reward is acquired by computation, for example.

FIG. 9 is a flowchart showing an example of the flow of reward acquisition processing in the embodiment. The learning control unit 111 initializes the predicted success number Q (step S301). As a result of the initialization, 0 is assigned to the number Q of predicted successes. Next, the learning control unit 111 selects one prediction result Oi for which prediction success/failure has not yet been determined from among the prediction result set {Oi} obtained by the enhanced prediction process (step S302). Next, the learning control unit 111 determines whether the prediction result Oi selected in step S302 is successful or not (step S303). Determining the success or failure of prediction specifically means determining whether or not at least one of the correct gaze regions {Sj} has been predicted. If at least one of the correct fixation regions {Sj} could be predicted, the prediction was successful; if none of the correct fixation regions {Sj} could be predicted, the prediction was not successful (i.e., no ) means that

The method of determining whether prediction is successful or not depends on the prediction result expression format. As an example, an example of a method for judging the success or failure of prediction will be described for the case where the region of interest is given by the point of interest as shown in FIG. 2 or 3 . In this case, both Oi and {Sj} represent points. Therefore, S* is determined to be the closest distance to Oi from among {Sj}. If the distance between S* and Oi is less than a certain value, the prediction is judged to be successful, and if it is greater than the certain value, the prediction is to fail. The success or failure of the prediction is determined by the determination method.

As another example of the method for determining the success or failure of prediction, an example of the method for determining the success or failure of prediction will be described for the case where the regions are discretized as shown in FIG. In this case, if Oi and at least one S* of {Sj} indicate the same area, the prediction is determined to be successful, and if not, the prediction is determined to be unsuccessful. Success or failure is determined. In addition, when the distribution is represented as shown in FIG. 5, prediction is performed when the overlap between the area covered by the distribution of Oi and the area covered by at least one S* of {Sj} is greater than or equal to a certain amount. The success or failure of the prediction is judged by a method of judging the prediction as failure when the value is successful or less than a certain value.

After step S303, the learning control unit 111 updates the number Q of successful predictions, and removes the correct fixation region predicted by the prediction result Oi from the correct fixation region set {Sj} (step S304). Specifically, updating the predicted success number Q is a process of increasing the value of the predicted success number by one.

Next, the learning control unit 111 determines whether or not the reward acquisition condition is satisfied (step S305). The reward acquisition condition is at least the condition that the prediction result Oi of all the prediction results obtained by the enhanced prediction process is judged to be successful or not, and the condition that the set {Sj} of the normal gaze regions is an empty set. It is a condition that one is satisfied. Note that the process of determining whether the prediction is successful or not is specifically the process of step S303.

If the reward acquisition condition is not satisfied (step S305: NO), the process returns to step S302. On the other hand, if the reward acquisition condition is satisfied (step S305: YES), the learning control unit 111 acquires a predefined reward value (step S306).

<About rewards>
Describe rewards. The reward is, for example, the quantity R _pred defined by Equation (3) below.

A desirable predictive model is more likely to be obtained if the reward is appropriate. Further, it is preferable to perform learning that can more accurately predict as many gaze regions as possible among a plurality of correct gaze regions with as few predictions as possible. Therefore, by giving a larger reward when as many gaze regions as possible can be predicted more accurately with as few predictions as possible, the probability of favorable learning is increased.

The value of the left side of the above formula (3) is a value of 0 or more and 1 or less. The value of the left side of the above equation (3) has a maximum value of 1.0 when the number of predictions N is the same as the number of correct gaze regions M and all predictions are successful, that is, when M=N=Q. . Therefore, the reward R _pred defined by Equation (3) has the property of giving a high reward when as many of the correct gaze regions as possible can be predicted more accurately with as few predictions as possible. meet. Therefore, the reward R _pred defined by Equation (3) is suitable for learning prediction models.

Reinforcement prediction learning processing will be described. The reinforcement prediction learning process updates the prediction model based on the reward obtained in the reward acquisition process. Reinforcement prediction learning processing updates NN101, LSTM102, and prediction FC103 based on a reward, for example, when predictor 10 is provided with CNN101, LSTM102, prediction FC103, and stop judgment FC104. For simplicity of explanation, a deep neural network combining the NN 101, the LSTM 102 and the prediction FC 103 is represented as φ.

Reinforcement predictive learning processing is processing that updates the network φ based on rewards. The process of updating the network φ based on the reward in the reinforcement predictive learning process may be any method as long as it is based on the reward. For example, the update method may be the policy gradient method. When the update method is the policy gradient method, the weight Wφ of the network φ is updated based on the following equations (4) and (5).

α is any positive real value representing the learning rate. α is, for example, 0.01.

As described above, in the prediction model reinforcement learning process, the prediction model is used to perform the process of estimating the region of interest based on the image data included in the training data (that is, the reinforcement prediction process). In the prediction model reinforcement learning process, by executing the reward acquisition process and the reinforcement prediction learning process, the prediction model is adjusted so that the estimation result obtained by the reinforcement prediction process approaches the gaze area indicated by the correct gaze area data included in the training data. Updated.

Here, when reinforcement learning is used for learning the network φ, a prediction model with higher estimation accuracy can be generated than when using supervised learning. Explain why reinforcement learning has higher estimation accuracy.

One of the reasons is that it is not obvious which of the plurality of correct gazing regions {Sj} should be compared with the i-th prediction result Oi of the predictor 10 . One of the reasons is that the prediction by the predictor 10 is repeatedly executed until the prediction end condition is satisfied, so it is necessary to proceed with learning by comprehensively judging the results of multiple predictions. For example, even if the first prediction result O1 correctly indicates the first one of the correct fixation regions S1, if the second and subsequent prediction results O2, . It is difficult to say that the second and subsequent prediction results are meaningful prediction results.

In this way, the quality of a prediction model cannot be determined based on each prediction result, and must be determined based on multiple prediction results (that is, comprehensively determined). Furthermore, it is desirable that the prediction model can predict the correct fixation region just enough. One of the reasons is that it is desirable that not only the prediction result {Oi} of the region of interest but also the stop determination information {ti} be highly accurate information.

The stop determination learning process will be described. The stop determination learning process is a process of updating the contents of the stop determination process. The stop judgment learning process updates the CNN 101, the LSTM 102 and the stop judgment FC 104, for example. The stop determination process is desirably a process of determining the timing at which all the correct fixation regions have been predicted as a stop. The timing at which all correct fixation regions can be predicted is, for example, the timing at which {Sj} becomes an empty set in step S305.

Unlike the prediction process, the stop determination process provides clear teaching data for the output. Therefore, even if the stop determination process is updated by the supervised machine learning method, the process is generated with a high accuracy equal to or higher than that in the case of the reinforcement learning.

The stop judgment learning process updates the CNN 101, LSTM 102 and stop judgment FC 104 provided in the predictor 10. FIG. For simplicity of explanation, a deep neural network connecting the NN 101, the LSTM 102 and the stop determination FC 104 will be represented as ψ. The learning method executed by the stop determination learning process may be any method as long as it is a supervised learning method. The stop judgment learning process updates the contents of the stop judgment process by, for example, the gradient method. As an example, the stop determination learning process will be described for a case where the learning method executed by the stop determination learning process is the gradient method.

For the sake of simplicity of description, the correct stop determination information (hereinafter referred to as "correct stop determination information") indicating the correct content is represented by ui as the correct stop determination information for the identifier i. For example, the correct stop determination information ui for the identifier i indicates 1 when {Sj} is empty, and indicates 0 otherwise. The stop determination learning process updates the weight Wψ of ψ so that the loss function of Equation (6) below becomes small.

The loss function of Equation (6) is a loss function called Binary Cross Entropy. The loss function of Equation (6) takes a small value when the stop determination information ti matches the correct stop determination information ui. Therefore, by updating Wψ so that the value of the loss function (that is, the loss) in Equation (6) becomes smaller, ψ that can output ti close to ui is obtained. The update method is, for example, the gradient method.

The learning device 1 learns the prediction model by repeating the sequential execution of subroutines of reinforcement prediction processing, reward acquisition processing, reinforcement prediction learning processing, and stop determination learning processing.

FIG. 10 is a diagram showing an example of the hardware configuration of the learning device 1 according to the embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) connected via a bus and a memory 92, and executes a program. The learning device 1 functions as a device having a control unit 11, an input/output interface 12, and a storage unit 13 by executing a program.

More specifically, in the learning device 1, the processor 91 reads the program stored in the storage unit 13, and causes the memory 92 to store the read program. The processor 91 executes a program stored in the memory 92 , whereby the learning device 1 functions as a device comprising the control section 11 , the input/output interface 12 and the storage section 13 .

The control unit 11 controls operations of various functional units included in the learning device 1 . The input/output interface 12 includes an interface for connecting the learning device 1 to an external device. The input/output interface 12 communicates with an external device via wire or wireless. The external device is the prediction device 2, for example. The input/output interface 12 includes input devices such as a mouse, a keyboard, and a touch panel. The input/output interface 12 may include an interface that connects these input devices to the learning device 1 .

The input/output interface 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The input/output interface 12 may include an interface that connects these display devices to the study device 1 . Training data, for example, is input to the input/output interface 12 .

The storage unit 13 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information about the learning device 1 . The storage unit 13 stores, for example, various information generated as a result of processing executed by the control unit 11 . The storage unit 13 stores, for example, parameter values of the predictor 10 . The storage unit 13 stores auxiliary information in advance, for example. The storage unit 13 may store the maximum number of times of prediction T in advance, for example.

FIG. 11 is a diagram showing an example of the functional configuration of the control section 11 in the embodiment. The controller 11 includes a learning controller 111 and a predictor 10 . A learning control unit 111 controls the operation of the predictor 10 and executes learning of the predictor 10 . The learning control unit 111 executes prediction model reinforcement learning processing, for example.

FIG. 12 is a diagram showing an example of the hardware configuration of the prediction device 2 in the embodiment. The prediction device 2 includes a control unit 21 including a processor 93 such as a CPU (Central Processing Unit) connected via a bus and a memory 94, and executes a program. The prediction device 2 functions as a device including a control unit 21, an input/output interface 22, and a storage unit 23 by executing a program.

More specifically, the prediction device 2 causes the processor 93 to read the program stored in the storage unit 23 and store the read program in the memory 94 . The processor 93 executes the program stored in the memory 94 so that the prediction device 2 functions as a device including the control unit 21 , the input/output interface 22 and the storage unit 23 .

The control unit 21 controls operations of various functional units included in the prediction device 2 . The input/output interface 22 includes an interface for connecting the prediction device 2 to an external device. The input/output interface 22 communicates with an external device via wire or wireless. The external device is the learning device 1, for example. The input/output interface 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input/output interface 22 may comprise an interface connecting these input devices to the prediction device 2 .

The input/output interface 22 includes a display device such as a CRT display, a liquid crystal display, an organic EL display, or the like. The input/output interface 22 may comprise an interface connecting these display devices to the prediction device 2 . For example, image data to be estimated is input to the input/output interface 22 .

The storage unit 23 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information regarding the prediction device 2 . The storage unit 23 stores various information generated as a result of processing executed by the control unit 21, for example. The storage unit 23 stores, for example, a learned prediction model. The storage unit 23 stores auxiliary information in advance, for example. The storage unit 23 may store the maximum number of times of prediction T in advance, for example.

FIG. 13 is a diagram showing an example of the functional configuration of the control section 21 in the embodiment. The control unit 21 includes a predictor control unit 211 and the predictor 10 . The predictor control unit 211 controls the operation of the predictor 10 included in the prediction device 2 . That is, the predictor control unit 211 causes the predictor 10 to execute the learned prediction model.

<Application example>
An application example of the prediction system 100 will be described. The prediction device 2 is installed, for example, in a vehicle. In this case, the prediction device 2 predicts a region to be watched outside the visible region based on the image acquired by the camera mounted on the vehicle. After prediction, the prediction device 2 indicates the areas to be watched or noticed, for example via visual or auditory warnings.

The prediction device 2 may be mounted on a robot, for example. In this case, the prediction device 2 predicts an area to be watched outside the visible area from the image acquired by the camera mounted on the robot. After the prediction, the prediction device 2 controls the action of the robot by transmitting the prediction result to the control device that controls the action of the robot based on the prediction result.

The prediction system 100 configured in this manner includes a learning device 1 that obtains a mathematical model representing the relationship between image data and a gaze region in space that is not captured in the image represented by the image data. Therefore, the prediction system 100 can predict a region of interest in a region in space that is not imaged. This also applies to video data instead of image data as described above.

The prediction system 100 also includes a prediction device 2 that predicts a region of interest using the mathematical model obtained by the learning device 1 . Therefore, the prediction system 100 can predict a region of interest in a region in space that is not imaged. This also applies to video data instead of image data as described above.

(Modification)
Note that each of the prediction system 100, the learning device 1, and the prediction device 2 may be implemented using a plurality of information processing devices that are communicably connected via a network. Note that all or part of each function of the prediction system 100, the learning device 1, and the prediction device 2 is hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). may be implemented using The program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. The program may be transmitted over telecommunications lines. Note that the learned prediction model is an example of an updated mathematical model. Note that the prediction device 2 is an example of a prediction device. Prediction is an example of guessing.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 100... Prediction system 1... Learning apparatus 2... Prediction apparatus 10... Predictor 11... Control part 12... Input/output interface 13... Storage part 111... Learning control part 21... Control part 22... Input Output interface 23 Storage unit 211 Predictor control unit 91 Processor 92 Memory 93 Processor 94 Memory

Claims (7)

Based on the training data, which is a pair of the image data and the correct gaze region data, which is the data indicating the gaze region in the space that is not shown in the image indicated by the image data, the image indicated by the image data and the image indicated by the image data. is a control unit that updates a mathematical model that shows the relationship with the gaze area in the space that is not captured,
with
The control unit uses the mathematical model to estimate a gaze area based on the image data included in the training data, and uses the mathematical model so that the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data. update the model,
learning device. The control unit updates the mathematical model by a method of reinforcement learning,
A learning device according to claim 1. The mathematical model is expressed using a long-term memory network,
The mathematical model is information input to the long short-term memory network, the indicated value is a value corresponding to the timing of input to the long short-term memory network, and the value has a predetermined distribution with a non-zero variance. estimating the gaze region based on auxiliary information that is a value according to
3. The learning device according to claim 1 or 2. Based on the training data, which is a pair of the image data and the correct gaze region data, which is the data indicating the gaze region in the space that is not shown in the image indicated by the image data, the image indicated by the image data and the image indicated by the image data. a control unit that updates a mathematical model indicating the relationship with the gaze area in a space that is not captured, the control unit estimates the gaze area based on the image data included in the training data using the mathematical model, and estimates updated until a predetermined termination condition is satisfied by a learning device that updates the mathematical model so that the result of is closer to the gaze region indicated by the correct gaze region data included in the training data A control unit that estimates, based on the input image data, a gaze area in a space that is not reflected in the image indicated by the image data, using the mathematical model of
Guessing device with Based on the training data, which is a pair of the image data and the correct gaze region data, which is the data indicating the gaze region in the space that is not shown in the image indicated by the image data, the image indicated by the image data and the image indicated by the image data. is a control step that updates the mathematical model that shows the relationship with the gaze area in the space that is not captured,
has
In the control step, a gaze area is estimated based on image data included in the training data using the mathematical model, and the estimation result approaches the gaze area indicated by the correct gaze area data included in the training data. the model is updated,
learning method. Based on the training data, which is a pair of the image data and the correct gaze region data, which is the data indicating the gaze region in the space that is not shown in the image indicated by the image data, the image indicated by the image data and the image indicated by the image data. a control unit that updates a mathematical model indicating the relationship with the gaze area in a space that is not captured, the control unit estimates the gaze area based on the image data included in the training data using the mathematical model, and estimates updated until a predetermined termination condition is satisfied by a learning device that updates the mathematical model so that the result of is closer to the gaze region indicated by the correct gaze region data included in the training data A control step of estimating, based on the input image data, a gaze region in a space that is not reflected in the image indicated by the image data, using the mathematical model of
Guessing method with A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

Images