JP7245750B2 - Evaluation support system, evaluation support method, and program - Google Patents

Description

The present invention relates to an evaluation support system, an evaluation support method, and a program.

Conventionally, techniques for supporting work in the construction industry have been studied. For example, Patent Document 1 discloses a technique for automating face evaluation using a learning model obtained by learning teacher data indicating the relationship between a photographed image of a face during underground observation and evaluation results for observation items of the face. is described. For example, in Non-Patent Document 1, there is a technique that tracks the line of sight of an operator who operates a shield machine on the operation screen, and analyzes a series of processes in which the operator recognizes and deals with fluctuations that occur in the suction pressure of the pump. Are listed.

JP 2019-023392 A

Proceedings of the 32nd Annual Conference of the Japanese Society for Artificial Intelligence (2018), "Estimation of Cognitive Processes of Shield Machine Operators by Eye Tracking", Nao Fujimoto, Junya Morita, Yasushi Okubo, Nobuhiko Obayashi, Takehiro Shirai

When evaluating an evaluation object such as a face, the important part that the evaluator should focus on differs depending on various conditions such as the environment of the construction site. A skilled person can identify the important part from his/her own experience, but it is difficult for an inexperienced person to identify the important part. In this respect, Patent Literature 1 does not describe the important part, and only the features of the face shown in the captured image are learned evenly as a whole, so it is possible to sufficiently improve the accuracy of the learning model. Can not. Non-Patent Document 1 only tracks the line of sight on the operation screen on which the suction pressure of the pump is displayed, and does not specify the important part of the evaluation target. For this reason, the conventional technology cannot sufficiently support the task of evaluating the evaluation target.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object thereof is to provide an evaluation support system, an evaluation support method, and a program capable of supporting work for evaluating an evaluation target.

In order to solve the above-described problems, an evaluation support system according to an aspect of the present invention includes a teacher-photographed image of an evaluation target at a construction site, teacher-gazing-point information corresponding to the teacher-photographed image, and Teacher data acquisition means for acquiring teacher data indicating the relationship between a teacher evaluation result to be evaluated, learning means for learning an evaluation result output model based on the teacher data, and an input to the evaluation result output model Input means for inputting a captured image and input gaze point information corresponding to the input captured image; and output evaluation result obtaining means for obtaining an output evaluation result output from the evaluation result output model. characterized by

An evaluation support method according to an aspect of the present invention includes a teacher-photographed image of an evaluation target at a construction site, teacher gaze point information corresponding to the teacher-photographed image, a teacher evaluation result of the evaluation target by an evaluator, a learning step of learning an evaluation result output model based on the training data; and an input photographed image and the input photographed image for the evaluation result output model. and an input step of inputting input gaze point information corresponding to and an output evaluation result obtaining step of obtaining an output evaluation result output from the evaluation result output model.

A program according to an aspect of the present invention provides a relationship between a teacher-captured image in which an evaluation target is captured at a construction site, teacher gaze point information corresponding to the teacher-captured image, and a teacher evaluation result of the evaluation target by an evaluator. a learning means for learning an evaluation result output model based on the training data; an input photographed image and an input corresponding to the input photographed image for the evaluation result output model; The computer functions as input means for inputting point-of-regard information, and output evaluation result acquisition means for acquiring output evaluation results output from the evaluation result output model.

In one aspect of the present invention, the evaluation support system further includes teacher gaze point information acquisition means for acquiring the teacher gaze point information based on the evaluator's gaze detected by the gaze detection means, The teacher data is characterized in that it indicates the relationship between the teacher photographed image, the teacher gaze point information acquired by the teacher gaze point information acquisition means, and the teacher evaluation result.

In one aspect of the present invention, the teacher gazing point information acquisition means specifies, from among the gaze of the evaluator detected by the sight line detection means, the gaze toward the screen on which the teacher's photographed image is displayed. and acquiring the teacher gaze point information based on the specified line of sight.

Further, in one aspect of the present invention, the teacher data indicates the relationship among the characteristic information of construction at the construction site, the teacher-captured image, the teacher gaze information, and the teacher evaluation result, and The support system further includes feature information acquisition means for acquiring feature information of the construction corresponding to the input photographed image, and the input means receives the feature information acquired by the feature information acquisition means for the evaluation result output model. , the input photographed image, and the input gaze point information.

Further, in one aspect of the present invention, the learning means learns the evaluation result output model based on the teacher data corresponding to the feature information for each feature information of construction at the construction site, and the evaluation support system further includes feature information acquisition means for acquiring feature information of the construction corresponding to the input photographed image, and the input means receives the evaluation result output model corresponding to the feature information acquired by the feature information acquisition means. and inputting the input photographed image and the input gaze point information.

Further, in one aspect of the present invention, the evaluation support system converts the output gaze point information output when the teacher photographed image is input to the trained gaze point information output model into the teacher gaze point information wherein the teacher data is a relationship between the teacher captured image, the teacher gazing point information acquired by the teacher gazing point information acquiring means, and the teacher evaluation result. It is characterized by showing

Further, in one aspect of the present invention, the evaluation support system converts the output gaze point information output when the input photographed image is input to the trained gaze point information output model into the input gaze point information The input means obtains the input gazing point information acquired by the input gazing point information acquiring means for the evaluation result output model, and is input.

In one aspect of the present invention, each of the teacher captured image and the input captured image has the same size, and each of the teacher gazing point information and the input gazing point information has a gazing point represented by a color. Further, it is an image having the same size as each of the teacher captured image and the input captured image.

Further, in one aspect of the present invention, the evaluation target is a tunnel face, the teacher-captured image is an image of the tunnel face at the construction site, and the teacher gaze point information is obtained by the evaluator. A gaze point when evaluating the tunnel face is indicated, the input captured image is an image of the tunnel face captured at the construction site or another construction site, and the input gaze point information is the input captured image. Characterized by indicating the part to be viewed when evaluating the indicated tunnel face.

According to the present invention, it is possible to support work for evaluating an evaluation target.

It is a figure showing the whole evaluation support system composition concerning an embodiment. It is a figure which shows an example of the picked-up image by which the face was image|photographed. FIG. 10 is a diagram showing how an expert evaluates a face by looking at a photographed image; FIG. 4 is an explanatory diagram showing an overview of a point-of-regard image output model; FIG. 4 is an explanatory diagram showing an outline of an evaluation result output model; 3 is a functional block diagram showing an example of functions realized by the evaluation support system; FIG. FIG. 4 is a diagram showing a data storage example of first teacher data; FIG. 10 is a diagram showing a data storage example of second teacher data; FIG. 10 is a flow diagram showing learning processing; FIG. 11 is a flowchart showing evaluation support processing; It is a functional block diagram concerning a modification.

[1. Overall configuration of evaluation support system]
FIG. 1 is a diagram showing the overall configuration of an evaluation support system according to an embodiment. As shown in FIG. 1, the evaluation support system S includes a learning terminal 10 and a line-of-sight detection device 20, which are communicably connected to each other. Note that the evaluation support system S may include other computers such as a server computer.

The learning terminal 10 is a computer that executes the processing described in this embodiment, and is, for example, a personal computer, a personal digital assistant (including a tablet computer), or a mobile phone (including a smart phone). For example, the study terminal 10 includes a control unit 11 , a storage unit 12 , a communication unit 13 , an operation unit 14 and a display unit 15 .

Control unit 11 includes at least one processor. The control unit 11 executes processing according to programs and data stored in the storage unit 12 . The storage unit 12 includes a main storage unit and an auxiliary storage unit. For example, the main memory is volatile memory such as RAM, and the auxiliary memory is nonvolatile memory such as hard disk or flash memory. The communication unit 13 includes a communication interface for wired communication or wireless communication, and performs data communication via a network, for example. The operation unit 14 is an input device such as a touch panel, a pointing device such as a mouse, or a keyboard. The operation unit 14 transmits operation contents to the control unit 11 . The display unit 15 is, for example, a liquid crystal display unit or an organic EL display unit.

The line-of-sight detection device 20 is a device that detects a human line of sight, and includes, for example, a camera, an infrared sensor, or an infrared light emitting unit. The line-of-sight detection device 20 is sometimes called an eye tracker and detects the movement of human eyes. Any method can be used as the line of sight detection method itself. may be used. Although the line-of-sight detection device 20 is an external device of the learning terminal 10 in this embodiment, the line-of-sight detection device 20 may be incorporated as part of the learning terminal 10 . Also, although the stationary line-of-sight detection device 20 will be described as an example, the line-of-sight detection device 20 may be a wearable device such as a head-mounted display or smart glasses.

Note that the programs and data described as being stored in the storage unit 12 may be supplied to the study terminal 10 via a network. Also, the hardware configuration of the study terminal 10 is not limited to the above example, and various hardware can be applied. For example, the study terminal 10 includes a reading unit (eg, an optical disc drive or memory card slot) that reads a computer-readable information storage medium, and an input/output unit (eg, a USB terminal) for direct connection with an external device. It's okay. In this case, the programs and data stored in the information storage medium may be supplied to the study terminal 10 via the reading section or the input/output section.

[2. Overview of Evaluation Support System]
The evaluation support system S supports the work of the evaluator who evaluates the evaluation target of the construction site. The evaluator is the person in charge of the evaluation, and may be an employee of the construction company or a person entrusted with the evaluation work by the construction company. A construction site is a place where construction is performed, and may be above ground or underground. The work may be any type of work, for example civil engineering work or building work.

An evaluation target is an object to be evaluated by an evaluator. In other words, the object of evaluation can be said to be what the evaluator sees in order to judge the progress of the construction work. For example, an object to be evaluated is an object to be removed by construction, an object to be created by construction, or an object to be demolished by construction. Evaluation means observing an object to be evaluated. In other words, evaluation is checking the condition, quality, or goodness or badness of an object to be evaluated.

In this embodiment, the processing of the evaluation support system S will be described by taking as an example a scene of evaluating a tunnel face (hereinafter simply referred to as a face) in mountain tunnel construction. A face is an excavation surface or an excavation site. A face is an example of an object to be evaluated, and the part described as a face in the present embodiment can be read as an object to be evaluated. The evaluator may go to the construction site and directly look at the face for evaluation, but in this embodiment, a case will be described in which the face is evaluated by looking at a photographed image of the face photographed with a camera.

FIG. 2 is a diagram showing an example of a photographed image in which a face is photographed. As shown in FIG. 2, for example, a photographed image I1 shows a face photographed from the front. The evaluator visually evaluates the face shown in the photographed image I1 and creates a face observation list. The face observation record may be electronic data or paper. In the face observation book, the evaluation results of the face are entered or entered. There are evaluation items such as presence/absence or presence/absence of deterioration due to water.

The evaluator evaluates each portion of the face, such as the left side, the crest, and the right side, for each item to be evaluated. In principle, the evaluator must evaluate the face at least once a day, and create a face observation list by looking at the most recent photographed images of the condition of the face. For this reason, preparation of the face observation log is one of the factors that increase the workload of the evaluator. In addition, since face evaluation requires highly specialized knowledge and extensive experience, it is difficult for an inexperienced evaluator to perform an accurate evaluation. Furthermore, evaluation of the working face changes depending on the viewpoint of the evaluator and the situation at the site, and it is difficult to make a universal evaluation.

Therefore, the evaluation support system S tracks the line of sight of the expert when evaluating the face by looking at the photographed image, prepares a machine learning model that learns the part that the expert focuses on, and uses it to evaluate the face. We are trying to support In addition, the evaluation support system S also prepares a machine learning model learned from a face observation book created by an expert to support face evaluation work.

FIG. 3 is a diagram showing how a skilled person evaluates a face by looking at a photographed image. As shown in FIG. 3, the expert displays a photographed image I1 on the display unit 15, evaluates the face, and creates a face observation book B1. The line-of-sight detection device 20 is arranged, for example, under the display unit 15 so as to detect the line of sight of the expert. The line-of-sight detection device 20 has a detection range in the front direction of the screen, and detects the line of sight of the expert. Based on the detection result of the line-of-sight detection device 20, the learning terminal 10 creates a point-of-regard image I2 indicating a portion that the expert views with emphasis.

The point-of-regard image I2 is an image also called a heat map, and the part seen by the expert is expressed in color. The color of the point-of-regard image I2 represents the degree of attention, and the portions with a strong degree of attention and the portions with a weak degree of attention are color-coded. The degree of fixation is the degree to which an expert attaches importance. In other words, the degree of gaze can also be said to be the time or frequency of gaze by the expert. For example, the point-of-regard image I2 may use a color gradation to indicate the degree of attention, or may be color-coded into a plurality of colors such as red, blue, and yellow without using a gradation to indicate the degree of attention. may be shown.

In FIG. 3, the color depth is indicated by the density of halftone dots, and the darker the color of the gaze point image I2, the higher the degree of gaze. The gaze point image I2 of this embodiment has the same size as the captured image I1, and the face position indicated by the captured image I1 and the gaze point position indicated by the gaze point image I2 correspond to each other. Therefore, the point-of-regard image I2 in FIG. 3 means that the expert is focusing on the upper right, lower left, and lower center states of the face.

The evaluation support system S has an expert evaluate various captured images I1 to create a gaze point image I2. A plurality of experts may participate in creating the point-of-regard image I2, or a photographed image I1 obtained by photographing each face of a plurality of construction sites may be used. The evaluation support system S accumulates pairs of the captured image I1 and the gaze point image I2, and creates a gaze point image output model for generating the gaze point image I2 from the captured image I1.

FIG. 4 is an explanatory diagram showing an overview of the point-of-regard image output model. As shown in FIG. 4, the point-of-regard image output model M1 stores a large number of pairs of a photographed image I1 evaluated by an expert and a point-of-regard image I2 created using the line-of-sight detection device 20. 1 This is a machine learning model that is trained on one teacher data D1. Machine learning itself can use a known method, and for example, a convolutional neural network or a recursive neural network may be used. For example, an inexperienced evaluator inputs a photographed image I3 of a face to be evaluated in the work of the day to the learned point-of-regard image output model M1.

The captured image I3 is an unknown image that has not been learned by the point-of-regard image output model M1. The point-of-regard image output model M1 outputs the point-of-regard image I4 when the photographed image I3 is input. Since the point-of-regard image output model M1 has learned the first training data D1, the point-of-regard image I4 is presumed to be viewed by an expert when it is assumed that the expert evaluates the face of the captured image I3. It will show the part that The evaluator may evaluate the face by looking at the photographed image I3 while referring to the gaze point image I4. It is

FIG. 5 is an explanatory diagram showing an overview of the evaluation result output model. As shown in FIG. 5, the evaluation result output model M2 includes second teacher data D2 that stores a large number of pairs of the above-described photographed image I1 and point-of-regard image I2, and face observation book B1 filled in by an expert. It is a trained machine learning model. In this embodiment, when the captured image I3 is input to the point-of-regard image output model M1 to create the point-of-regard image I4, the captured image I3 and the point-of-regard image I4 are input to the evaluation result output model M2. be.

The evaluation result output model M2 outputs a face observation list B2 when the photographed image I3 and the gazing point image I4 are input. Since the point-of-regard image output model M1 has already learned the second teacher data D2, the face observation book B2 is a face observation book presumed to be created by an expert when it is assumed that an expert evaluates the face of the photographed image I3. Indicates the contents of the book. The face observation record B2 may be used as it is as the evaluation result of the day, or may be used as reference information for the evaluator in charge of the evaluation of the day.

As described above, the evaluation support system S mainly includes the first configuration for acquiring the point-of-regard image I4 using the point-of-regard image output model M1, and the face observation record B2 using the evaluation result output model M2. and a second configuration that acquires, so that it is possible to support the work of evaluating the evaluation target. Hereinafter, the details of the evaluation support system S will be described.

[3. Functions realized in the present embodiment]
FIG. 6 is a functional block diagram showing an example of functions realized by the evaluation support system S. As shown in FIG. As shown in FIG. 6, the evaluation support system S includes a data storage unit 100, a teacher gazing point image acquiring unit 101, a first teacher data acquiring unit 102, a first learning unit 103, a first input unit 104, an output gazing point image Acquisition unit 105, second teacher data acquisition unit 106, second learning unit 107, input gaze point image acquisition unit 108, second input unit 109, and output evaluation result acquisition unit 110 are implemented.

The data storage unit 100, the teacher gazing point image acquisition unit 101, the first teacher data acquisition unit 102, the first learning unit 103, the first input unit 104, and the output gazing point image acquisition unit 105 mainly relate to the first configuration. It is a function. Further, the data storage unit 100, the second teacher data acquisition unit 106, the second learning unit 107, the input gaze point image acquisition unit 108, the second input unit 109, and the output evaluation result acquisition unit 110 mainly have the second configuration. It is a function related to In this embodiment, the case where each of these functions is realized by the study terminal 10 will be described, but each function may be realized by another computer such as a server computer as in a modified example described later.

[Data storage part]
The data storage unit 100 is realized mainly by the storage unit 12 . The data storage unit 100 stores data necessary for executing the processing of this embodiment. For example, the data storage unit 100 stores first teacher data D1 and second teacher data D2.

FIG. 7 is a diagram showing a data storage example of the first teacher data D1. As shown in FIG. 7, the first training data D1 is data indicating the relationship between a teacher-captured image of a face at a construction site and a teacher gaze point image of an evaluator who evaluated the face. In this embodiment, the teacher's point-of-regard image acquisition unit 101, which will be described later, acquires the teacher's point-of-regard image. 3 shows the relationship between the viewpoint image and the In this embodiment, the teacher's point-of-regard image indicates the point-of-regard when the evaluator evaluates the face by looking at the teacher-captured image. Instead, the point of gaze when evaluating the actual face directly may be indicated.

The first training data D1 is created at an arbitrary timing after the evaluation by the expert, and stores a plurality of pairs of teacher-captured images and teacher-gazing-point images. The number of pairs stored in the first training data D1 may be arbitrary, and may be, for example, about ten to several tens, or about one hundred to several tens of thousands. The first training data D1 can be said to be data that defines the correspondence between the input and the output of the point-of-regard image output model M1. Although FIG. 7 shows the case where the image file is stored in the first teacher data D1, the feature amount of each image may be stored in the first teacher data D1. The feature quantity may be expressed in any format such as vector or array.

A teacher captured image is a captured image used as teacher data. In other words, the teacher captured image is a captured image used for training a machine learning model. Note that teacher data is also called training data or learning data. A photographed image I1 shown in FIGS. 3 to 5 corresponds to a teacher photographed image. In the teacher-captured image, an evaluation target at a construction site is captured. In the present embodiment, the face corresponds to the evaluation target, so the teacher-captured image is an image in which the face is captured.

The teacher-captured image may be data with any extension, such as a JPEG, PNG, BMP, or GIF image. In the present embodiment, a case where the captured teacher image is a color image will be described, but the captured teacher image may be in other formats such as a grayscale image or a monochrome image. The size, resolution, and number of bits of the teacher-captured image may be arbitrary.

In the present embodiment, the face imaging conditions are the same for each teacher-photographed image, and the extension, format, size, resolution, and number of bits are the same, but these may be different. The shooting conditions are the conditions for shooting the face, such as the positional relationship between the face and the camera (relative position, orientation, and height of the camera with respect to the face), the ratio of the face in the image, the brightness of the proof at the time of shooting, and the Settings such as color or the format of the image that the camera produces. Further, the positional relationship between the face and the camera (imaging direction of the face) is the same for all teacher-captured images, but the positional relationship between the face and the camera may be slightly different for each teacher-captured image.

A teacher gazing point image is a gazing point image used as teacher data. In other words, the teacher gaze image is the gaze image used to train the machine learning model. In the teacher gaze point image, the gaze point (degree of gaze) is represented by color. The point-of-regard image I2 shown in FIGS. 3 to 5 corresponds to the teacher's point-of-regard image. In this embodiment, since the face corresponds to the evaluation target, the teacher gaze point image is an image showing the gaze point when the evaluator evaluates the face.

The teacher gaze point image may be data with any extension, such as a JPEG, PNG, BMP, or GIF image. In this embodiment, a case where the teacher's point-of-regard image is a color image will be described, but the teacher's point-of-regard image may be in other formats such as a grayscale image or a monochrome image. The size, resolution, and number of bits of the teacher gaze image may be arbitrary.

In this embodiment, the extension, format, size, resolution, and number of bits of the teacher-captured image and the teacher-gazing-point image are the same, but they may be different. Further, in the present embodiment, a point-of-regard image will be described as an example of the point-of-regard information, but the point-of-regard information may be in a format other than an image that can be visually recognized by humans. , or information in the form of a mathematical formula, or the like. The part described as the point-of-regard image in the present embodiment can be read as point-of-regard information. For example, a teacher gazing point image, an output gazing point image, and an input gazing point image can be read as teacher gazing point information, output gazing point information, and input gazing point information, respectively.

FIG. 8 is a diagram showing a data storage example of the second teacher data D2. As shown in FIG. 8, the second training data D2 includes a teacher-photographed image of an evaluation target at a construction site, a teacher-gazing-point image corresponding to the teacher-photographed image, and a teacher evaluation result of the evaluation target by an evaluator. , is data showing the relationship between . In this embodiment, the teacher's point-of-regard image is acquired by the teacher's point-of-regard image acquisition unit 101, which will be described later. 4 shows the relationship between images and teacher evaluation results.

The second teacher data D2 is created at an arbitrary timing after the expert's evaluation, and stores a plurality of pairs of teacher-captured images, teacher-gazing-point images, and teacher evaluation results. The number of pairs stored in the second training data D2 may be arbitrary, and may be, for example, about ten to several tens, or about one hundred to several tens of thousands. The second teacher data D2 can be said to be data that defines the correspondence relationship between the input and the output of the evaluation result output model M2. The teacher photographed image and the teacher gaze point image correspond to the input, and the teacher evaluation result corresponds to the output. . Although FIG. 8 shows the case where the image file is stored in the second teacher data D2, the feature amount of each image may be stored in the second teacher data D2.

The teacher's point-of-regard image corresponding to the teacher's shot image is the gaze-point image created when the teacher's shot image is evaluated by an expert, or the gaze-point image when it is assumed that the teacher's shot image is evaluated by an expert. be. For example, if the photographed image I1 shown in FIGS. 3 to 5 is a teacher photographed image, a gaze point image I2 created using the line-of-sight detection device 20 is a teacher gaze point image corresponding to the teacher photographed image. Note that, as in a modified example described later, the point-of-regard image I4 that is output when the teacher's captured image is input to the point-of-regard image output model M1 may correspond to the teacher's point-of-regard image corresponding to the teacher's captured image. .

A teacher evaluation result is an evaluation result used as teacher data. In other words, the teacher evaluation result is the evaluation result used for training the machine learning model. The face observation book B1 shown in FIGS. 3 and 5 corresponds to the teacher evaluation result.

The teacher evaluation result includes evaluation results for each of the plurality of evaluation items. The evaluation items are items that serve as criteria for evaluation, and include, for example, the above-described stability of the working face and the autonomy of the excavation surface. In this embodiment, a case where evaluation results are indicated by numerical values will be described. For example, whether or not an item corresponds to an evaluation item may be indicated by a numerical value of "0" or "1", or a corresponding numerical value may be selected from three or more numerical values.

For example, the numerical value indicating the evaluation result may be selected from a plurality of predetermined numerical values such as "very good", "good", "bad", and "very bad". A score, such as a point, may be indicated. Note that the evaluation results are not limited to numerical values, and may be indicated by symbols, characters, or the like. Also, the evaluation result may include only one evaluation item. The evaluation result can be said to be the contents of the evaluation, and it can also be said to be the progress of the construction or the judgment result of good or bad.

The data storage unit 100 also stores not only the first teacher data D1 and the second teacher data D2, but also the point-of-regard image output model M1 and the evaluation result output model M2. These models are sometimes called artificial intelligence or engines. The data storage unit 100 stores programs and parameters (coefficients) of each of the point-of-regard image output model M1 and the evaluation result output model M2. Each of the point-of-regard image output model M1 and the evaluation result output model M2 stored in the data storage unit 100 has already been learned by each of the first learning unit 103 and the second learning unit 107, which will be described later. and the second teacher data D2, parameters and the like are adjusted.

A known algorithm can be used for the machine learning model itself that forms the basis of the point-of-regard image output model M1. For example, the point-of-regard image output model M1 outputs an output point-of-regard image when an input captured image is input, so a machine learning model used in image conversion can be used. For example, ``Automatic coloring of black-and-white photographs by learning global and local features using deep networks'' (Satoshi Iizuka, Edgar Simosela, Hiroshi Ishikawa, http://iizuka.cs.tsukuba.ac.jp/projects/colorization /en/), a machine learning model technique that converts a monochrome image into a color image may be used. In addition, for example, cycleGAN (Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, https://arxiv.org/pdf/1703.10593. pdf) may be used.

A known algorithm can also be used for the machine learning model that forms the basis of the evaluation result output model M2. When the input captured image and the input gaze point image are input, the evaluation result output model M2 outputs the corresponding output evaluation result, and the output evaluation result is regarded as the classification result of the input captured image and the input gaze point image. can be used, a so-called classification learner can be used. A classification learner is a machine learning model that classifies (labels) input data. For example, it outputs 0 or 1 as to whether or not it corresponds to a certain classification, or outputs a score indicating the probability that it corresponds to the classification. or For example, a technique called Grad-CAM that classifies objects such as dogs and cats shown in captured images may be used. Alternatively, for example, FasterRNN, Yolo, or SSD may be used.

Note that the data stored in the data storage unit 100 is not limited to the above examples. For example, the data storage unit 100 may store teacher input images that have not been evaluated by experts. Further, for example, the data storage unit 100 may store an input captured image, which will be described later. Alternatively, for example, the data storage unit 100 may store a program for learning each of the point-of-regard image output model M1 and the evaluation result output model M2. may store a program for creating

[Teacher gaze point image acquisition unit]
The teacher gaze point image acquisition unit 101 is realized mainly by the control unit 11 . A teacher gazing point image acquisition unit 101 acquires a teacher gazing point image. The teacher's point-of-regard image can be acquired by any method. In this embodiment, the teacher's point-of-regard image acquisition unit 101 acquires the teacher's point-of-regard image based on the evaluator's line of sight detected by the line-of-sight detection device 20. The case of acquisition will be explained.

The line-of-sight detection device 20 is an example of line-of-sight detection means according to the present invention. The line-of-sight detection means is not limited to the line-of-sight detection device 20, and may be any means capable of detecting the line of sight. For example, a camera of a personal computer, a tablet computer, or a smart phone may correspond to the line-of-sight detection means. Alternatively, for example, a line-of-sight sensor incorporated in a head-mounted display or smart glasses may correspond to the line-of-sight detection means. In this case, the expert evaluates the face while wearing a head-mounted display or smart glasses. As described above, in the present embodiment, a case will be described in which the expert evaluates the face by looking at the image and acquires the teacher gaze point image at that time. may be performed, and the teacher gaze point image at that time may be acquired.

A well-known tool can be used for the acquisition method of the teacher gaze point image itself. For example, the teacher gazing point image acquisition unit 101 records the position of the gazing point in time series based on the detection result of the line-of-sight detection device 20 . The teacher gaze point image acquisition unit 101 calculates the gaze time for each position on the screen based on the position of the gaze point recorded in chronological order. The teacher gaze point image acquisition unit 101 acquires the teacher gaze image by determining the pixel value of each pixel based on the gaze time at each position. The relationship between the gaze time and the pixel value may be determined in advance. For example, it may be determined so that the longer the gaze time, the closer to a predetermined color, or the longer the gaze time, the darker the color. good too.

In this embodiment, the teacher gazing point image acquisition unit 101 identifies the line of sight of the evaluator detected by the line of sight detection device 20 toward the screen on which the teacher captured image is displayed, and determines the identified line of sight. Based on , the teacher gaze image is obtained. The teacher-captured image may be displayed on the entire screen, or may be displayed on only part of the screen. It is assumed that the position of the area on the screen of the display unit 15 where the teacher's photographed image is displayed is recorded in advance in the data storage unit 100 by initial setting. The teacher's point-of-regard image acquisition unit 101 acquires the teacher's point-of-regard based on the line-of-sight toward the area where the teacher's photographed image is displayed (the point of gaze is the line-of-sight within the area) among the lines of sight detected by the line-of-sight detection device 20. The image is acquired, and the information of the line of sight to the outside of the area (the line of sight whose gaze point is outside the area) is not included in the teacher gaze point image. Note that the teacher-captured image may include some line of sight to the outside of the screen.

[First training data acquisition unit]
The first teacher data acquisition unit 102 is realized mainly by the control unit 11 . The first teacher data acquisition unit 102 acquires the first teacher data D1. In this embodiment, the first teacher data D1 is stored in the data storage unit 100, so the first teacher data acquisition unit 102 refers to the data storage unit 100 and acquires the first teacher data D1. When the first teacher data D1 is stored in another computer or an external information storage medium other than the study terminal 10, the first teacher data acquiring unit 102 retrieves the first teacher data D1 stored in the other computer or the external information storage medium. Acquire teacher data D1.

[1st learning section]
First learning unit 103 is implemented mainly by control unit 11 . The first learning unit 103 learns the point-of-regard image output model M1 based on the first teacher data D1. As for the learning method itself, a known machine learning model technique may be used, for example, a convolutional neural network or recursive neural network learning technique may be used. This also applies to the second learning unit 107, which will be described later.

The first learning unit 103 adjusts the parameters of the point-of-regard image output model M1 so as to obtain the relationship between the input and the output indicated by the first teacher data D1. For example, the first learning unit 103 converts each of the teacher-captured image and the teacher-gazing-point image of the first training data D1 into a feature quantity, and when the feature-value of the teacher-captured image is input, the feature-value of the teacher-gazing-point image is is output, the parameters of the point-of-regard image output model M1 are adjusted. Note that the first teacher data D1 may include pre-calculated and stored feature amounts of each of the teacher-captured image and the teacher-gazing-point image. In this case, the first learning unit 103 does not need to calculate feature amounts during learning.

[First input section]
First input unit 104 is realized mainly by control unit 11 . The first input unit 104 inputs an input captured image to the point-of-regard image output model M1.

The input captured image is a captured image input to the point-of-regard image output model M1. In other words, the input captured image is a captured image of a face to be evaluated by an evaluator (for example, an inexperienced evaluator). The captured image I3 shown in FIGS. 4 and 5 corresponds to the input captured image. In the present embodiment, the case where the input captured image is not stored in the first teacher data D1 and the second teacher data D2 will be described. may be the input captured image.

In the input photographed image, an evaluation target at a construction site is photographed. In this embodiment, since the face corresponds to the evaluation target, the input photographed image is an image of the face at the construction site of the teacher photographed image or another construction site. The input photographed image may show the same face at the construction site as the face photographed in the teacher photographed image (for example, the face after further excavation of the face photographed in the teacher photographed image), or a completely different face. A completely different face at another construction site at the site may be shown.

The extension, format, size, resolution, and number of bits of the input photographed image may be arbitrary, as in the case of the teacher photographed image. In the present embodiment, the face shooting conditions are the same between the teacher shot image and the input shot image, and the extension, format, size, resolution, and bit number of each of the teacher shot image and the input shot image are different from each other. Although they are the same, they may be different from each other. In addition, the positional relationship between the face and the camera (imaging direction of the face) is the same between the input captured image and the teacher captured image. Relationships may be subtly different.

In the present embodiment, an input photographed image is stored in the data storage unit 100, and the first input unit 104 acquires the input photographed image stored in the data storage unit 100. However, the input photographed image is , which can be obtained in any way. For example, the first input unit 104 may acquire the input captured image from a computer other than the study terminal 10 or an external information storage medium. Further, for example, the first input unit 104 may acquire the input captured image directly from the camera.

The first input unit 104 inputs the acquired input captured image to the point-of-regard image output model M1. In this embodiment, a case will be described in which an algorithm for calculating the feature amount of the input captured image is incorporated in the point-of-regard image output model M1. good too. In this case, the first input unit 104 may input an input captured image to an algorithm for calculating a feature amount, and input the feature amount of the input captured image output from the algorithm to the point-of-regard image output model M1. .

[Output gaze point image acquisition unit]
The output gaze point image acquisition unit 105 is realized mainly by the control unit 11 . The output point-of-regard image acquisition unit 105 acquires the output point-of-regard image output from the point-of-regard image output model M1.

The output point-of-regard image is the point-of-regard image output from the point-of-regard image output model M1. In other words, the output point-of-regard image indicates a portion that is assumed to be watched by an expert when it is assumed that the expert evaluates the input captured image. In the output gaze point image, the gaze point (degree of gaze) is represented by color. The point-of-regard image I4 shown in FIGS. 4 and 5 corresponds to the output point-of-regard image. In this embodiment, the case where the output gaze point image is not stored in the first teacher data D1 and the second teacher data D2 will be described. and second teacher data D2, and may be used as teacher data.

In this embodiment, the face corresponds to the evaluation target, and the input captured image captures the evaluation target at the construction site. indicate the part. The extension, format, size, resolution, and number of bits of the output gaze point image may be arbitrary, as is the case with the teacher gaze point image. In this embodiment, the teacher gazing point image and the output captured image have the same extension, format, size, resolution, and number of bits, but these may differ from each other.

[Second teacher data acquisition unit]
Second teacher data acquisition unit 106 is realized mainly by control unit 11 . The second teacher data acquisition unit 106 acquires the second teacher data D2. In this embodiment, the second teacher data D2 is stored in the data storage unit 100, so the second teacher data acquisition unit 106 refers to the data storage unit 100 and acquires the second teacher data D2. When the second teacher data D2 is stored in another computer or an external information storage medium other than the study terminal 10, the second teacher data acquisition unit 106 acquires the second teacher data D2 stored in the other computer or the external information storage medium. Acquire teacher data D2.

[Second study part]
Second learning unit 107 is implemented mainly by control unit 11 . The second learning unit 107 learns the evaluation result output model M2 based on the second teacher data D2. The second learning unit 107 adjusts the parameters of the evaluation result output model M2 so as to obtain the relationship between the input and the output indicated by the second teacher data D2. For example, the second learning unit 107 converts each of the teacher-captured image and the teacher-gazing-point image of the second training data D2 into a feature quantity, and when the feature quantity of each of the teacher-captured image and the teacher-gazing-point image is input, the teacher Parameters of the evaluation result output model M2 are adjusted so that evaluation results are output. Note that the feature amounts of each of the teacher captured image and the teacher gaze point image may be calculated in advance and stored in the second teacher data D2. In this case, the second learning unit 107 does not need to calculate feature amounts during learning.

[Input fixation point image acquisition unit]
The input gaze point image acquisition unit 108 is realized mainly by the control unit 11 . An input point-of-regard image acquisition unit 108 acquires an input point-of-regard image. In this embodiment, the input point-of-regard image acquisition unit 108 acquires an output point-of-regard image output when an input photographed image is input to the trained point-of-regard image output model M1 as an input point-of-regard image. do.

The input gazing point image is the gazing point image input to the gazing point image output model M1. In other words, the input point-of-regard image indicates the part to be viewed in the input captured image. The point-of-regard image I4 shown in FIGS. 4 and 5 corresponds to the input point-of-regard image. In the present embodiment, the face corresponds to the evaluation target, and the input photographed image is a photograph of the face at the construction site. indicates

In this embodiment, the output gaze point image corresponds to the input gaze point image, and the second input unit 109 acquires the output gaze point image output from the gaze point image output model M1 as the input gaze point image. As will be described, the input point-of-regard image can be obtained in any manner. For example, when the point-of-regard image output model M1 is not used, the second input unit 109 may acquire an input point-of-regard image based on an operation from the operation unit 14, or may acquire an input point-of-regard image from another device other than the learning terminal 10. The input gaze point image may be obtained from a computer or an external information storage medium.

[Second input section]
The second input unit 109 is implemented mainly by the control unit 11 . The second input unit 109 inputs an input captured image and an input gaze point image corresponding to the input captured image to the evaluation result output model M2. The input captured image is as described in the description of the first input unit 104 . The input point-of-regard image can be obtained by any method. In this embodiment, the input-point-of-regard image is obtained by the input-point-of-regard image obtaining unit 108, which will be described later. An input photographed image and an input point-of-regard image acquired by the input point-of-regard image acquiring unit 108 are input to the model.

The second input unit 109 inputs the input photographed image and the input gaze point image to the evaluation result output model M2. In this embodiment, a case will be described in which an algorithm for calculating the feature values of each of the input captured image and the input gaze point image is incorporated in the evaluation result output model M2. It may be prepared separately. In this case, the second input unit 109 inputs each of the input captured image and the input gazing point image to an algorithm for calculating the feature amount, and inputs the feature amount output from the algorithm to the evaluation result output model M2. Just do it.

[Output evaluation result acquisition unit]
The output evaluation result acquisition unit 110 is realized mainly by the control unit 11 . The output evaluation result acquisition unit 110 outputs an output evaluation of the face shown in the input captured image, which is output when the input captured image and the input gaze point image are input to the learned evaluation result output model. Get results.

The output evaluation result is the evaluation result output from the evaluation result output model M2. In other words, the output evaluation result indicates the evaluation result and estimated contents when it is assumed that an expert has evaluated the input captured image. The face observation list B2 shown in FIG. 5 corresponds to the output evaluation result. The output evaluation result indicates numerical values corresponding to the input photographed image and the input gaze point image for each evaluation item. The numerical value can be said to be the classification result of the input photographed image and the input gaze point image, and corresponds to the label in the classification learning device.

In this embodiment, the output evaluation result is displayed on the display unit 15, but the output evaluation result may be used for any purpose. For example, the output evaluation result may be printed from a printer, or may be sent as an attachment to e-mail or the like. Alternatively, for example, a file showing output evaluation results may be recorded in the learning terminal 10 or another computer.

[4. Processing executed in the present embodiment]
Next, processing executed by the evaluation support system S will be described. Here, a learning process for learning each of the point-of-regard image output model M1 and the evaluation result output model M2, and an evaluation support process for supporting the evaluator's evaluation work using these models will be described. do. The processes described below are executed by the control unit 11 operating according to the programs stored in the storage unit 12 . Also, the processing described below is an example of the processing executed by the functional blocks shown in FIG.

[4-1. learning process]
FIG. 9 is a flow diagram showing learning processing. As shown in FIG. 9, first, the control unit 11 causes the display unit 15 to display a teacher-captured image for the expert to evaluate based on the detection signal from the operation unit 14 (S100). In S<b>100 , the control unit 11 causes the display unit 15 to display the teacher-photographed image selected by the expert by operating the operation unit 14 from among the teacher-photographed images stored in the storage unit 12 . The teacher-captured image displayed in S100 is an image for which a corresponding teacher gaze point image has not been created, and is a teacher-captured image that has not yet been stored in the first teacher data D1.

The control unit 11 acquires the detection result of the expert's line of sight by the line of sight detection device 20 (S101). In S<b>101 , the control unit 11 records in the storage unit 12 the line of sight of the expert detected by the line of sight detection device 20 (for example, the coordinates of the gaze point) in time series. The expert evaluates the face shown in the teacher-captured image displayed on the display unit 15, and inputs the evaluation results of the evaluation items in the face observation book. The evaluation result may be input from the operation unit 14 or may be input from a tablet terminal or the like at hand of the expert and sent to the learning terminal 10 . In addition, for example, the evaluation results may be entered in a paper face observation book, captured by a scanner after the fact, or input from the operation unit 14 .

The control unit 11 determines whether or not the evaluation by the expert has been completed based on the detection signal from the operation unit 14 (S102). In S102, the control unit 11 determines whether or not all the evaluation items of the face observation sheet have been input and a predetermined end operation has been performed based on the input result from the operation unit 14, the expert's terminal, or the like. do.

If it is determined that the evaluation by the expert has not been completed (S102; N), the evaluation by the expert has not been completed, so the process returns to S101. In this case, the expert continues to evaluate the face, and the detection result of the expert's line of sight is recorded.

On the other hand, when it is determined that the input of the evaluation result has been received (S102; Y), the control unit 11 creates a teacher gaze point image based on the result of detecting the line of sight of the expert (S103). In S103, the control unit 11 creates a teacher gaze point image based on the detection results of the line of sight from the start of the evaluation by the expert until the end of the evaluation. The control unit 11 creates a teacher gaze point image such that the longer the expert gazes, the darker the color becomes.

The control unit 11 stores the pair of the teacher captured image displayed in S100 and the teacher gaze point image created in S103 in the first teacher data D1 (S104). The control unit 11 stores a pair of the teacher captured image displayed in S100, the teacher gaze point image created in S103, and the teacher evaluation result, which is the evaluation completed in S102, in the second teacher data D2 (S105). . At S104, the first teaching data D1 is created, and at S105, the second teaching data D2 is created.

The control unit 11 determines whether or not to execute the learning process based on the detection signal from the operation unit 14 (S106). The learning process may be executed at any timing, for example, when a predetermined operation is performed from the operation unit 14. Note that the learning process may be executed when a predetermined time has passed, or may be executed each time an expert evaluates. Alternatively, for example, it may be executed when a certain number or more of new data are added to each of the first teacher data D1 and the second teacher data D2.

If it is determined not to execute the learning process (S106; N), this process ends. In this case, the processing may be repeated from S1, and new data may be added to each of the first teacher data D1 and the second teacher data D2. Further, the subsequent processing of S107 and S108 can be executed at any timing, and does not have to be after new data is added to each of the first teacher data D1 and the second teacher data D2.

On the other hand, when it is determined to execute the learning process (S106; Y), the control unit 11 executes the learning process of the point-of-regard image output model M1 based on the first teacher data D1 (S107). In S107, the control unit 11 adjusts the parameters of the point-of-regard image output model M1 based on a known learning algorithm so as to obtain the relationship between the input and the output indicated by the first teacher data D1.

The control unit 11 executes the learning process of the evaluation result output model M2 based on the second teacher data D2 (S108), and ends this process. In S108, the control unit 11 adjusts the parameters of the evaluation result output model M2 based on a known learning algorithm so as to obtain the relationship between the input and the output indicated by the second teacher data D2.

[4-2. Evaluation support processing]
FIG. 10 is a flowchart showing evaluation support processing. The evaluation support process is executed after the learning process is executed. As shown in FIG. 10, first, the control unit 11 acquires an input captured image (S200). In S<b>200 , among the input captured images stored in the storage unit 12 , the input captured image selected by the evaluator by operating the operation unit 14 is obtained.

The control unit 11 inputs the input captured image acquired in S200 to the point-of-regard image output model M1 (S201), and acquires the output point-of-regard image output from the point-of-regard image output model M1 as the input point-of-regard image. (S202). When the input captured image is input in S201, the point-of-regard image output model M1 calculates the feature amount of the input captured image. The point-of-regard image output model M1 outputs an output point-of-regard image based on the calculated feature amount.

The control unit 11 inputs the input captured image acquired in S200 and the input gaze point image acquired in S202 to the evaluation result output model M2 (S203), and outputs the output from the evaluation result output model M2. An evaluation result is acquired (S204). When the input photographed image and the input point-of-regard image are input in S203, the evaluation result output model M2 calculates the feature amount of each of the input photographed image and the input point-of-regard image. The evaluation result output model M2 outputs output evaluation results based on these feature amounts.

The control unit 11 causes the display unit 15 to display the output evaluation result obtained in S204 (S205), and the process ends. The evaluator uses the output evaluation result displayed on the display unit 15 as a reference for evaluation, prints the output evaluation result displayed on the display unit 15, and performs evaluation work for the face of the day.

According to the first configuration of the evaluation support system S, by inputting an input captured image to the point-of-regard image output model M1 learned based on the first teacher data D1 and obtaining an output point-of-regard image, Can assist the evaluator in their work. For example, by inputting the output point-of-regard image as the input point-of-regard image into the evaluation result output model M2 to obtain the output evaluation result, the evaluator can obtain a hint that an expert would obtain such an evaluation result. , the output evaluation result can be used as it is as the evaluation result of the day.

In addition, the evaluation support system S makes the point-of-regard image output model M1 learn the relationship between the teacher's photographed image and the teacher's point-of-regard image acquired by the line-of-sight detection device 20, thereby obtaining the detection result of the expert's line of sight. The accuracy of the point-of-regard image output model M1 can be improved by making the point-of-regard image output model M1 learn.

In addition, the evaluation support system S identifies the line of sight of the evaluator detected by the line of sight detection device 20 toward the screen on which the teacher-captured image is displayed, and acquires the teacher gaze point image. It is possible to eliminate the line of sight irrelevant to the evaluation of , effectively increasing the accuracy of the point-of-regard image output model M1.

In addition, the evaluation support system S inputs the input captured image and the output gaze point image to the learned evaluation result output model M2, and obtains the output evaluation result of the face shown in the input captured image. Information indicating such an evaluation result for an expert is output, and the work of the evaluator can be effectively supported.

In addition, the evaluation support system S effectively increases the accuracy of the point-of-regard image output model M1 by setting the size of each of the teacher's photographed image, the input photographed image, the teacher's point-of-regard image, and the output point-of-regard image to be the same. be able to.

In addition, the evaluation support system S can support the work of the evaluator of the tunnel construction by setting the evaluation target to be the face.

According to the second configuration of the evaluation support system S, the input photographed image and the output gaze point image are input to the evaluation result output model M2 learned based on the second teacher data D2, and the output evaluation result is obtained. By doing so, it is possible to support the work of the evaluator. For example, the evaluator can obtain a hint that an expert will have such an evaluation result, or can use the output evaluation result as it is as the evaluation result of the day.

In addition, the evaluation support system S causes the evaluation result output model M2 to learn the relationship between the teacher's photographed image, the teacher gaze point image acquired by the line-of-sight detection device 20, and the teacher's evaluation result. The accuracy of the evaluation result output model M2 can be improved by making the evaluation result output model M2 learn the detection result.

In addition, the evaluation support system S identifies the line of sight of the evaluator detected by the line of sight detection device 20 toward the screen on which the teacher-captured image is displayed, and acquires the teacher gaze point image. It is possible to eliminate the line of sight irrelevant to the evaluation of , effectively increasing the accuracy of the evaluation result output model M2.

Further, the evaluation support system S inputs an input photographed image to the learned point-of-regard image output model M1, acquires the output gaze-point image as an input gaze-point image, and inputs it to the evaluation result output model M2. , the input gaze point image can be automatically acquired to effectively support the work of the evaluator.

In addition, the evaluation support system S effectively increases the accuracy of the evaluation result output model M2 by setting the size of each of the teacher captured image, the input captured image, the teacher gaze point image, and the output gaze point image to be the same size. can be done.

In addition, the evaluation support system S can support the work of the evaluator of the tunnel construction by setting the evaluation target to be the face.

[5. Modification]
It should be noted that the present invention is not limited to the embodiments described above. Modifications can be made as appropriate without departing from the gist of the present invention.

FIG. 11 is a functional block diagram according to a modification. As shown in FIG. 11, in the modified example described below, a characteristic information acquisition unit 111 is implemented in addition to the functions described in the embodiment. Although the feature information acquisition unit 111 is implemented by the study terminal 10, it may be implemented by another computer such as a server computer.

[5-1. Modification of First Configuration]
(1-1) First, a modification of the first configuration will be described. For example, even if the tunnel construction is the same, the location that the evaluator should look at may change between the mountain tunnel construction and the underground tunnel construction. For this reason, the point-of-regard image output model M1 may be made to learn the feature information of the construction work, and an output point-of-regard image corresponding to the feature information may be output.

The feature information is information relating to the features of construction, and can also be called a type of construction. For example, the feature information is the location of the construction site, construction method, ground, weather, average precipitation, equipment, materials, or workers. The feature information may indicate the feature of each of the plurality of items, or may indicate only one of the features. Each item of the characteristic information may be indicated by a numerical value, or may be indicated by a symbol or a character.

The first training data D1 of this modified example indicates the relationship between the characteristic information of the construction at the construction site, the teacher-captured image, and the teacher gaze point image. That is, the first teacher data D1 stores a pair of feature information, a teacher-captured image, and a teacher gaze point image. The feature information and the teacher captured image correspond to the input, and the teacher gaze point image corresponds to the output. The feature information stored in the first training data D1 is the feature information of the construction indicated by the teacher-captured image. or obtained.

The evaluation support system S of this modified example includes a feature information acquisition unit 111 . The feature information acquisition unit 111 is implemented mainly by the control unit 11 . The characteristic information acquisition unit 111 acquires construction characteristic information corresponding to the input captured image. The feature information of the construction corresponding to the input captured image is the feature information of the construction indicated by the input captured image. be done.

For example, an evaluator in charge of a construction site of a mountain tunnel inputs from the operation unit 14 that it is an input photographed image of a mountain tunnel. Input that there is from the operation unit 14 . Note that the characteristic information may be acquired by any method instead of being input from the operation unit 14 . For example, the feature information may be associated in advance with the input captured image, or the feature information may be associated in advance with the person in charge.

The first input unit 104 inputs the feature information acquired by the feature information acquisition unit 111 and the input captured image to the point-of-regard image output model M1. The point-of-regard image output model M1 converts both the feature information and the input captured image into feature quantities, and outputs an output point-of-regard image. The feature information and the feature amount of the input captured image may also be expressed in any format such as vector or array. As described in the embodiment, the algorithm for calculating the feature amount may be outside the point-of-regard image output model M1.

According to the modified example (1-1), the point-of-regard image output model M1 learns the relationship between the characteristic information of the construction at the construction site, the teacher-captured image, and the teacher-at-a-glance image. Therefore, it is possible to obtain an output gazing point image with a higher accuracy, thereby improving the accuracy of the output gazing point image.

(1-2) For example, in the modified example (1-1), the case where feature information is learned by the point-of-regard image output model M1 has been described. may be prepared separately. For example, the point-of-regard image output model M1 for mountain tunnels and the point-of-regard image output model M1 for underground tunnels may be prepared separately.

The data storage unit 100 of this modification stores the first teacher data D1 and the point-of-regard image output model M1 for each construction feature information. For example, the data storage unit 100 stores the first teacher data D1 and the point-of-regard image output model M1 for each location of the construction site, or stores the first teacher data D1 and the point-of-regard image output model M1 for each construction method. do. The method of creating the first teacher data D1 itself is as described in the embodiment, and the first teacher data D1 is stored in the data storage unit 100 in association with the construction feature information.

The first learning unit 103 of this modified example learns the point-of-regard image output model M1 based on the first teacher data D1 corresponding to each feature information of construction at the construction site. The first learning unit 103 learns the point-of-regard image output model M1 associated with each piece of feature information based on the first teacher data D1 associated with the feature information. The learning method itself for the point-of-regard image output model M1 is as described in the embodiment.

The first input unit 104 inputs an input captured image to the point-of-regard image output model M1 corresponding to the feature information acquired by the feature information acquisition unit 111 . The first input unit 104 inputs the input captured image to the target point image output model M1 associated with the feature information of the input captured image among the target point image output models M1 stored in the data storage unit 100 . The processing after the input photographed image is input is as described in the embodiment.

According to the modified example (1-2), by preparing the point-of-regard image output model M1 for each feature information of the construction work at the construction site, it is possible to acquire the output point-of-regard image corresponding to the feature of the construction work, and output it. It is possible to improve the accuracy of the gaze point image.

(1-3) Further, for example, the output point-of-regard image output by the point-of-regard image output model M1 may be used for any purpose, and may be used for purposes other than being input to the evaluation result output model M2. good too. For example, the output gaze point image may be printed from a printer, or may be sent as an attachment to e-mail or the like. Alternatively, for example, a file indicating the output gaze point image may be recorded in the learning terminal 10 or another computer. In this case, the output point-of-regard image may be used for training an inexperienced evaluator, and the output point-of-regard image may be shown to the evaluator to teach the expert what part to see. Further, for example, the teacher gaze point image may be acquired without using the detection result of the line-of-sight detection device 20 . For example, an expert may be shown a teacher-captured image, and may be instructed to manually designate a portion that was viewed with emphasis during evaluation.

[5-2. Modification of Second Configuration]
(2-1) Next, a modification of the second configuration will be described. For example, in the modified example (1-1), the case where the point-of-regard image output model M1 is made to learn the feature information of construction work has been described. You may let M2 learn.

The second teacher data D2 of this modified example indicates the relationship between the feature information of the construction at the construction site, the teacher's photographed image, the teacher's gaze point image, and the teacher's evaluation result. That is, the second teacher data D2 stores a pair of feature information, a teacher-captured image, a teacher-gazing-point image, and a teacher evaluation result. The feature information, the teacher's photographed image, and the teacher's point-of-regard image correspond to the input, and the teacher's evaluation result corresponds to the output. The feature information stored in the second training data D2 is the feature information of the construction indicated by the teacher-captured image. or obtained.

The second input unit 109 inputs the feature information acquired by the feature information acquisition unit 111, the input captured image, and the input gaze point image to the evaluation result output model M2. The evaluation result output model M2 converts each of the feature information, the input captured image, and the input gazing point image into feature quantities, and outputs an output evaluation result. The feature information, the input captured image, and the feature amount of the input gaze point image may also be expressed in any format such as vector or array. As described in the embodiment, the algorithm for calculating the feature amount may be outside the evaluation result output model M2.

According to the modified example (2-1), the evaluation result output model M2 learns the relationship between the characteristic information of the construction work at the construction site, the teacher-captured image, the teacher-gazing-point image, and the teacher evaluation result. It is possible to obtain an output evaluation result according to the characteristics of , and to improve the accuracy of the output evaluation result.

(2-2) For example, in the modified example (1-2), a case was explained in which a dedicated gaze point image output model M1 was separately prepared for each piece of construction feature information, but the evaluation result output model M2 is also Similarly, a dedicated evaluation result output model M2 may be separately prepared for each construction feature information. For example, an evaluation result output model M2 for mountain tunnels and an evaluation result output model M2 for underground tunnels may be prepared separately.

The data storage unit 100 of this modified example stores the second teacher data D2 and the evaluation result output model M2 for each construction feature information. For example, the data storage unit 100 stores the second teacher data D2 and the evaluation result output model M2 for each location of the construction site, or stores the second teacher data D2 and the evaluation result output model M2 for each construction method. The method of creating the second teacher data D2 itself is as described in the embodiment, and the second teacher data D2 is stored in the data storage unit 100 in association with the construction feature information.

The second learning unit 107 learns the evaluation result output model M2 based on the second teacher data D2 corresponding to each feature information of construction at the construction site. The second learning unit 107 learns the evaluation result output model M2 associated with the feature information for each feature information based on the second teacher data D2 associated with the feature information. The learning method itself for the evaluation result output model M2 is as described in the embodiment.

The second input unit 109 inputs the input captured image and the input gaze point image to the evaluation result output model M2 corresponding to the feature information acquired by the feature information acquisition unit 111 . The second input unit 109 inputs the input captured image to the evaluation result output model M2 associated with the feature information of the input captured image among the evaluation result output models M2 stored in the data storage unit 100 . The processing after the input photographed image is input is as described in the embodiment.

According to the modified example (2-2), by preparing the evaluation result output model M2 for each characteristic information of the construction work at the construction site, it is possible to obtain the output evaluation result corresponding to the characteristic of the construction work. accuracy can be improved.

(2-3) Further, for example, the output gaze point image output by the gaze point image output model M1 may be stored as a teacher gaze point image in the second teacher data D2. That is, the second teacher data D2 need not only be the teacher gaze point image acquired by detecting the line of sight of the expert.

The teacher gazing point image acquisition unit 101 of this modified example acquires, as a teacher gazing point image, an output gazing point image that is output when a teacher captured image is input to the trained gazing point image output model M1. . The method itself for acquiring the output gaze point image by the teacher gaze point image acquisition unit 101 is as described in the embodiment. The second teacher data D2 of this modified example indicates the relationship between the teacher captured image, the teacher gazing point image acquired by the teacher gazing point image acquisition unit 101, and the teacher evaluation result.

According to the modification (2-3), the output point-of-regard image output from the point-of-regard image output model M1 is stored in the second teacher data D2 as the teacher point-of-regard image, and is used as the second teacher data. The number of D2 can be increased, and the accuracy of the evaluation result output model M2 can be improved.

(2-4) Further, for example, the input point-of-regard image input to the point-of-regard image output model M1 may not be the output point-of-regard image output by the point-of-regard image output model M1. For example, the input gazing point image may be acquired by the evaluator selecting a portion that the evaluator wants to focus on on the display unit 15 displaying the input captured image. In this case, if the evaluator is an expert, he or she can select an accurate part, so that the expert can automatically create a face observation list simply by selecting the part that the expert wants to focus on. can support In addition, for example, when the place where the face should be seen in the evaluation work on a certain day is not much different from the previous day, the part that the expert saw intensively on the previous day is recorded, and that part becomes the input gaze point image. may In addition, for example, when similar construction was done in another place in the past, if the place where the face should be seen is not much different from that time, the part that the expert saw intensively at that time It may be recorded and the relevant portion may be used as the input gaze point image.

[5-3. Other Modifications]
(3) Further, for example, the above modifications may be combined.

Also, for example, the case where each of the teacher captured image, the input captured image, the teacher gazing point image, and the output gazing point image has the same size has been described, but these sizes may be different from each other. In addition, for example, these resolutions and the like may be different from each other.

Also, for example, the algorithm of the point-of-regard image output model M1 is not limited to the example described in the embodiment. For example, each pixel may be labeled by setting a threshold for the teacher gaze image, and the labeled image may be classified by semantic segmentation as teacher data. That is, by dividing the number of stages of colors (for example, 256 stages) that can be represented by the point-of-regard image into predetermined stages with an arbitrary threshold value and dropping them into labels based on pseudo semantic segmentation, the output point-of-regard image is created. can be

Further, for example, in the embodiment and the modified example, the case of mainly evaluating the face of a mountain tunnel was taken as an example, but it is also applicable to the scene of evaluating the face of an underground tunnel, and has been described in the embodiment and the modified example. The process may support the work of the evaluator. Further, for example, although the case where the evaluation support system S is used for face evaluation has been described, it can be applied to any evaluation target.

For example, it can be applied to construction work other than tunnel construction, and can also be applied to situations such as evaluating concrete cracks and evaluating the shape and balance of buildings. For example, the evaluation target may be a building, bridge, dam, steel frame, column, wall, or the like. When evaluating these evaluation targets, the same processing as the processing described in the embodiment and the modified example is performed to acquire an output gaze point image indicating a portion to be viewed by the expert, and to estimate the result of evaluation by the expert. It suffices to acquire the output evaluation result to be used.

Further, for example, in the embodiment, the case where each function is realized by the learning terminal 10 has been described, but when the evaluation support system S includes a plurality of computers, each computer may share the function. For example, the data storage unit 100 is implemented by a server computer, and the learning terminal 10 stores first teacher data D1, second teacher data D2, gaze point image output model M1, and evaluation result output model M2 stored in the server computer. Each may be used.

S Evaluation support system 10 Learning terminal 11 Control unit 12 Storage unit 13 Communication unit 14 Operation unit 15 Display unit 20 Line of sight detection device B1, B2 Face observation record D1 First teacher data D2 Second Teacher data I1, I3 photographed images I2, I4 gaze point images M1 gaze point image output model M2 evaluation result output model 100 data storage unit 101 teacher gaze point image acquisition unit 102 first teacher data acquisition unit 103 first learning unit 104 first input unit 105 output gaze point image acquisition unit 106 second teacher data acquisition unit 107 second learning unit 108 input gaze point image acquisition unit 109 second input unit 110 output evaluation result acquisition unit 111 feature information acquisition unit;

Claims (11)

Acquisition of teacher data for acquiring teacher data indicating a relationship between a teacher-photographed image of an evaluation object photographed at a construction site, teacher gaze point information corresponding to the teacher-photographed image, and a teacher evaluation result of the evaluation object by an evaluator means and
learning means for learning an evaluation result output model based on the teacher data;
input means for inputting an input captured image and input gaze point information corresponding to the input captured image to the evaluation result output model;
output evaluation result acquisition means for acquiring an output evaluation result output from the evaluation result output model;
An evaluation support system comprising: The evaluation support system further includes teacher gaze point information acquisition means for acquiring the teacher gaze point information based on the gaze of the evaluator detected by the gaze detection means,
The teacher data indicates the relationship between the teacher captured image, the teacher gaze point information acquired by the teacher gaze point information acquiring means, and the teacher evaluation result.
The evaluation support system according to claim 1, characterized by: The teacher gaze point information acquisition means includes:
identifying, from among the line of sight of the evaluator detected by the line of sight detection means, a line of sight toward a screen on which the teacher-captured image is displayed;
obtaining the teacher gaze point information based on the identified line of sight;
3. The evaluation support system according to claim 2, characterized by: The teacher data indicates the relationship between the feature information of construction at the construction site, the teacher-captured image, the teacher gaze information, and the teacher evaluation result,
The evaluation support system further includes feature information acquisition means for acquiring feature information of the construction work corresponding to the input captured image,
The input means inputs the feature information acquired by the feature information acquisition means, the input captured image, and the input gaze point information to the evaluation result output model.
4. The evaluation support system according to any one of claims 1 to 3, characterized by: The learning means learns the evaluation result output model based on the teacher data corresponding to the feature information for each piece of feature information of construction at the construction site,
The evaluation support system further includes feature information acquisition means for acquiring feature information of the construction work corresponding to the input captured image,
The input means inputs the input captured image and the input gaze point information to the evaluation result output model corresponding to the feature information acquired by the feature information acquisition means.
5. The evaluation support system according to any one of claims 1 to 4, characterized by: The evaluation support system includes a teacher point-of-regard information acquisition means for acquiring, as the teacher point-of-regard information, output gaze-point information output when the teacher-captured image is input to a learned point-of-regard information output model. further comprising
The teacher data indicates the relationship between the teacher captured image, the teacher gaze point information acquired by the teacher gaze point information acquiring means, and the teacher evaluation result.
6. The evaluation support system according to any one of claims 1 to 5, characterized by: The evaluation support system includes input point-of-regard information acquisition means for acquiring, as the input point-of-regard information, output point-of-regard information output when the input photographed image is input to a learned point-of-regard information output model. further comprising
The input means inputs the input photographed image and the input gaze point information acquired by the input gaze point information acquisition means into the evaluation result output model.
7. The evaluation support system according to any one of claims 1 to 6, characterized by: each of the teacher-captured image and the input-captured image has the same size;
Each of the teacher point-of-regard information and the input point-of-regard information is an image of the same size as each of the teacher-captured image and the input captured image, in which the point-of-regard is represented by color.
The evaluation support system according to any one of claims 1 to 7, characterized by: The evaluation target is a tunnel face,
The teacher-captured image is an image of a tunnel face captured at the construction site,
The teacher gaze point information indicates the gaze point when the evaluator evaluates the tunnel face,
The input photographed image is an image obtained by photographing a tunnel face at the construction site or another construction site,
The input gaze point information indicates a portion to be viewed when evaluating the tunnel face shown in the input captured image,
The evaluation support system according to any one of claims 1 to 8, characterized by: Acquisition of teacher data for acquiring teacher data indicating a relationship between a teacher-photographed image of an evaluation object photographed at a construction site, teacher gaze point information corresponding to the teacher-photographed image, and a teacher evaluation result of the evaluation object by an evaluator a step;
a learning step of learning an evaluation result output model based on the teacher data;
an input step of inputting an input captured image and input gaze point information corresponding to the input captured image to the evaluation result output model;
an output evaluation result obtaining step for obtaining an output evaluation result output from the evaluation result output model;
An evaluation support method characterized by comprising: Acquisition of teacher data for acquiring teacher data indicating a relationship between a teacher-photographed image of an evaluation object photographed at a construction site, teacher gaze point information corresponding to the teacher-photographed image, and a teacher evaluation result of the evaluation object by an evaluator means,
learning means for learning an evaluation result output model based on the teacher data;
input means for inputting an input captured image and input gaze point information corresponding to the input captured image to the evaluation result output model;
output evaluation result acquisition means for acquiring output evaluation results output from the evaluation result output model;
A program that allows a computer to function as a

Images