WO2022209912A1

WO2022209912A1 - Concentration value calculation system, concentration value calculation method, program, and concentration value calculation model generation system

Info

Publication number: WO2022209912A1
Application number: PCT/JP2022/012007
Authority: WO
Inventors: 徹臼倉
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2021-03-30
Filing date: 2022-03-16
Publication date: 2022-10-06
Also published as: JP7531148B2; JPWO2022209912A1

Abstract

A concentration value calculation system (100) comprises an image acquisition unit (11) that acquires a stream of images obtained by imaging a person (99), a region acquisition unit (12) that acquires a plurality of concentration target regions each corresponding to each of a plurality of concentration target objects to be gazed at by the person (99), and a concentration value calculation unit (13) that calculates a concentration value of the person (99), wherein the concentration value calculation unit (13) calculates the orientation of the face of or orientation of the line of sight of the person (99) from the acquired stream of images, determines if the person (99) is in a gaze state in which the person (99) is gazing at any of the plurality of concentration target objects on the basis of the calculated orientation of the face of or orientation of the line of sight of the person (99) and the plurality of acquired concentration target regions, and calculates and outputs a concentration value of the person (99) on the basis of the result of determination.

Description

Concentrated Value Calculation System, Concentrated Value Calculation Method, Program, and Concentrated Value Calculation Model Generation System

The present disclosure relates to a concentration value calculation system and a concentration value calculation method.

Conventionally, an information processing device that calculates a person's degree of concentration (also called a concentration value) is known. For example, in the information processing apparatus described in Patent Literature 1, the maximum value of the concentration value is set to 100, and the sum of the amount of change in facial expression and the amount of change in action multiplied by the facing rate is subtracted to obtain the concentration value. calculate.

JP 2014-120137 A

In the conventional information processing apparatus described above, the more concentrated one is looking at one object, the higher the calculated concentration value becomes. However, in reality, the concentration value of a subject looking at one object is not necessarily high, and the concentration value of a subject looking at a plurality of objects is not necessarily low. For example, when a target person who is doing intellectual work such as study or office work is watching TV, the conventional information processing apparatus calculates a high concentration value even though it cannot be said that the person is concentrating. be. Further, when the user alternately looks at a reference book and a notebook during study, the conventional information processing apparatus calculates a low concentration value even though it can be said that the student is concentrating on the study. As described above, the conventional information processing apparatus has a problem that the accuracy of the calculated concentration value is low.

Therefore, the present disclosure provides a concentration value calculation system and a concentration value calculation method capable of calculating concentration values with higher accuracy.

In order to solve the above problems, a concentration value calculation system according to an aspect of the present disclosure includes: an image acquisition unit that acquires an image stream in which a subject is imaged; A region acquisition unit that acquires a plurality of concentration target regions, each of which corresponds to each of a plurality of concentration targets to be watched by the target person, and a concentration value calculation unit that calculates the concentration value of the target person. and the concentration value calculation unit calculates the face orientation or line-of-sight orientation of the subject from the acquired image stream, and calculates the calculated face orientation or line-of-sight orientation of the subject; Determining whether or not the target person is gazing at any one of the plurality of focus target objects based on the obtained plurality of focus target regions, Calculate and output the concentration value.

Further, in a concentration value calculation method according to an aspect of the present disclosure, an image stream in which a subject is imaged is acquired, and a plurality of concentration target areas, each of the plurality of concentration target areas, each of which is the subject's Obtaining a plurality of intensive target areas corresponding to each of a plurality of intensive target objects to be gazed at, and calculating the direction of the face or the direction of the line of sight of the target from the obtained image stream, and calculating the target person. determining whether or not the target person is in a gaze state in which he or she is gazing at any one of the plurality of focused objects based on the orientation of the face or the orientation of the line of sight and the plurality of acquired focused target areas; A concentration value of the subject is calculated and output based on the determination result.

Also, one aspect of the present disclosure can be implemented as a program for causing a computer to execute the concentration value calculation method. Alternatively, it can be realized as a computer-readable recording medium storing the program.

According to the present disclosure, the concentration value can be calculated with higher accuracy.

FIG. 1 is a diagram for explaining a usage example of a concentration value calculation system according to an embodiment. FIG. 2 is a block diagram showing the configuration of the concentration value calculation system according to the embodiment. FIG. 3 is a diagram for explaining a concentration target area according to the embodiment. FIG. 4 is a flow chart showing a concentration value calculation method according to the embodiment. FIG. 5 is a diagram for explaining determination of a gaze state according to the embodiment. FIG. 6 is a diagram for explaining transition state determination according to the embodiment. FIG. 7 is a diagram for explaining concentration values calculated in the embodiment. FIG. 8 is a diagram explaining another example of the focused object. FIG. 9 is a block diagram showing a functional configuration of a concentration value calculation unit according to another example.

(Knowledge leading to the present disclosure)
As shown in Japanese Patent Laid-Open No. 2002-200012, an information processing apparatus that calculates a concentration value of a person is conventionally known. The information processing device disclosed in Patent Literature 1 can calculate the concentration value only in a specific situation. Specifically, in the information processing apparatus, the more concentrated one object is viewed, the higher the calculated concentration value. Therefore, in order to use the information processing device for a target person who is working and calculate the concentration value of the target person for the work, it is necessary that the work performed by the target person is a work performed while gazing at one object. , and can only be applied to limited applications.

For example, an office worker who performs office work using a computer equipped with a display unit and a sub-display naturally selects both the display unit provided in the computer and the sub-display (that is, alternatively to). Therefore, in a configuration in which a concentration value is calculated based only on gazing at a display unit provided in a computer, an inaccurate concentration value may be calculated in a work scene in which the user gazes at the sub-display even in a concentrated state. In addition, it is naturally assumed that a student who studies using a tablet terminal that reproduces a lecture movie, a text, and a notebook selectively gazes at all of the tablet terminal, the text, and the notebook. Therefore, in a configuration in which a concentration value is calculated based only on gazing at a tablet terminal, an inaccurate concentration value may be calculated in a work situation in which the user gazes at a text or a notebook even in a concentrated state.

Thus, the concentration value calculated using the information processing device cannot be said to have high accuracy.

The present disclosure has been made in view of the above circumstances, and provides a concentration value calculation system and the like that can be applied to a wide range of uses and that can calculate a subject's concentration value with high accuracy.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be noted that all of the embodiments described below represent comprehensive or specific examples of the present disclosure. Therefore, numerical values, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present disclosure. Therefore, among constituent elements in the following embodiments, constituent elements not described in independent claims of the present disclosure will be described as optional constituent elements.

In addition, each figure is a schematic diagram and is not necessarily strictly illustrated. Therefore, scales and the like are not always the same in each drawing. In each figure, the same reference numerals are assigned to substantially the same configurations, and overlapping descriptions are omitted or simplified.

Note that in the embodiment described below, the first concentrated object and the second concentrated object are exemplified as the plurality of concentrated objects, but there may be three or more concentrated objects.

(Embodiment)
[Configuration of centralized value calculation system]
First, using FIG. 1, a concentration value calculation system according to an embodiment will be described. FIG. 1 is a diagram for explaining a concentration value calculation system according to an embodiment.

As shown in FIG. 1, a concentration value calculation system 100 (see FIG. 2 to be described later) according to the present embodiment is built in, for example, a computer (an example of a first concentration object 97) used by a subject 99. Realized. By implementing the concentration value calculation system 100 in a form incorporated in a computer or the like used by the subject 99, a camera, display, etc. mounted on the computer can be used as a part of the concentration value calculation system 100. In this embodiment, an externally connected camera attached to a sub-display (an example of the second focused object 98) used by the subject 99 is used as the imaging device 20. FIG.

In this way, by incorporating the concentration value calculation system 100 into the computer or the like used by the subject 99, the input to the concentration value calculation system 100 can be obtained using a camera, and the Output can be presented using a display. Further, when the work performed by the subject 99 is work using a computer, the computer used for the work can be used to calculate the concentration value of the subject 99 in parallel with the work. It should be noted that the centralized value calculation system 100 may have a part of processing functions, information storage functions, etc. implemented by a cloud server or the like.

The concentration value calculation system 100 according to the present disclosure is a system that uses an image stream in which the subject 99 is captured and calculates the concentration value of the subject 99 on the image stream. Therefore, if the concentration value calculation system 100 can acquire an image stream in which the subject 99 is imaged, the concentration value calculation system 100 can concentrate even in a situation where the gaze target of the subject 99 shifts to each of two or more focused objects. A value can be calculated. In other words, the concentration value calculation system 100 can be applied to the subject 99 who gazes at each of a plurality of concentration objects in a time division manner.

Next, with reference to FIG. 2, the functional configuration of the concentration value calculation system 100 according to this embodiment will be described in detail. FIG. 2 is a block diagram showing the configuration of the concentration value calculation system according to the embodiment.

As shown in FIG. 2, the concentration value calculation system 100 according to the present embodiment includes an arithmetic device 10, an imaging device 20, a storage device 30, and an output device 40. Each device constituting the centralized value calculation system 100 may be housed in one housing or the like and integrated, or may be connected to each other via a communication line to form a plurality of individual devices. may be implemented as a device of

The computing device 10 has an image acquisition unit 11 , an area acquisition unit 12 , and a concentration value calculation unit 13 . Arithmetic device 10 is implemented by a processor, a memory, and a program executed using these. The computing device 10 is installed, for example, as one of the functions in a computer, which is an example of the first concentration object 97 .

The image acquisition unit 11 is a functional unit that acquires an image stream in which the subject 99 is imaged. For example, the image acquisition unit 11 acquires an image stream in which the subject 99 is imaged by the imaging device 20 . Note that the image acquisition unit 11 may be integrated with the imaging device 20 . In one example of the present embodiment, the concentration value of the subject 99 is immediately calculated from the image stream acquired by the image acquisition section 11 . Here, "immediately" includes a delay of several milliseconds to several seconds considering the time required for calculation processing, data transfer, and the like.

Note that the image acquisition unit 11 may be implemented in any way as long as it can acquire an image stream. For example, the image acquisition unit 11 may acquire an image stream stored in the storage device 30 in the past. In this way, the measurement of the concentration value by the concentration value calculation system 100 does not have to be instantaneous. The image acquisition unit 11 transmits the acquired image stream to the concentration value calculation unit 13 .

The area acquisition unit 12 acquires a plurality of concentration target areas corresponding to a plurality of concentration objects to be watched by the subject 99, including the first concentration object 97 and the second concentration object 98. The concentration target area is one area set for each one concentration target object, and when the target person 99 gazes at the concentration target object, the direction of the face of the target person 99 falls within the area. It is a set area. Such a concentration target area will be described. FIG. 3 is a diagram for explaining a concentration target area according to the embodiment. In FIG. 3, the concentration target areas set for the first concentration target object 97 and the second concentration target object 98 when the target person 99 is viewed from above are indicated by dot hatching.

As shown in FIG. 3, one focused target area is set for each of the first focused target object 97 and the second focused target object 98 . Specifically, a first concentration target area 97a is set for the first concentration target object 97, and a second concentration target area 98a is set for the second concentration target object 98, respectively. The focused target area is, for example, a virtual line connecting the center of the visual field 99a of the target person 99 (that is, the position of the target person 99, more specifically, the position of the eyes of the target person 99) to one end of the focused target, It is set between the virtual line connecting to the other end. That is, the focused target area is set within a predetermined angular range in the field of view 99 a of the subject 99 .

Since the concentration target area is an area dependent on the target person 99 in this way, it is preferable to set it for each target person 99 . Therefore, for example, an operation for setting a concentration target region for each subject is performed prior to calculation of the concentration value. In the present embodiment, for example, by displaying an instruction such as "Look at the four corners of the screen in turn" on the screen of the computer, the concentration target area can be determined based on the orientation of the face of the subject 99. is determined experimentally. It should be noted that the calculation of the orientation of the face of the target person 99 is performed based on the region determination image acquired by imaging the target person 99 with the imaging device 20 when the above instruction is presented. Since this operation is the same as the calculation of the face direction of the subject 99 from the image stream performed by the arithmetic unit 10, the explanation here will be omitted by referring to the explanation of the calculation of the concentration value described later. do.

In addition, by performing the same operation on the sub-display and the like, it is possible to determine the second concentration target area 98a. Information about the determined concentration target area is stored in the storage device 30 in advance. Then, the area acquisition unit 12 acquires the necessary concentration target area by referring to the information stored in the storage device 30 . In other words, acquiring the concentration target area means reading and acquiring information indicating the concentration target area. Further, it is not essential to refer to the storage device 30, and the concentration target area determined by actual measurement may be obtained as it is and used for calculating the concentration value. Here, a first concentration target region 97a and a second concentration target region 98a are acquired as the concentration target regions.

In the above description, an instruction to gaze at four corners of the focused target is given, but depending on the shape of the focused target, five or more corners may be gazed at, or in a horizontal or vertical direction. Only one end and the other end of the focused object in a predetermined direction may be gazed at. In addition, if the focused object is of a size that fits in the effective visual field of the target person 99 (that is, the visual field area in which information can be effectively obtained that spreads from the central visual field in one direction to the surroundings), simply gaze at the center of the focused object. Only can be used. In this case, an area of about plus or minus 10 degrees from the direction of the face of the subject 99 is automatically determined as the concentration target area.

Also, the concentration target area may be determined by machine learning the direction in which the target person 99 is likely to gaze during work. In this case, the focus target area can be automatically determined by detecting the position of the center of the field of view 99a of the subject 99 without actually depending on a physical focus target such as a computer or a sub-display. .

Furthermore, the concentration target area may be determined according to the space used by the target person 99. For example, a subject 99 using a desk on which a computer with two displays is installed is naturally expected to gaze at the two displays. Therefore, if the position of the center of the field of view 99a of the subject 99 is detected, the focused target area can be determined based on the positional relationship between the imaging device 20 and the two displays.

Returning to FIG. 2, the concentration value calculation unit 13 has a function of calculating the concentration value of the subject 99 based on the image stream acquired by the image acquisition unit 11 and the concentration target area acquired by the area acquisition unit 12. Department. The calculation of the concentration value, which is the main function of the concentration value calculator 13, will be described later in detail. The concentration value of the subject 99 calculated by the concentration value calculator 13 is output and presented on, for example, a computer screen. In addition, the calculated concentration value of the target person 99 is output to and stored in a server device (not shown) or the like, and can be made available for confirmation by a manager or the like who is in a position to supervise the work of the target person 99. .

The imaging device 20 is a camera that captures images as described above. The imaging device 20 continuously acquires images to generate and output an image stream.

The storage device 30 is a device for storing information such as a semiconductor memory. The storage device 30 stores information such as the concentration target area, and receives reference to the information by the area acquisition unit 12 or the like.

The output device 40 is, for example, a display controller, converts the information into a presentation image in order to present the information of the concentration value calculated by the concentration value calculation unit 13 on the display, and outputs a signal for presenting the presentation image. output to

[Operation of centralized value calculation system]
Next, operation of the concentration value calculation system 100 will be described with reference to FIG. FIG. 4 is a flow chart showing a concentration value calculation method according to the embodiment. Note that the concentration value calculation system 100 may perform some operations not shown in FIG. 4, such as determination of a concentration target region. As shown in FIG. 4, when the concentration value calculation system 100 starts operating, the image acquisition unit 11 acquires an image stream (S101). An image stream is a group of images that are captured in sequence. Therefore, the image acquisition unit 11 acquires an image stream by sequentially acquiring a plurality of continuously captured images.

Also, the area acquisition unit 12 acquires a plurality of concentration target areas corresponding to each of the plurality of concentration objects by referring to the storage device 30 (S102). Acquisition of the concentration target area (S102) may be performed before acquisition of the image stream (S101). Thus, the order of some operations of the centralized value calculation system 100 may be changed.

Next, the concentration value calculator 13 calculates the concentration value of the subject 99 based on the acquired image stream and the acquired concentration target area (S103). Specifically, the concentration value calculation unit 13 calculates the facial orientation and the like of the subject 99 from the acquired image stream. Based on the corresponding plurality of concentration target areas, it is determined whether or not the target person 99 is in a gaze state in which one of the plurality of concentration targets is being gazed at, and the concentration of the target person 99 is determined based on the determination result. Calculate and output the value.

[Calculation of Concentration Value by Concentration Value Calculation System]
The operation of the concentration value calculator 13 described above will be described in more detail below. First, calculation of the orientation of the face of the subject 99 will be described. The subject's 99 face orientation is calculated based on the captured image stream. The concentration value calculation unit 13 inputs the obtained image stream to a machine-learned face orientation calculation model, thereby obtaining the face orientation of the subject 99 on the image stream as an output. More specifically, the face orientation calculation model outputs the face orientation of the subject 99 as a normal vector of the front side of the face. In the concentration value calculation system 100, when the direction connecting the target person 99 and the imaging device 20 imaging the target person 99 is assumed to be 0 degrees, the output normal vector of the front side of the face is The relative angle is treated as the orientation of the face of the subject 99, and more specifically, the orientation of said subject's face with respect to the imaging device capturing the image stream.

The face orientation calculation model is an example of the orientation calculation model, and outputs the face orientation for each of a plurality of images forming the image stream, so that the face orientation of the subject 99 can be changed according to the image stream. be able to. The face orientation calculation model consists of a teacher image of the subject 99 (corresponding to the image stream) and the correct face orientation data corresponding to the teacher image (corresponding to the relative angle indicating the face orientation of the subject). ) is a trained model that has been trained in advance using a dataset that is a combination of

Note that the calculation of the face orientation of the subject 99 is not limited to the example using the face orientation calculation model described above. For example, in each of the images that make up the image stream, the feature points of the face of the subject 99 (the corners of the eyes, the tip of the nose, the corners of the mouth, the chin, etc.) of the subject 99 are used to fit the subject to a three-dimensional model. The orientation of the person's face 99 may be calculated. Alternatively, the concentration value calculator 13 may calculate the orientation of the face of the subject 99 from the image stream using any existing technique.

In the present embodiment, an example using the orientation of the face of the subject 99 will be described, but instead of the orientation of the face of the subject 99, the orientation of the line of sight of the subject 99 can be used. The direction of the line of sight of the subject 99 can be calculated by image analysis centering on the eyeball of the subject 99 . In addition, the orientation of the line of sight of the target person 99 can be handled in substantially the same way as the orientation of the target person's 99 face. Therefore, an example using the above-mentioned direction of sight line will be explained by appropriately reading "direction of sight line" for "direction of face" in the description of the present disclosure.

Here, the state of the subject 99 in this embodiment will be described with reference to FIGS. 5 and 6. FIG. FIG. 5 is a diagram for explaining determination of a gaze state according to the embodiment. Moreover, FIG. 6 is a diagram for explaining determination of a transitional state according to the embodiment. 5 and 6 show the subject 99 from the same viewpoint as in FIG. In FIG. 5, the subject 99 is in a fixation state. Also, in FIG. 6, the subject 99 is in a transitional state.

The face orientation of the subject 99 described above is indicated as direction 99b by the dashed arrow (that is, the normal vector) in FIGS. As shown in FIG. 5, the concentration value calculation unit 13 determines that the target person 99 is in the gaze state if the direction 99b is within the concentration target area. In other words, the direction 99b only needs to fall within the angular range of either the first focused target area 97a or the second focused target area 98a.

In addition, since the human visual field has an effective visual field, the face orientation may have a certain angular range. That is, the face direction may be in range 99c or the like. Note that the direction 99b is the direction that bisects the range 99c (that is, the center line). When the range 99c is used, for example, it may be determined that the target person 99 is in the gaze state if the range 99c partially overlaps the concentration target region.

On the other hand, as shown in FIG. 6, if the direction 99b does not fall within the angular range of the first focused target area 97a nor does it fall within the angular range of the second focused target area 98a, conventionally, A non-gazing state, that is, a state of looking away can be determined. On the other hand, in the present embodiment, since the direction 99b is located between the first concentration target region 97a and the second concentration target region 98a, the target person 99 at this time is a plurality of concentration target objects. It is determined that there is a transition state in which the line of sight transitions between two of them. In this way, in the calculation of the concentration value in the present embodiment, by classifying whether the target person 99 is in the gazing state, the transitional state, or the non-gazing state, the To calculate a concentration value with high accuracy.

In addition to the above, the determination as to whether the subject is in the transitional state or the non-gazing state may be calculated based on the movement vector of the subject's 99 face direction over time. In this example, the transition of the face orientation of subject 99 from a region that is neither of the plurality of focus target regions, nor to a region that is none of the plurality of focus target regions, happens to be two focus target regions. When passing between areas, it is sufficient not to judge this as a transitional state. Further, the duration of a state in which the target person 99 is stationary at a fixed position may be taken into consideration. In this example, even if the direction 99b is located between the first concentration target region 97a and the second concentration target region 98a, if this state continues for a certain period of time, it is considered a non-gazing state. It should be judged.

In addition, between one region and another region is a line segment that connects an arbitrary point in one region and an arbitrary point in another region. It refers to the area to which it does not belong.

The concentration value calculation unit 13 further calculates a performance value, which is a unit concentration value of the subject 99, from the acquired image stream. The performance value is a numerical value that is the base of the concentration value calculated from the body movement, posture, facial expression, etc. of the subject 99 on the image. Any existing technique may be used to calculate the performance value. For example, in the above body movement, if an image is acquired in which body movement is greater in number and degree than in the previous image, the performance value of the subject 99 is calculated to be low. Further, for example, in the above posture, a performance value is linked in advance for each posture of the subject 99, and the linked performance value is calculated by matching the posture on the acquired image. Also, for example, in the facial expression described above, numerical values are assigned to several features appearing in the subject's facial expression, and the performance value is calculated by summing the numerical values of the features seen in the subject 99 on the image. In calculating the performance value, the state of the target person 99 (whether it is a gaze state, a transitional state, or a non-gazing state) may be taken into account.

For the performance value calculated in this way, in the present embodiment, the state of the subject 99 is further integrated to optimize the performance value and calculate the concentration value. For example, even if the performance value is a high value, the concentration value should be calculated to be low if the target person 99 is actually in a non-gazing state. Therefore, the concentration value calculation unit 13 multiplies the calculated performance value by the first coefficient when it is determined that the target person 99 is in the gaze state, and when it is determined that the target person 99 is in the transition state, multiplies the calculated performance value by a second coefficient less than or equal to the first coefficient, and if it is determined that the target person 99 is not in the gaze state and is not in the transition state, the calculated performance value is multiplied by the second coefficient A concentration value of the subject 99 is calculated by multiplying by a third coefficient that is smaller than .

However, the condition of the subject 99 and each coefficient for optimizing the performance value may contribute to the habits that the subject 99 can take when concentrating, so it is possible to conduct a preliminary test in advance. Each coefficient may be set for each subject 99 . Also, the transitional state may be treated in the same way as the gaze state. That is, the first coefficient and the second coefficient may have the same value.

Another example of concentration value calculation will be described below with reference to FIG. FIG. 7 is a diagram for explaining concentration values calculated in the embodiment. FIG. 7 shows a graph of concentration values calculated for each image forming an image stream, that is, per time. In this figure, when the performance values are the same during the period indicated by the double-headed arrow on the time axis, the target person 99 is in a transitional state (solid line graph), or in a gaze state and a non-gazing state. An example is shown in which the calculated concentration value differs depending on which one (broken line graph).

Here, the reliability of the performance value calculated when the target person 99 is in the transitional state is lower than in the case where the subject 99 is in the gaze state and the non-gazing state. A concentration value is calculated to reduce the influence of the performance value on the concentration value.

In this example, the concentration value is calculated according to the formula {concentration value at second time point=(β×performance value at first time point)+((1−β)×performance value at second time point)}.

Here, the performance value at the first point in time and the performance value at the second point in time are used to calculate the concentration value at the second point in time. The second point in time is a point in time that follows the first point in time, and includes the minimum unit period for calculating the concentration value in the concentration value calculation system 100 . Note that the minimum unit period for calculating the concentration value in the concentration value calculation system 100 is, for example, one second. Therefore, the concentration value for 1 second is calculated at the second time immediately after the concentration value for 1 second is calculated at the first time.

The first term on the right side of the above formula is the first value obtained by multiplying the performance value at the first point in time when it is determined that the subject 99 is not in the transitional state by the first weighting factor, or the first value when the subject 99 is in the transitional state. The third value obtained by multiplying the performance value at the first point in time by the third weighting factor is shown. In addition, the second term on the right side of the above equation is the second value obtained by multiplying the performance value at the second time point by the second weighting factor when it is determined that the subject 99 is not in the transition state, or A fourth value obtained by multiplying the performance value at the second time point by a fourth weighting factor when it is determined to be in the state is shown.

　Here, β in the formula is a weighting factor for determining which of the performance value at the first time point and the performance value at the second time point should be emphasized. For example, in a reliable gaze state or non-gaze state, the performance values at the first point in time need not be considered as much. That is, since the performance value at the second point in time in this case has sufficient reliability, it is appropriate to increase the weight of the performance value at the second point in time. Therefore, β should be set to a relatively small value. β should be a value greater than 0 and less than 1 in order to satisfy the above equation. If β is set to 0, the concentration value at the second point in time can be calculated from only the performance value at the second point in time without considering the performance value at the first point in time.

On the other hand, by emphasizing the performance value at the first time point in the low-reliability transitional state, it is possible to reduce the influence of the low-reliability performance value at the second time point on the concentration value. Therefore, β should be set to a relatively large value. In this way, the reliability of the performance value may be set as a numerical value (that is, a weighting factor) based on the state of the subject 99, and the previous performance value may be incorporated into the calculation of the concentration value for one minimum unit period.

As a result, as shown in FIG. 7, changes in the concentration value in the time domain in the transitional state become more gradual than in the case of the gaze state or the non-gazing state (the effect of the performance value at the second point in time becomes less ).

As described above, the concentration value of the target person 99 can be calculated with higher accuracy while considering the state of the target person 99 when there are a plurality of focused objects.

[Effects, etc.]
As described above, the concentration value calculation system 100 according to the present embodiment includes the image acquisition unit 11 that acquires an image stream in which the subject 99 is imaged, and a plurality of concentration target regions, each of which is a plurality of concentration target regions. a region acquisition unit 12 for acquiring a plurality of concentration target regions, each corresponding to each of a plurality of concentration targets to be watched by the target person 99; and a concentration value calculation unit 13 for calculating the concentration value of the target person 99. , the concentration value calculation unit 13 calculates the face direction or the line-of-sight direction of the subject 99 from the acquired image stream, and the calculated face direction or line-of-sight direction of the subject 99 and the acquired It is determined whether or not the target person 99 is gazing at any one of the plurality of focused objects based on the determined concentration target areas, and the concentration value of the target person 99 is determined based on the determination result. Calculate and output.

Such a concentration value calculation system 100 can calculate the concentration value of the subject 99 based on whether the subject 99 is gazing at any one of the plurality of concentration objects. When the target person 99 is gazing at the focused object, the concentration value is relatively high, and when the target person 99 is not gazing at the focused object, the concentration value is relatively low. can be done. That is, even when there are two or more focused objects, a high concentration value is calculated by gazing at one of the plurality of focused objects. On the other hand, if none of the plurality of concentration targets is being watched, the concentration value is calculated to be low. The concentration value calculation system 100 can thus calculate the concentration value with higher accuracy even when there are a plurality of concentration objects.

Further, for example, each of the plurality of concentration target areas is determined as an area imaged when the subject 99 is presented with an instruction to gaze at the concentration target corresponding to the concentration target area. It may be determined based on the image and stored in advance in the storage unit, and the area acquiring unit 12 may acquire a plurality of concentration target areas by referring to the storage unit.

According to this, in response to an instruction to gaze at the focused object, it is possible to acquire the focused target area determined by the direction of the face or the direction of the line of sight when the target person 99 gazes.

In addition, for example, each of the plurality of concentration target areas is set to the target person 99 who gazes at one end in response to an instruction to gaze at one end and the other end of the concentration target object corresponding to the concentration target area. It may be determined as an area between the face direction or line of sight direction and the face direction or line of sight direction of the subject 99 who gazes at the other end, and stored in advance in the storage unit.

According to this, it is possible to obtain a focused target area determined as an area between the orientation of the face or the orientation of the line of sight of the target person 99 when gazing at one end and the other end.

Further, for example, the concentration target area is set in advance for each space used by the subject 99 and stored in the storage unit, and the area acquisition unit 12 refers to the storage unit to determine the plurality of concentration target areas. may be obtained.

According to this, it is possible to acquire a concentration target area preset for each space.

Further, for example, the concentration target area is set in advance for each subject 99 and stored in the storage unit, and the area acquisition unit 12 may acquire a plurality of concentration target areas by referring to the storage unit. good.

According to this, it is possible to obtain a concentration target area preset for each target person 99 .

Also, for example, the orientation of the subject's 99 face is calculated from the acquired image stream as a normal vector of the subject's 99 face, and each of the plurality of concentrated target regions is centered on the subject's 99 position. has a predetermined angle range, and the concentration value calculation unit 13 determines that the target person 99 corresponds to the concentration target region having the predetermined angle range when the calculated normal vector is within the predetermined angle range. It may be determined that the gaze state is gazing at the focused object.

According to this, the normal vector of the target person's 99 face can be calculated from the image stream, and it can be determined whether or not the target person's 99 is in a gaze state.

Further, for example, when the concentration value calculation unit 13 determines that the target person 99 is not in the gaze state, it further determines whether the target person 99 is in a transition state in which the line of sight transitions between two of the plurality of concentration objects. may be determined, and the concentration value of the subject 99 may be calculated based on the determination result.

According to this, when the target person 99 is not in the gaze state, it is not simply determined as the non-gazing state, but it is determined whether the line of sight transitions between a certain concentration target area and another concentration target area. I can judge. A concentration value can be calculated based on this determination result.

Further, for example, the concentration value calculation unit 13 further calculates a performance value, which is a unit concentration value, of the target person 99 from the acquired image stream, and when it is determined that the target person 99 is in the gaze state, , the calculated performance value is multiplied by the first coefficient, and if it is determined that the target person 99 is in a transitional state, the calculated performance value is multiplied by a second coefficient that is equal to or less than the first coefficient, and the target person 99 is in the gaze state. If it is determined that it is not, and if it is determined that the state is not transitional, the concentration value of the subject 99 may be calculated by multiplying the calculated performance value by a third coefficient that is smaller than the second coefficient.

According to this, the concentration value is the highest value based on the performance value in the gaze state, the following value in the gaze state based on the performance value in the transition state, and the non-gazing state If there is, it is calculated based on the performance value so that it will be a smaller value than in the case of transitional state.

Further, for example, the concentration value calculation unit 13 further calculates a performance value, which is a unit concentration value, of the subject 99 from the acquired image stream at the first time point and at a second time point following the first time point, When it is determined that the target person 99 is in the gaze state, and when it is determined that the target person 99 is not in the gaze state and is not in the transition state, the calculated performance value at the first time point is and a second value obtained by multiplying the calculated performance value at the second time point by a second weighting factor that is the difference between the first weighting factor and 1. If it is determined that the subject 99 is in a transitional state, a third value obtained by multiplying the calculated performance value at the first time point by a third weighting factor different from the first weighting factor, and the calculated A concentration value of the subject 99 is calculated by adding a fourth value obtained by multiplying the performance value at the second time point by a fourth weighting factor which is a difference between the third weighting factor and 1, and The 1 weighting factor and the 3rd weighting factor may be numerical values greater than 0 and less than 1.

According to this, it is possible to calculate the concentration value at the second point in time, taking into consideration the performance values calculated at the first point in time and the second point in time. At this time, the weight (that is, the first A more accurate concentration value can be calculated by changing the degree to which the performance value at the time point affects the performance value at the second time point.

In addition, the concentration value calculation method according to the present embodiment obtains an image stream in which the subject 99 is captured, and obtains a plurality of concentration target areas, each of which is a concentration target area that the subject 99 is gazing at. A plurality of focused target areas corresponding to each of a plurality of focused targets are acquired, the direction of the face or the direction of the line of sight of the target person 99 is calculated from the acquired image stream, and the calculated face direction of the target person 99 is calculated. Based on the orientation or the direction of the line of sight and the acquired plurality of focused target areas, it is determined whether or not the target person 99 is in a gaze state in which one of the plurality of focused targets is being gazed at, and a determination result is obtained. , the concentration value of the subject 99 is calculated and output.

Such a concentration value calculation method can provide the same effects as the concentration value calculation system described above.

Also, for example, it may be a program for causing a computer to execute the concentration value calculation method described above.

According to this, it is possible to cause the computer to execute the concentration value calculation method described above.

(Other embodiments)
Although the concentration value calculation system, the concentration value calculation method, and the program according to the present disclosure have been described above based on the above embodiments and the like, the present disclosure is not limited to the above embodiments. For example, a form obtained by applying various modifications that a person skilled in the art can think of for each embodiment, etc., or a form obtained by arbitrarily combining the constituent elements and functions of each embodiment within the scope of the present disclosure. Also included in the present disclosure is the form of

For example, the centralized value calculation system may be realized only by the arithmetic device by providing only the arithmetic device described above and connecting the arithmetic device to an external imaging device, an external storage device, and an external output device. Thus, the imaging device, storage device, and output device are not essential components.

Also, for example, as shown in FIG. 8, each of the plurality of focused objects may not be a physical object. In the figure, the first application window 97b displayed on the display 96 of the computer may be the first focus object, and the second application window 98b may be the second focus object, and the subject matter of the present disclosure may be applied. With the recent increase in size of display devices, it was not possible to calculate the concentration value of a subject with high accuracy using conventional techniques, but this can be made possible by applying the contents of the present disclosure. .

Also, as shown in FIG. 9, a learning model learned by machine learning can be used in the concentration value calculation unit in the above embodiment. FIG. 9 is a block diagram showing a functional configuration of a concentration value calculation unit according to another example. In this example, the computing device 10 includes a concentration value calculator 13a in place of the concentration value calculator 13 in the embodiment. The concentration value calculation unit 13a can directly output the concentration value of the subject 99 by inputting the obtained image stream and the obtained concentration target region to the concentration value calculation model 13b. Then, as the concentration value of the subject 99, the output result output from the concentration value calculation model 13b is output as it is.

The concentration value calculation model 13b is a learning model in which the correlation between the image stream, the concentration target region, and the concentration value is learned in advance by machine learning. In this example, the concentration value calculation system 100 further includes a model generation unit 13c for generating (learning) the concentration value calculation model 13b. In order to generate the concentration value calculation model 13b, the model generation unit 13c generates input data corresponding to the two pieces of information of the image stream and the concentration target area, and correct (or correct and incorrect) output data for the input data. and are used as training data.

In this example, as input data, an image stream and a teacher image/teacher region D1 corresponding to the concentration target region are input for learning. In this example, the teacher concentration value D2 of the subject 99 is input as the output data for learning. Then, the concentration value calculation model 13b is adjusted using a data set combining the teacher image/teacher region D1 and the teacher concentration value D2. For example, in the case of the centralized value calculation model 13b composed of a neural network consisting of a plurality of layers, the weighting coefficient assigned to each neuron is adjusted by a method such as back propagation to obtain an appropriate value for the input data. Machine learning is performed to obtain output data.

Then, the concentration value calculation unit 13a inputs the acquired image stream and the acquired concentration target area to the learned concentration value calculation model 13b, so that an appropriate concentration value of the subject 99 is output. In this way, the calculation of the concentration value by the concentration value calculation unit 13a can also be realized using a machine-learned learning model.

In this example, the concentration value calculation system 100 including the model generating unit 13c has been described. After that, it is also possible to realize a configuration in which only the recorded concentration value calculation model 13b is used without going through the learning process. is also possible.

In addition, for example, the present disclosure can be realized not only as a centralized value calculation system, but also as a program including, as steps, processes performed by each component of the centralized value calculation system, and a computer-readable recording medium recording the program. You can also The program may be pre-recorded on a recording medium, or may be supplied to the recording medium via a wide area network including the Internet.

That is, the general or specific aspects described above may be embodied in a system, device, integrated circuit, computer program or computer readable recording medium, and any of the system, device, integrated circuit, computer program and recording medium may be implemented. may be implemented in any combination.

REFERENCE SIGNS LIST 11 image acquisition unit 12 area acquisition unit 13 concentration value calculation unit 99 subject 100 concentration value calculation system

Claims

an image acquisition unit that acquires an image stream in which a subject is imaged;
a region acquisition unit configured to acquire a plurality of concentration target regions, each of which corresponds to each of a plurality of concentration targets to be gazed at by the subject;
a concentration value calculation unit that calculates the concentration value of the subject,
The concentration value calculation unit
calculating the orientation of the face or the orientation of the line of sight of the subject from the acquired image stream;
Based on the calculated face direction or line-of-sight direction of the subject and the plurality of acquired focus target areas, whether the subject is in a gaze state gazing at any one of the plurality of focused objects determine whether or not
A concentration value calculation system for calculating and outputting a concentration value of the subject based on a determination result.
Each of the plurality of concentration target areas is based on an area determination image captured when the subject is presented with an instruction to gaze at a concentration target corresponding to the concentration target area. and is stored in advance in the storage unit,
The concentration value calculation system according to claim 1, wherein the area acquisition unit acquires the plurality of concentration target areas by referring to the storage unit.
Each of the plurality of concentration target areas is configured such that, in response to an instruction to cause the subject to gaze at one end and the other end of the concentration target corresponding to the concentration target area, the face of the subject gazes at the one end. 3. The concentration value according to claim 2, which is determined as an area between the orientation or the direction of the line of sight and the orientation of the face or the direction of the line of sight of the subject gazing at the other end, and stored in advance in a storage unit. calculation system.
The concentration target area is set in advance for each space used by the target person and stored in a storage unit,
The concentration value calculation system according to claim 1, wherein the area acquisition unit acquires the plurality of concentration target areas by referring to the storage unit.
The concentration target area is set in advance for each target person and stored in a storage unit,
The concentration value calculation system according to claim 1, wherein the area acquisition unit acquires the plurality of concentration target areas by referring to the storage unit.
the orientation of the subject's face is calculated from the acquired image stream as a normal vector of the subject's face;
each of the plurality of focused target areas has a predetermined angular range centered on the subject's position;
The concentration value calculation unit, when the calculated normal vector is within the predetermined angle range, causes the subject to gaze at the concentration target object corresponding to the concentration target region having the predetermined angle range. 6. The concentration value calculation system according to any one of claims 1 to 5, wherein the gaze state is determined to be one of the
The concentration value calculation unit further determines whether or not the target person is in a transition state in which the line of sight transitions between two of the plurality of concentration objects when determining that the gaze state is not the state. and calculating the concentration value of the subject based on the determination result.
The concentration value calculation unit
Further, calculating a performance value, which is a unit concentration value of the subject, from the acquired image stream,
When the target person is determined to be in the gaze state, the calculated performance value is multiplied by a first coefficient, and when the target person is determined to be in the transition state, the calculated performance value is multiplied by a second coefficient equal to or less than the first coefficient, and when it is determined that the target person is not in the gaze state and is not in the transition state, the calculated performance value is added to the second coefficient 8. The concentration value calculation system according to claim 7, wherein the concentration value of the subject is calculated by multiplying by a third coefficient smaller than .
The concentration value calculation unit
Furthermore, from the acquired image stream, a performance value, which is a unit concentration value of the subject, is calculated at a first time point and a second time point following the first time point,
When it is determined that the target person is in the gaze state, and when it is determined that the target person is not in the gaze state and is not in the transitional state, A first value obtained by multiplying the performance value by a first weighting factor, and a second weighting factor, which is a difference between the first weighting factor and 1, to the calculated performance value at the second point in time. Add the multiplied second value,
When the subject is determined to be in the transition state, a third value obtained by multiplying the calculated performance value at the first time point by a third weighting factor different from the first weighting factor, and the calculated adding a fourth value obtained by multiplying the performance value at the second time point by a fourth weighting factor which is a difference between the third weighting factor and 1 to calculate the concentration value of the subject; ,
8. The concentration value calculation system according to claim 7, wherein the first weighting factor and the third weighting factor are numerical values greater than 0 and less than 1.
The direction of the subject's face or the direction of the line of sight is the direction in which the correlation between the image stream and the direction of the subject's face or the direction of the line of sight with respect to an imaging device that captures the image stream is learned by machine learning. The concentration value calculation system according to any one of claims 1 to 9, which is calculated as an output result by inputting the image stream to a calculation model.
obtaining an image stream in which the subject is imaged;
obtaining a plurality of focused regions of interest, each of the plurality of focused regions of interest corresponding to each of a plurality of focused objects to be gazed at by the subject;
calculating the orientation of the face or the orientation of the line of sight of the subject from the acquired image stream;
a gaze state in which the subject is gazing at any one of the plurality of focused objects based on the calculated face orientation or line-of-sight orientation of the subject and the plurality of acquired focused target areas; determine whether there is
A concentration value calculation method for calculating and outputting a concentration value of the subject based on a determination result.
A program for causing a computer to execute the concentration value calculation method according to claim 11 .
an image acquisition unit that acquires an image stream in which a subject is imaged;
a region acquisition unit configured to acquire a plurality of concentration target regions, each of which corresponds to each of a plurality of concentration targets to be gazed at by the subject;
a concentration value calculation unit that calculates the concentration value of the subject,
The concentration value calculation unit is configured to apply the image stream and the plurality of concentration target regions to a concentration value calculation model in which a correlation between the image stream and the plurality of concentration target regions and the concentration value of the subject has been learned by machine learning. A concentration value calculation system for calculating a concentration value of the subject by inputting a concentration target region.
A model generation unit that generates the concentration value calculation model according to claim 13,
Concentrated value calculation model generation system.