JP6055535B1 - Concentration processing system - Google Patents

Abstract

Provided is a concentration processing system having a mechanism for facilitating use of concentration. Software constituting a concentration processing system has functions divided into an engine unit, a tool unit, and an interface unit. The engine unit 501 generates basic data such as vector data from the image data stream output from the imaging device 411. The tool unit 503 calculates useful data for the customer system 505 such as the degree of concentration based on the data output from the engine unit 501. In addition to the degree of concentration, the tool unit 503 outputs various information of the content 102, information for specifying the user 101, and the like. The interface unit 502 manages the exchange of data such as passing data output from the engine unit 501 to the tool unit 503 and passing data output from the tool unit 503 to the customer system 505. [Selection] Figure 5

Description

  The present invention relates to a concentration processing system that detects and uses information related to the concentration level indicated by a user for content provided to the user by a computer, an electronic device, or the like.

Conventionally, “household audience rating” is used as an index indicating how much video content broadcast in television broadcasting (hereinafter “TV broadcasting”) is viewed by viewers. The household audience rating in TV broadcasting is measured by installing a device for measuring the audience rating in a sample home, and the device is a channel that is displayed when a television receiver (hereinafter “TV”) is on. Information about the information is sent to the aggregation base in near real time. In other words, the household audience rating is a result of totaling information on viewing time and viewing channel, and it is not known in what state the viewer has viewed the program (video content) from the information of household audience rating.
For example, when the viewer is watching the TV program without listening to the screen and listening like a radio, the program is not watched in a concentrated state for the viewer. In such a viewing mode, the advertising effect of a commercial (hereinafter referred to as “CM”) sandwiched between TV programs cannot be expected so much.

Several techniques for knowing how concentrated a viewer is watching a TV program have been studied.
Japanese Patent Laid-Open No. 2004-151867 discloses a technical content that defines how much a viewer is concentrated on a TV program as a “concentration”, learns the concentration, and uses it.
Patent Document 2 discloses a technical content for evaluating a user's gaze state (equal to the degree of concentration) using two image sensors.
Patent Document 3 discloses a technical content for evaluating a concentration degree of a test taker in a test using a terminal device, not a TV program, using one image sensor.

JP 2003-111106 A Japanese Patent Laying-Open No. 2015-82247 US Patent Application Publication No. 2015/0044649

  As shown in the above-mentioned patent document 3, the concentration of viewers is not necessarily limited to TV programs. All content is targeted. Here, on-line or off-line, the target person can understand character strings, audio, still images, video (moving images) provided through computers and electronic devices, and presentations and games that combine these. Information enjoyed by contents is generically referred to as contents. Further, hereinafter, in this specification, persons who enjoy and / or use content are collectively referred to as users, not viewers.

  Until now, several techniques for measuring the degree of concentration of a user with respect to content have been proposed, for example, as described in Patent Documents 1, 2, and 3 described above, but how to use the degree of concentration measured by the user. There wasn't much discussion about that.

  The present invention has been made in view of such problems, and an object thereof is to provide a concentration processing system having a mechanism for facilitating use of the concentration.

In order to solve the above-described problem, the concentration processing system of the present invention includes a display unit that displays content, an imaging device that is installed in a direction in which a face of a user who views the display unit can be photographed , and an engine core unit. The engine unit has a software function other than the engine core unit, and includes an engine unit that outputs vector data indicating the direction of the user's face and the direction of the user's line of sight from the image data stream output from the imaging apparatus.
In addition, the first tool unit having a concentration level calculation function for calculating the concentration level of the content displayed on the display unit by the user from the vector data, and an input for outputting the vector data and / or the concentration level to a predetermined information processing apparatus. An output control unit .
The engine core unit detects whether the user's face exists, outputs coordinate information of the user's face extraction point from the three-dimensional image of the user's face, and performs software functions other than the engine core unit. The engine unit has vector data indicating the orientation of the user's face and the direction of the user's line of sight, and the amount of change that is fluctuation on the time axis of the vector data. Whether or not the displayed content is being watched is output as data having a predetermined range .

The present invention can provide a concentration processing system having a mechanism for facilitating the use of concentration.
Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is the schematic explaining the mechanism of the user's concentration degree in the concentration degree processing system which concerns on this embodiment. It is a figure which shows the kind of display, and the variation of a camera. It is a figure which shows the arrangement | positioning relationship in which the kind of display and a camera are mounted | worn. It is a block diagram which shows the hardware constitutions of a concentration degree processing system. It is a block diagram which shows the software function of the concentration processing system which concerns on 1st embodiment of this invention. It is a block diagram which shows the software function in the implementation example of the concentration processing system which concerns on 2nd embodiment of this invention. It is a block diagram which shows the software function in the implementation example of the concentration processing system which concerns on 3rd embodiment of this invention. It is a block diagram which shows the software function in the example of mounting of the concentration processing system which concerns on 4th embodiment of this invention. It is a block diagram which shows the software function in the implementation example of the concentration degree processing system which concerns on 5th embodiment of this invention, and a figure which shows an example of the field structure of the table which comprises the database of a customer system. It is a flowchart which shows the flow of a process which estimates a user's attribute information in the concentration processing system which concerns on 5th embodiment of this invention. It is a block diagram which shows the software function in the implementation example of the concentration processing system which concerns on 6th embodiment of this invention. It is the graph which shows the fluctuation | variation of the concentration degree with the passage of time of the user who reproduces a moving image with a moving image reproducing apparatus, and the time chart which shows the process in which a moving image reproducing apparatus pauses reproduction | regeneration of a moving image and reproduces again. It is a block diagram which shows the software function in the example of mounting of the concentration processing system which concerns on 7th embodiment of this invention.

The concentration processing system according to the present embodiment uses the concentration measurement technique disclosed in Patent Document 3. In addition, the concentration processing system provides a framework that facilitates the development and use of software with respect to the concentration that is assumed to be in various usage forms.
Note that the concentration processing system in the present embodiment targets content that uses vision. Therefore, the content of only audio is not subject to the measurement and use of the concentration level in the concentration level processing system of this embodiment.

[About concentration]
1A and 1B are schematic diagrams for explaining a mechanism of user concentration in the concentration processing system according to the present embodiment.
In FIG. 1A, the user 101 is watching the display 103 displaying the content 102. A camera 104 is mounted immediately above the display 103. The camera 104 is directed in a direction in which the face of the user 101 existing in front of the display 103 can be photographed. An information processing apparatus (not shown) (described later in FIG. 4 and later) is connected to the camera 104. The information processing apparatus detects from the image data obtained from the camera 104 whether or not the face direction and / or line of sight of the user 101 is in the direction of the display 103, and whether or not the user 101 is gazing at the content 102. Are output as data having a value having a predetermined range such as 0 to 1, 0 to 255, or 0 to 1023. This value is the degree of concentration.

In FIG. 1B, the user 101 is not gazing at the display 103 displaying the content 102. The information processing apparatus connected to the camera 104 outputs a concentration level lower than the concentration level in FIG. 1A from the image data obtained from the camera 104.
As described above, the concentration processing system according to the present embodiment determines whether or not the face direction and / or line of sight of the user 101 is toward the display 103 displaying the content 102, and the image data obtained from the camera 104. Calculate from

2A, 2B, and 2C are diagrams showing the types of the display 103 and variations of the camera 104. FIG.
3A and 3B are diagrams illustrating the types of the display 103 and the arrangement relationship in which the camera 104 is mounted.
FIG. 2A is an example in which an external USB type web camera 202 is mounted on a stationary LCD display 201.
FIG. 2B shows an example in which a web camera 205 is embedded in the frame of the LCD display 204 of the notebook computer 203.
FIG. 2C is an example in which a self-portrait in-camera 208 is embedded in the frame of the LCD display 207 of the portable wireless terminal 206 such as a smartphone.
2A, 2B, and 2C are that the camera 104 is provided in the vicinity of the center line of the display 103. FIG.

FIG. 3A is a diagram illustrating a region of an optimal arrangement position of the camera 104 in the horizontal display 103a corresponding to FIGS. 2A and 2B.
FIG. 3B is a diagram illustrating an area of an optimal arrangement position of the camera 104 in the vertical display 103a corresponding to FIG. 2C.
3A and the display 103b of FIG. 3B, that is, whether the display is a horizontal type or a vertical type, regions 301a, 301b, and 303a passing through the center lines L302 and L304 on the upper and lower sides of the displays 103a and 103b. And 303b are positions where the camera 104 can correctly capture the face and line of sight of the user 101 without adjustment. If the camera 104 is arranged in the vicinity of the center lines L302 and L304, even if the face of the user 101 exists at a position shifted from the front in front of the displays 103a and 103b, the face and line of sight of the user 101 from the displays 103a and 103b. The angle can be correctly grasped without adjustment.

  When the camera 104 is installed at a position outside these areas, the user's 101 face and line of sight are displayed in advance on the display 103 in order to detect whether the face and line of sight of the user 101 are correctly facing the display 103. It is preferable to detect information on the face and line-of-sight direction of the user 101 viewed from the camera 104 when facing the camera and store it in the nonvolatile storage 406 (see FIG. 4) or the like.

[Centralized processing system: hardware configuration]
FIG. 4 is a block diagram illustrating a hardware configuration of the concentration processing system 401.
The client 402 is a general computer, and a CPU 403, a ROM 404, a RAM 405, a nonvolatile storage 406, a real-time clock (hereinafter “RTC”) 407 that outputs current date and time information, and an operation unit 408 are connected to the bus 409. A display unit 410 (display 103) and an imaging device 411 (camera 104) having an important role in the concentration processing system 401 are also connected to the bus 409.
The client 402 communicates with the server 414 via the Internet 413 through a NIC (Network Interface Card) 412 connected to the bus 409.
The server 414 is also a general computer, and a CPU 415, ROM 416, RAM 417, nonvolatile storage 418, and NIC 419 are connected to the bus 420.

  Although FIG. 4 illustrates a general client / server system, the concentration processing system 401 does not necessarily require the Internet 413 (network) or the server 414. In the embodiments described later, there may be a stand-alone device without the server 414 or the network.

[First Embodiment: Overview of Software Functions]
FIG. 5 is a block diagram showing software functions of the concentration processing system 401 according to the first embodiment of the present invention. The block diagram of FIG. 5 describes software functions assumed in the concentration level processing system 401 in a high-level concept.
The concentration processing system 401 includes an imaging device 411, an engine unit 501, an interface unit 502, a tool unit 503, a content reproduction unit 504, a customer system 505, and a customer tool 506.

  Image data output from the imaging device 411 is supplied to the engine unit 501. The engine unit 501 calculates the orientation of the face of the user 101, the orientation of the line of sight of the user 101, and their movements from the image data output from the imaging device 411. First, the engine unit 501 recognizes the image data output from the imaging device 411 as a still image, undergoes a known three-dimensional analysis process such as polygon processing, recognizes the presence of the face of the user 101, and determines the face direction and line of sight. Is calculated as vector data. Next, the still image continues on the time axis to be captured as moving image data, and how the vector data of the face direction and the line-of-sight direction fluctuates on the time axis is calculated. After such arithmetic processing, the engine unit 501 finally has a vector indicating the orientation of the face of the user 101 and a vector indicating the orientation of the line of sight of the user 101 in a three-dimensional space such as an orthogonal coordinate system or a three-dimensional polar coordinate system. , And fluctuations on the time axis of these vectors, that is, fluctuation amounts are output.

An engine core unit 507 is provided inside the engine unit 501. The engine core unit 507 has a minimum necessary function as a basis for calculating vector data output from the engine unit 501. Specifically, for example, there are a function for detecting whether or not the face of the user 101 exists in the image data and a function for generating vector data by performing polygon processing on the face of the user 101.
If a low-resource device such as a one-chip microcomputer that has limited computing power and storage capacity is used as the client 402, it may be difficult to implement the engine unit 501 as software for the client 402. . In such a situation, if image data output from the imaging device 411 is continuously transmitted as a video data stream to the customer system 505 via a network such as the Internet 413, a great load is applied to the network. Therefore, as a countermeasure for such a case, in order to reduce the amount of communication data, a minimum necessary portion of the engine unit 501 is configured as an engine core unit 507.

Data output from the engine unit 501 or the engine core unit 507 is supplied to the interface unit 502. The interface unit 502, which can be referred to as an input / output control unit, inputs / outputs data to / from the tool unit 503 when the tool unit 503 exists, and also inputs / outputs data to / from the customer system 505.
The tool unit 503 supplies data other than the data output from the engine unit 501 or the engine core unit 507 to the interface unit 502. Various forms of the tool unit 503 are assumed depending on the mounting form of the client 402. There may be a client 402 that does not include the tool unit 503, and there may also be a client 402 that includes a plurality of tool units 503. In FIG. 5, three types of tool parts 503 are illustrated as typical examples.

The first tool unit 508 is a function (software component or library module) that calculates a concentration level that is assumed to be implemented in many clients 402. The first tool unit 508 is based on information such as the size of the display unit 410 (display 103) and the installation position of the imaging device 411 (camera 104), which is stored in the ROM 404 or the nonvolatile storage 406 in advance. It is calculated how much the angle between the vector indicating the direction of the face 101 and the vector indicating the direction of the line of sight of the user 101 is deviated from the vector connecting the user 101 and the display unit 410. This angle is the data that is the basis for calculating the degree of concentration. Then, the variation amount of the vector is compared with a predetermined threshold value, and addition / subtraction is performed on the basic data of the concentration level to finally calculate the concentration level of a value having a predetermined range such as 0 to 1.
The engine unit 501, the engine core unit 507, the interface unit 502, and the first tool unit 508 are standard components that are created assuming that they are commonly implemented in many clients 402.

The second tool unit 509 supplies information on the content 102 to the interface unit 502. For example, when the content playback processing unit 512 is playing back the content 102 held in the network storage 510 or the local storage 511, the playback time or playback scene of the content 102 currently played back by the content playback processing unit 512, etc. The information is supplied to the interface unit 502 together with content information that uniquely identifies the content 102.
The third tool unit 513 supplies information other than the content 102 to the interface unit 502. For example, current date and time information output by the RTC 407, user information (hereinafter referred to as “UID”) that is uniquely stored in the nonvolatile storage 406 and that uniquely identifies the user 101 using the client 402, and the like.
The second tool unit 509 and the third tool unit 513 are optional components that can be added or deleted according to the required specifications of the client 402 and the implementation form.
Note that the customization component prepared to meet the customer's required specifications is mounted outside the interface unit 502.

  For example, when the content 102 is a video held in the local storage 511 such as an optical disc, the second tool unit 509 first sends the content information of the content 102 that the content playback processing unit 512 intends to play back to the interface unit 502. After the output, the content playback processing unit 512 outputs the playback time of the video currently being played back to the interface unit 502. Then, the third tool unit 513 outputs user information that is stored in the nonvolatile storage 406 and uniquely identifies the user 101 who uses the client 402 to the interface unit 502. Then, the interface unit 502 combines the degree of concentration obtained from the first tool unit 508, the content information obtained from the second tool unit 509, the playback time of the content 102, and the user information obtained from the third tool unit 513. The information can be output to the customer system 505 as real-time information on the concentration level of the specific user 101.

The interface unit 502 outputs the degree of concentration or information according to this to the customer system 505. The customer system 505 receives a degree of concentration or information according to the degree of concentration from the interface unit 502 and executes a predetermined process. At this time, the customer system 505 uses the customer tool 506 as a part of the predetermined processing to process the degree of concentration or information equivalent thereto.
Note that the customer system 505 and the customer tool 506 in FIG. 5 may be implemented as the server 414 in FIG. 4, and the server 414 and the network may be eliminated and included in the client 402, that is, a stand-alone configuration may be used.

[Second Embodiment: Implementation in Low Resource Device]
FIG. 6 is a block diagram showing software functions in an implementation example of the concentration processing system 601 according to the second embodiment of the present invention.
The concentration processing system 601 illustrated in FIG. 6 assumes a case where a low resource device is used as the client 402. For example, when the client 402 is a portable game machine, most of the computer resources such as the computing capacity of the CPU 403 and the capacity of the RAM 405 are consumed for executing the game software, and there is no room to run sufficient computer resources for the concentration processing. Can be considered. In such a situation, it is preferable that the client 402 is equipped with the minimum necessary processing for calculating the degree of concentration. Specifically, only the engine core unit 507 and the interface unit 502 are mounted on the client 402. The engine unit 501, the first tool unit 508, and the like are mounted on a server 414 that is a customer system 505.

[Third embodiment: Variation of engine unit 501 and tool unit 503]
7A and 7B are block diagrams showing software functions in an implementation example of the concentration processing system 601 according to the third embodiment of the present invention.
Unlike the case of FIG. 6, the concentration level processing system 701 shown in FIG. 7A and the concentration level processing system 702 shown in FIG. 7B are assumed to implement software functions in a general personal computer or the like.
In the concentration processing system 701 in FIG. 7A, the first engine unit 703 includes an engine core unit 507. The engine core unit 507 calculates a three-dimensional image of the face of the user 101 from the image data received from the imaging device 411, and outputs coordinate information of face extraction points (points that specify eyes, nose, mouth, etc.). To do. The software functions of the first engine unit 703 other than the engine core unit 507 are based on the coordinate information of the extraction points output by the engine core unit 507, and the vectors of the face direction and line-of-sight direction of the user 101 and the amount of change thereof. Calculate and output to the interface unit 502.

Instead of the first engine unit 703, a second engine unit 704 is used as a second engine unit 704 to capture a vein that exists directly under the face skin of the user 101 from image data obtained from the imaging device 411, and the vein expands and contracts. It is possible to exchange for the function of extracting as data. Similarly to the first engine unit 703, the second engine unit 704 also includes an engine core unit 507. The software function of the second engine unit 704 other than the engine core unit 507 is based on the coordinate information of a part of the extracted points (the local part of the face of the user 101) output by the engine core unit 507, and the image data of the local part of the face. Is output to the interface unit 502. The image data of the local part of the face includes a vein that exists directly under the user's 101 face skin. While changing the software function other than the engine core unit 507 from the first engine unit 703 to the second engine unit 704, the software function for calculating the face and line-of-sight vector and the amount of change thereof is maintained while the engine core unit 507 remains common. It can be changed to a software function that outputs local image data of the face.
When the engine unit 501 is replaced from the first engine unit 703 to the second engine unit 704, the tool unit 503 is also used as the fourth tool unit 705 from the concentration degree calculation process that is the first tool unit 508. Therefore, it is necessary to replace the process for calculating the pulse from the above.

  In the concentration processing system 702 of FIG. 7B, the engine unit 501 includes a vector indicating the orientation of the face of the user 101, a vector indicating the orientation of the line of sight of the user 101 in a three-dimensional space such as an orthogonal coordinate system or a three-dimensional polar coordinate system, Fluctuations on the time axis of these vectors, that is, fluctuation amounts are output. The engine unit 501 includes an engine core unit 507. The engine core unit 507 detects whether the face of the user 101 exists in the image data, and performs polygon processing on the face of the user 101. The 3D data is generated. Then, the first tool unit 508 performs a concentration degree calculation process based on the data of the engine unit 501. Instead of the first tool unit 508, a fifth tool unit 706 can be replaced with a function for analyzing the facial expression of the user 101 from the 3D data of the user 101's face obtained from the engine core unit 507.

[Fourth embodiment: Send vector data to network instead of concentration]
FIG. 8 is a block diagram showing software functions in an implementation example of the concentration processing system 801 according to the fourth embodiment of the present invention.
The concentration processing system 801 shown in FIG. 8 assumes a case where a low resource apparatus similar to that in FIG. As in the case of FIG. 6, when the concentration processing is installed in the low resource device, most of the computer resources of the low resource device are consumed for content reproduction, and there is no room for using sufficient computer resources for the concentration processing. There are cases. In such a situation, it is preferable that the client 402 is equipped with the minimum necessary processing for calculating the degree of concentration. 8 differs from the concentration processing system 801 in FIG. 6 in that an engine unit 501, an interface unit 502, a second tool unit 509, and a third tool unit 513 are mounted on the client 402. The first tool unit 508 for calculating the degree of concentration is installed in the server 414 that is the customer system 505. In this case, a vector indicating the orientation of the face of the user 101, a vector indicating the direction of the line of sight of the user 101 in a three-dimensional space such as an orthogonal coordinate system or a three-dimensional polar coordinate system output from the engine unit 501, and The fluctuation on the time axis, that is, the fluctuation amount is output to the customer system 505 through the network.

In the database 802 of the customer system 505, the current date and time information obtained from the RTC 407 output from the third tool unit 513, the user information held in the nonvolatile storage 406, and the content identification output from the second tool unit 509 Information, current playback time of the content 102 or scene information, and the like, and the engine unit 501 outputs a vector of the face direction of the user 101, a vector of the line of sight of the user 101, a variation amount of these vectors on the time axis, and a display unit 410 And information such as the angle difference between the user 101 and the face of the user 101 is accumulated.
The first tool unit 508 that performs the concentration degree calculation process operates when various data analysis operations are executed in the customer system 505.
The concentration processing system 801 shown in FIG. 8 does not store the concentration as the calculation result of the first tool unit 508 in the database 802 of the customer system 505, but the basis for calculating the concentration that the engine unit 501 outputs. By storing the data, the first tool unit 508 can be immediately upgraded.

[Fifth embodiment: categorization of user 101]
FIG. 9A is a block diagram showing software functions in an implementation example of the concentration processing system 901 according to the fifth embodiment of the present invention.
A concentration processing system 901 shown in FIG. 9 is assumed to be a system that uses a general personal computer or the like as the client 402 and has a server 414.
The database 802 of the customer system 505 stores, in the interface unit 502, the degree of concentration output from the first tool unit 508, the content identification information output from the second tool unit 509, the current playback time or scene information of the content 102, and the like. Information such as current date and time information obtained from the RTC 407 and user information held in the nonvolatile storage 406, which is output from the third tool unit 513, is accumulated.
The customer system 505 includes a cluster analysis processing unit 902 as a kind of customer tool 506. The cluster analysis processing unit 902 calculates the similarity for the change in the degree of concentration for each user 101 stored in the database 802, and differentiates the user 101 into several clusters based on the similarity.

FIG. 9B is a diagram illustrating an example of a field configuration of a table configuring the database 802 of the customer system 505.
The database 802 includes a user master 903, a concentration level table 904, and a cluster table 905.
The user master 903 has a UID field, a sex field, an age field, and other attribute information fields.
The UID field stores a user ID (UID) that uniquely identifies the user 101.
The gender field stores the gender of the user 101, and the age field stores the age of the user 101.
In the other attribute information field, information related to the user 101 other than the gender and age, such as the living environment of the user 101, unmarried marriage, family composition, employment and schooling, is stored.

Concentration table 904 includes UID field, the content ID field, content playback date and time field, content playback time or the content chapters ID field, the degree of concentration fields.
The UID field is the same as the field with the same name in the user master 903.
In the content ID field, a content ID for uniquely identifying the content 102 is stored.
The content playback time field, date and time when the user 101 reproduces the content 102 is stored. In most cases, content playback time is the current date and time information RTC407 which incorporates the client 402 outputs.
The content playback time or content chapter ID field stores the playback time of the content 102 or information indicating the scene or chapter of the content 102, that is, the playback position information of the content 102.
The concentration degree field stores the concentration degree of the user 101 in the reproduction position information of the content 102 stored in the content reproduction time or the content chapter ID field.

The cluster table 905 includes a UID field, a content ID field, a content 102 reproduction date / time field, a concentration space coordinate field, a cluster ID field, an estimated gender field, an estimated age field, other estimated attribute information fields, and an estimated matching rate field.
The UID field, the content ID field, and the content 102 playback date / time field are the same as the field with the same name in the concentration degree table 904, respectively.
The coordinate field in the concentration degree space stores the reproduction point information of the content 102 and the position information of the user 101 in the virtual multidimensional coordinate space indicating the concentration degree of the user 101.
The cluster ID field stores a cluster ID for uniquely identifying an aggregate of users 101 having similar concentration tendency variation, which is formed in the virtual multidimensional coordinate space by the cluster analysis processing unit 902. The

In the estimated gender field, the estimated gender of the user 101 estimated as a result of the analysis by the cluster analysis processing unit 902 is stored.
In the estimated age field, the estimated age of the user 101 estimated as a result of the analysis by the cluster analysis processing unit 902 is stored.
In the other estimated attribute information field, the user 101 other than the gender and age, such as the estimated living environment of the user 101, the estimated unmarried marriage, the estimated family composition, and the estimated employment attendance estimated as the result of the analysis by the cluster analysis processing unit 902 Stored estimation information is stored.
In the estimated precision ratio field, a precision ratio indicating how much the user 101 matches with the past estimation result is stored. This estimated relevance rate is also the probability of the estimation result by the cluster analysis processing unit 902.

FIG. 10 is a flowchart showing a flow of processing for estimating attribute information of the user 101 in the concentration processing system 901 according to the fifth embodiment of the present invention. The processing in this flowchart is executed when the concentration level of each user 101 in the predetermined content 102 is stored in the database 802.
When the processing is started (S1001), the cluster analysis processing unit 902 refers to the concentration level table 904 and the user master 903, and the user 101 whose gender / age and other attribute information is known in the virtual concentration level space. The coordinates are calculated (S1002).
Next, the cluster analysis processing unit 902 calculates the distance between the users 101 in the concentration level space, and assigns a cluster ID for categorizing the users 101 whose distances are close to each other (S1003). For this cluster analysis, a known algorithm such as the Ward method or the K-means ++ method is used.
Next, the cluster analysis processing unit 902 obtains an average position in the cluster, that is, the center of gravity for each cluster to which the cluster ID is assigned (S1004).
Next, the cluster analysis processing unit 902 calculates the appearance frequency and variance necessary for statistical processing for the attribute information such as the sex and age of the user 101 belonging to the cluster to which the cluster ID is assigned (S1005).
At step S1005, clustering is completed for the user 101 whose gender / age and other attribute information is known.

Next, the cluster analysis processing unit 902 refers to the concentration level table 904 and the user master 903, and calculates the coordinates in the concentration level space for the user 101 whose gender / age and other attribute information are unknown (S1006).
Next, the cluster analysis processing unit 902 calculates the distance between the coordinate information calculated in step S1006 and the center of gravity of each cluster ID, and determines the cluster ID of the cluster with the shortest distance to the center of gravity, that is, the cluster with the closest center of gravity. This is given to the user 101 (S1007).
Finally, the cluster analysis processing unit 902 refers to the variance calculated in step S1006, and determines the gender / age of the user 101 based on the distance from the cluster to which the user 101 whose gender / age and other attribute information is unknown belongs. Other attribute information is estimated (S1008), and a series of processing ends (S1009).

By collecting the concentration level of the user 101 in the server 414 together with the gender / age of the user 101 and other attribute information, the tendency of the concentration level can be clustered with the attribute information of the user 101. Then, conversely, by calculating the degree of similarity by comparing the concentration data of the user 101 whose gender / age and other attribute information are unknown with a known clustered concentration degree tendency, the user 101 It is possible to analogize which cluster belongs to. The age and the like are estimated by referring to the distance between the center of gravity of the cluster and the user 101 in consideration of the variance in the cluster.
Further, by repeatedly estimating the attribute information of the user 101 for each content 102, it is possible to improve the accuracy of the attribute information estimation.

[Sixth Embodiment: Detection of Decreasing Concentration and Suspending Content 102]
FIG. 11 is a block diagram showing software functions in an implementation example of the concentration processing system 1101 according to the sixth embodiment of the present invention.
The concentration processing system 1101 according to FIG. 11 is assumed to be a moving image playback device such as a stand-alone optical disc device as an example.
In FIG. 11, the content reproduction processing unit 512 reproduces the content 102 such as a moving image recorded in the local storage 511 such as an optical disk and displays it on the display unit 410. At that time, the reproduction time information of the content 102 is output to the second tool unit 509.
The interface unit 502 outputs the concentration level of the user 101 output from the first tool unit 508 and the reproduction time information of the content 102 output from the second tool unit 509 to the concentration level determination processing unit 1102.
The concentration level determination processing unit 1102 detects a variation amount of the concentration level every predetermined unit time, and detects whether or not the variation amount of the concentration level falls below a predetermined first threshold value. Then, when the amount of change in the concentration level falls below a predetermined threshold value, a pause instruction for reproduction of the content 102 is output to the content reproduction processing unit 512.
Further, the concentration level processing unit outputs a content reproduction pause instruction to the content reproduction processing unit 512, and then detects whether or not the variation amount of the concentration level exceeds a predetermined second threshold. Then, when the fluctuation amount of the concentration level exceeds a predetermined second threshold value, a rewind instruction for several seconds and a content reproduction instruction are output to the content reproduction processing unit 512.
When the concentration processing unit outputs a content playback pause instruction to the content playback processing unit 512 and then detects that the user 101 has received a playback instruction from the content playback processing unit 512 via the operation unit 408, the content playback processing is performed. The unit 512 may output a rewind instruction for several seconds and a content reproduction instruction.

FIG. 12 is a graph showing a variation in the degree of concentration of the user 101 who plays a moving image on the moving image reproducing device over time, and a time when the moving image reproducing device pauses the reproduction of the moving image and performs the reproduction again. It is a chart.
At time T1201, the user 101 shows a high degree of concentration with respect to the content 102 being reproduced. However, due to the incoming call at time T1202, the concentration of the user 101 decreases rapidly at time T1203. The concentration level determination processing unit 1102 subtracts the concentration level at time T1202 from the concentration level at time T1203 to obtain a concentration level fluctuation amount. When the concentration level determination processing unit 1102 detects that the concentration level fluctuation amount has fallen below a predetermined first threshold value, the concentration level determination processing unit 1102 issues a command to pause the reproduction of the content 102 to the content reproduction processing unit 512. Further, the concentration degree determination processing unit 1102 holds the content reproduction time at the time T1202 in the RAM 417.
In FIG. 12, the content reproduction time at time T1202 is 10 minutes 23 seconds.

When the user 101 finishes the telephone contact at time T1204 and starts paying attention to the reproduction of the content 102 again, the concentration of the user 101 increases rapidly at time T1205. The concentration level determination processing unit 1102 subtracts the concentration level at the time point T1204 from the concentration level at the time point T1205 to obtain the concentration level fluctuation amount. When the concentration level determination processing unit 1102 detects that the variation amount of the concentration level exceeds a predetermined second threshold value, the instruction to restart the reproduction of the content 102 that has been paused is given to the content reproduction processing unit 512. To emit. At that time, the playback position of the content 102 is further rewound for several seconds from the playback time (10 minutes 23 seconds) at the time T1202 held at the time T1203.
In FIG. 12, at time T1205, the content playback time is rewound to 10 minutes 20 seconds and playback is resumed.

For example, when a moving image content such as an optical disc in which a movie or the like is recorded is reproduced by a disc reproducing apparatus and the user 101 is watching the content 102, the user 101 is in a state where the user 101 makes a call or the like. There may be a situation where the concentration level that has been reduced rapidly. The degree-of-concentration determination processing unit 1102 detects such a sudden decrease in the degree of concentration, and temporarily stops the content reproduction process in the disc reproduction apparatus. When the user's 101 concentration level recovers or when the user 101 presses the play button, the content playback is resumed from the playback position several seconds before the point when the concentration level decreased, thereby providing the user 101 with a comfortable content playback environment. can do.
In the sixth embodiment described above, the recorded content such as an optical disk has been described. However, content via a network and streaming content such as a TV can be dealt with by always recording.

[Seventh embodiment: In addition to the degree of concentration, the reproduction interruption of the content 102 is recorded in the database 802]
FIG. 13 is a block diagram showing software functions in an implementation example of the concentration processing system 1301 according to the seventh embodiment of the present invention.
The concentration processing system 1301 according to FIG. 13 is assumed to be a moving image playback device connected to a network as an example.
In FIG. 13, the content reproduction processing unit 512 reproduces the content 102 such as a moving image received from the local storage 511 such as an optical disc or the network storage 510 via the network, and displays the content on the display unit 410. At this time, in addition to the reproduction time information of the content 102, information indicating that the reproduction of the content 102 has been interrupted through the operation unit 408 is also output to the second tool unit 509. Although not shown in FIG. 13, broadcast content such as TV broadcast is also a processing target of the content reproduction processing unit 512.
The interface unit 502 displays the concentration of the user 101 output from the first tool unit 508, the content ID output from the second tool unit 509, the reproduction time information of the content 102, and the current output from the third tool unit 513. The date / time information, UID, and the like are transmitted to the customer system 505.
The customer system 505 records the UID, current date and time, content ID, content playback time, content playback interruption information, degree of concentration, etc. in the database 802.

For example, when the content 102 is a broadcast content or the like, when the user 101 feels uncomfortable with the content 102 or when the user 101 gets tired of the content 102, the viewing of the content 102 may be interrupted. If the user 101 gets tired of the content 102, the concentration of the user 101 may be low. On the other hand, when the user 101 feels uncomfortable with the content 102, it is conceivable that the concentration of the user 101 shows a correspondingly high value. That is, it is conceivable that the degree of concentration and the content 102 have a correlation. In the concentration processing system 1301 of the present embodiment, in addition to recording the concentration, information indicating that the user 101 has stopped viewing the content 102 by operating the operation unit 408 is also recorded. By recording the viewing stop information and the degree of concentration, whether the user 101 got tired of the content 102 and stopped viewing, or the content 102 caused discomfort to the user 101 and the user 101 stopped viewing, etc. It is possible to detect the correlation between the degree and the viewing stop timing.
Note that the detection of the viewing stop is not limited to the stop button, and it is preferable to record operation information of the operation buttons in the operation unit 408 such as a fast forward button and a rewind button. In broadcast content such as TV broadcast, a channel change instruction or the like can also be recorded as viewing stop information.

In the present embodiment, a concentration processing system has been disclosed.
Software constituting the concentration processing system has functions divided into an engine unit 501, a tool unit 503, and an interface unit 502 (see FIG. 5).
The engine unit 501 generates basic data such as vector data from the image data stream output from the imaging device 411.
The tool unit 503 calculates useful data for the customer system 505 such as the degree of concentration based on the data output from the engine unit 501. In addition to the degree of concentration, the tool unit 503 outputs various information of the content 102, information for specifying the user 101, and the like.
The interface unit 502 manages the exchange of data such as passing data output from the engine unit 501 to the tool unit 503 and passing data output from the tool unit 503 to the customer system 505.
In response to various customer requests, the data format request is absorbed by the interface unit 502, and the data type request is absorbed by the tool unit 503. Thus, by differentiating functions, it is possible to improve the software development efficiency and respond quickly to customer requests. In addition, the engine part 501 and the tool part 503 can be replaced | exchanged according to a use (refer FIG. 7A and FIG. 7B).

  For example, the first tool unit 508 that provides a concentration degree calculation function that requires a large amount of computer resources is not installed in the client 402 but is installed in the customer system 505. The software function to be implemented in the client 402 includes an engine unit 501 and an interface unit 502 (FIG. 8), or an engine core unit 507 and an interface unit 502 that are the core parts of the engine unit 501 (see FIG. 6). To do. That is, it can be configured so that the concentration processing system 1301 can be mounted particularly on a low resource apparatus. When software functions are differentiated in this way, flexible implementation can be realized for various types of systems.

  The degree of concentration measured by the degree-of-concentration processing system 1301 does not only represent the degree of concentration of the user 101 with respect to the content 102 but can also be information indicating the preference of the user 101. Therefore, by gathering the concentration level of the user 101 together with the gender / age of the user 101 and other attribute information on the server 414, the tendency of the concentration level can be clustered with the attribute information of the user 101. Then, conversely, by calculating the degree of similarity by comparing the concentration data of the user 101 whose gender / age and other attribute information are unknown with a known clustered concentration degree tendency, the user 101 It is possible to infer which cluster belongs to (see FIG. 9).

  The degree of concentration that the user 101 has maintained so far, such as when the user 101 is viewing the video content while the video content is being played on a predetermined playback device and the user 101 is viewing the video content, May drop sharply. In such a case, a sudden decrease in the degree of concentration of the user 101 per unit time is detected, and the content playback process in the playback device is paused. When the user's 101 concentration level is restored or when the user 101 presses the play button, the content playback is resumed from the playback position several seconds before the point when the concentration level decreased, thereby providing the user 101 with a comfortable content playback environment. (See FIG. 11).

  When the user 101 viewing the broadcast content or the like feels uncomfortable with the content 102 or when the user 101 gets tired of the content 102, the user 101 may interrupt the viewing of the content 102. Therefore, in the concentration processing system 1301, in addition to recording the concentration, information indicating that the user 101 has stopped viewing the content 102 by operating the operation unit 408 of the content reproduction apparatus is also recorded in the server 414. . By recording the viewing stop information and the degree of concentration, whether the user 101 got tired of the content 102 and stopped viewing, or the content 102 caused discomfort to the user 101 and the user 101 stopped viewing, etc. It is possible to detect the correlation between the degree and the viewing stop timing (see FIG. 13).

The embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and other modifications and application examples are provided without departing from the gist of the present invention described in the claims. including.
For example, the above-described embodiment is a detailed and specific description of the configuration of the apparatus and the system in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is also possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Each of the above-described configurations, functions, processing units, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, each of the above-described configurations, functions, and the like may be realized by software for interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function must be held in a volatile or non-volatile storage such as a memory, hard disk, or SSD (Solid State Drive), or a recording medium such as an IC card or an optical disk. Can do.
In addition, the control lines and information lines are those that are considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

  DESCRIPTION OF SYMBOLS 101 ... User, 102 ... Content, 103 ... Display, 104 ... Camera, 201 ... LCD display, 202 ... Web camera, 203 ... Laptop, 204 ... LCD display, 205 ... Web camera, 206 ... Portable wireless terminal, 207 ... LCD display, 208 ... In-camera, 401 ... Concentration processing system, 402 ... Client, 403 ... CPU, 404 ... ROM, 405 ... RAM, 406 ... Non-volatile storage, 407 ... RTC, 408 ... Operation unit, 409 ... Bus, 410: Display unit, 411 ... Imaging device, 412 ... NIC, 413 ... Internet, 414 ... Server, 415 ... CPU, 416 ... ROM, 417 ... RAM, 418 ... Non-volatile storage, 419 ... NIC, 420 ... Bus, 501 ... Engine part, 50 ... interface part 503 ... tool part 504 ... content reproduction part 505 ... customer system 506 ... customer tool 507 ... engine core part 508 ... first tool part 509 ... second tool part 510 ... network storage, 511 ... Local storage, 512 ... Content reproduction processing unit, 513 ... Third tool unit, 601 ... Concentration processing system, 701 ... Concentration processing system, 702 ... Concentration processing system, 703 ... First engine unit, 704 ... No. Two engine parts, 705 ... fourth tool part, 706 ... fifth tool part, 801 ... concentration processing system, 802 ... database, 901 ... concentration processing system, 902 ... cluster analysis processing part, 903 ... user master, 904 ... Concentration degree table, 905 ... Cluster table, 1101 ... Concentration Processing system, 1102 ... concentration determination process section, 1301 ... concentration level processing system

Claims (4)

A display for displaying content;
An imaging device installed in a direction in which the face of the user viewing the display unit can be photographed;
An engine core unit and an engine unit having a software function other than the engine core unit, vector data indicating the orientation of the user's face and the direction of the user's line of sight from an image data stream output from the imaging device; An engine unit that outputs a change amount that is fluctuation on a time axis of vector data;
A first tool unit having a concentration level calculation function for calculating a concentration level for the content displayed on the display unit by the user from the vector data and the change amount;
An input / output control unit that outputs the vector data and / or the degree of concentration to a predetermined information processing device,
The engine core unit detects whether or not the user's face is present, and outputs coordinate information of an extraction point of the user's face from a three-dimensional image of the user's face,
The engine unit having a software function other than the engine core unit calculates vector data indicating the orientation of the user's face and the direction of the line of sight of the user, and the amount of change of the vector data,
The first tool unit outputs whether or not the user is gazing at the content displayed on the display unit as data having a predetermined range.
Concentration processing system. Furthermore,
The display unit, the imaging device, the engine core unit, and the input / output control unit are equipped in a client,
The engine part having the software function other than the engine core part and the first tool part are equipped in the predetermined information processing apparatus,
The concentration processing system according to claim 1. Furthermore,
A second tool unit that outputs reproduction position information of the content displayed on the display unit to the input / output control unit;
A content reproduction processing unit that reproduces the content on the display unit and outputs reproduction position information of the content to the second tool unit;
When the amount of change in the degree of concentration when the degree of concentration at the previous time is subtracted from the degree of concentration at a certain point in time is below a predetermined first threshold value, , Instruct to pause playback of the content,
When the degree of concentration change when the degree of concentration at the previous time is subtracted from the degree of concentration at a certain point in time exceeds a predetermined second threshold value and the degree of concentration rises, the content reproduction processing unit On the other hand, a concentration level determination processing unit for instructing resumption of reproduction of the paused content,
The concentration processing system according to claim 1. Furthermore,
An operation unit for accepting an operation of the user, and transmits the reproduction instruction information and the reproduction stop instruction information before Symbol content,
Reproduces the contents on the display unit, and the playback position information of the content, the content reproduction process section for force out a reproduction instruction information and the reproduction stop instruction information of the contents from the operation unit,
A second content ID and reproduction position information of the content displayed on the display unit, a reproduction instruction information of the content, and a reproduction stop instruction information are received from the content reproduction processing unit and output to the input / output control unit A tool part;
A user ID for uniquely identifying the user, and a third tool unit that outputs current date and time information to the input / output control unit,
The input / output control unit includes the concentration level output from the first tool unit, the content ID and playback position information of the content output from the second tool unit, the playback instruction information of the content, and the playback stop instruction. Transmitting the information, the user ID output from the third tool part, and the current date and time information to a predetermined information processing device;
The predetermined information processing apparatus includes:
A database for recording the degree of concentration, the content ID and playback position information of the content, the playback instruction information and playback stop instruction information of the content, the user ID, and the current date and time information;
The concentration processing system according to claim 1.

Images