JP2019086912A - Degree-of-interest measurement system and method - Google Patents

Abstract

To determine whether or not a user is interested in contents of a lecture or the like even if a user's face has a certain size or less.SOLUTION: A degree-of-interest measurement system for measuring a user's degree of interest in an object of interest, includes an arithmetic operation device for obtaining a color image, as an input, from an imaging device for photographing a user's state. The arithmetic operation device is configured to be capable of switching between a first mode in which the user's face in the color image is recognized to count the number of faces and a second mode in which the area of pixels with a predetermined RGB value in the color image is calculated to estimate the number of faces on the basis of the area.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an interest level measurement technique that determines whether or not the user is interested in the content of a lecture or the like.

  There is known a technique for detecting an overlap of visual areas of a conference participant and detecting attention information (an attention area, an attention time).

  For example, the technology described in Patent Document 1 is known as a method of creating attention information. In Patent Document 1, “the focused information processing unit 108 detects an overlap of the visual recognition areas of the conference participants. Reference numeral 240 in FIG. 3C indicates an area (overlap area) in which the visual recognition areas overlap. The overlapping area 240 has two-dimensional coordinate values (x, y) of the upper left origin and the width and height (w, h) of the area, and can specify the place and calculate the size. Creates attention information based on the detection result of the overlapping area, which is information about a scene that has attracted the attention of the meeting participants.

  Specifically, the attention information includes at least a region of interest and a time when attention was focused (time of attention). The attention area is an area where the attention of the meeting participants is collected. The attention area is obtained by OR operation of the visual recognition areas between the continuous frames when the overlapping of the frames in which the visual recognition areas are detected in the conference data is continuous (OR area). The attention time corresponds to the number of consecutive frames in which overlapping of the viewing areas is detected in the conference data. Is stated. The detection method described in Patent Document 1 can provide a mechanism for focusing and reproducing the content of the participants' attention at the conference.

JP, 2014-204411, A

  In the technique of Patent Document 1, the visual recognition area is extracted by predicting based on the position of the pupil of the participant and the direction of the face, so the face can not be detected unless the size is a certain size or more. There was a problem that it could not count.

  One aspect of the present invention is a degree-of-interest measurement system that measures the degree of interest of a user to a target of interest, and includes an arithmetic device that obtains a color image from an imaging device that captures a user's state. Recognizes the user's face in the color image, calculates the area of the pixel with a predetermined RGB value in the color image and the first mode to count the number of faces, and estimates the number of faces based on the area The second mode is configured to be switchable.

  Another aspect of the present invention is a method of measuring the degree of interest of a user with respect to a target of interest in a hall using a computer provided with a computing device, a storage device, an input device, and an output device, using an input device A first mode for acquiring a color image from an imaging device that has captured the state of the user, the arithmetic device pattern-recognizing the user's face in the color image, and calculating the degree of interest based on the recognition result; The area of a predetermined color is calculated, and switching to the second mode in which the degree of interest is calculated based on the area is enabled.

  According to the present invention, it is possible to determine whether or not the user is interested in the contents of a lecture or the like even if the face is smaller than a certain size. Problems, configurations and effects other than the above are clarified by the description of the following embodiments.

FIG. 1 is a block diagram showing an example of a configuration of a computer system of a first embodiment. FIG. 2 is a functional block diagram showing a hardware configuration and a software configuration of a people counting device of a first embodiment. FIG. 7 is a table showing an example of a data structure of a hall and the number of people included in the image DB of the first embodiment. 7 is a flowchart illustrating an example of a process of counting the number of people according to the first embodiment. It is a conceptual diagram which shows the concept of interest degree measurement of an Example. FIG. 16 is a flowchart illustrating an example of the people counting process of the second embodiment. FIG. 16 is a flowchart illustrating an example of the people counting process of the third embodiment. FIG. 18 is a conceptual diagram for explaining an example of display of the third embodiment. FIG. 18 is a flowchart illustrating an example of the people counting process of the fourth embodiment. FIG. 18 is a conceptual diagram showing an example of the data structure of the fourth embodiment.

  Hereinafter, embodiments of the present invention will be described using the drawings. However, the present invention should not be construed as being limited to the description of the embodiments below. Those skilled in the art can easily understand that the specific configuration can be changed without departing from the spirit or the spirit of the present invention.

  In the configurations of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and overlapping descriptions will be omitted.

  In the present specification, the expressions “first”, “second”, “third” and the like are used to identify components, and are not necessarily limited in number or order.

  The positions, sizes, shapes, ranges, and the like of the components shown in the drawings and the like may not represent actual positions, sizes, shapes, ranges, and the like in order to facilitate understanding of the invention. Accordingly, the present invention is not limited to the position, size, shape, range, and the like disclosed in the drawings and the like.

  It will be as follows if a representative example of the following examples is shown. That is, it is a computer system that measures whether or not participants in a lecture or the like are interested in the contents of the story, and the computer system is capable of face detection in a narrow venue and the brightness and color of a face in a wide venue. A computing device is provided which detects a layer of a person of interest from the information. In addition, the computer system has an imaging device, a computing device, a storage device connected to the computing device, and an interface connected to the computing device, and counts the number of people using a color image acquired by the imaging device via the interface Connect with a database that stores images used for processing. The arithmetic device counts the number of people according to whether or not face detection can be performed, and counts the number of people from the number of face areas when face detection can be performed.

  FIG. 1 is a diagram illustrating an example of the configuration of the interest level measurement system 100 according to the first embodiment. The interest level measurement system 100 includes a computer system 101 and a mother brain computer system 102. The computer system 101 is also connected to the mother brain computer system 102 via the network 103. Although there is one computer system 101 shown in FIG. 1, there may be a plurality. The computer system 101 and the mother brain computer system 102 can both be configured by ordinary computers.

  In this embodiment, it is assumed that the computer system 101 is installed in one or more halls 104 where lectures and events are performed. The computer system 101 acquires data representing the status of a presentation or an exhibition at the hall 104. For example, it is various data acquired via an input device, such as a picture, a sound, or the number of visitors. The computer system 101 may further include a display function or an input function that assists the operation of the hall.

  In addition, the mother brain computer system 102 is located at a remote place, manages information, and controls one or more halls 104 entirely. However, it is also possible to configure the computer system 101 and the mother brain computer system 102 integrally and install them in the hall 104.

  It should be noted that instead of the computer system 101 in the hall 104, only devices configured with a display, a sound collection microphone, and speakers are placed, and data is transmitted / received via the network 103, so that the function of the computer system 101 can be mother-brained. It may be shared by the computer system 102.

  The interest level measurement system 100 according to the first embodiment measures the interest level of people who participated in a presentation or an exhibition (hereinafter referred to as a user) based on data representing the state of the presentation or the display acquired at the hall 104.

  FIG. 2 is a diagram showing a hardware configuration and a software configuration of the computer system 101 of the first embodiment. The computer system 101 can be configured by, for example, a general computer, and includes an arithmetic device 201, a storage device 202, a communication device 203, an imaging device 204, an audio reproduction device 205, and a sound collection device 206. The imaging device 204 and the sound collecting device 206 are a type of input device, and the sound reproducing device 205 is a type of output device. Other known input / output devices may be provided. The pieces of hardware are connected to one another via an internal bus or the like.

  The arithmetic device 201 is a device such as a CPU (Central Processing Unit). The arithmetic device 201 executes a program stored in the storage device 202. The arithmetic unit 201 executes processing according to a program to operate as a module that implements a specific function. In the following description, when processing is described using a module as a subject, it indicates that the arithmetic device 201 is executing a program that implements the module.

  The storage device 202 is a device such as a memory, and stores a program executed by the arithmetic device 201 and information necessary for executing the program. The storage device 202 also includes a work area used by the program. The computer system 101 may have a storage device such as a hard disk drive (HDD) and a solid state drive (SSD).

  In the present embodiment, the storage device 202 stores a program for realizing the control module 211 and stores the image DB 212. The communication device 203 is a device such as a network interface, and is connected to another device via the network 103. The imaging device 204 is a device such as a camera, and acquires a color image in a predetermined range. As will be described later, in the present embodiment, the imaging apparatus is installed, for example, at the position of a lecturer or a screen, and photographs the user's situation. The voice reproduction device 205 is a device such as a speaker, and reproduces the contents of speech as sound. The sound collection device 206 is a device such as a sound collection microphone and acquires voice.

  In the present embodiment, the degree of interest is measured using the image acquired by the imaging device 204. The measurement of the degree of interest is performed by determining whether the user's face is oriented in a predetermined direction. For example, in the case where 100 users participate, if 50 of them are facing a predetermined direction, it is determined that the degree of interest is 50%, assuming that the 50 people are focusing on the predetermined direction. Be done.

  For example, when measuring the degree of interest of an attendee in a lecture, a camera as the imaging device 204 is provided at the position of the lecturer or monitor screen, and the face of the attendee facing the camera is detected to calculate the degree of interest Do. That is, it is determined that a person who is facing downward, sideways or in another direction is not interested because it does not face the camera (it does not look at it). In this embodiment, as a face detection method, a first method (first mode) by counting the number of faces using pattern recognition, and a second method by area ratio of face parts using color information (skin color) By switching and using (second mode), accurate measurement of the degree of interest is enabled regardless of the characteristics of the venue.

  In the first method based on the count of the number of faces, the number of faces is counted by recognition of a face pattern which is known in digital cameras and the like. In the second method based on the area ratio of the color information face part, it is determined whether the RGB element of each pixel of the color image signal is a predetermined color range indicating skin color, and the skin color area in the image is the face part Indicates the area. In the second method, information on luminance may be used in addition to color.

  FIG. 3 is a view showing an example of the data structure of the management database 300 used by the interest level measurement system 100 of the first embodiment. In the example of the data of FIG. 3, data of a plurality of venues 104 are collectively managed, and is assumed to be managed by the mother brain computer system 102. However, the computer system 101 of each venue 104 may manage data of each venue.

  Information stored in the management database 300 will be described with reference to FIG. The management database 300 is a database used to hold the number of people when the users 110 are counted. The management database 300 includes an entry including a hall ID 301, which is an ID of the hall 104, a hall name 302, a number 303, a number of seats 304, a number of attention 305, and a date and time 306. One entry corresponds to one database group (partial group).

  The hall ID 301 is a field for storing identification information for each hall. The venue name 302 is a field for storing the venue name. The number of persons 303 is a field for storing the number of visitors. The number of visitors can be counted, for example, each time the user enters, and can be input from the input means of the computer system 101. Alternatively, it may be estimated that the number of tickets sold is equal to the number of visitors, and may be input based on the number of tickets. The number of seats 304 is the number of seats when there is a seat, and is a field for storing the number corresponding to the capacity of the hall. The attention number of people 305 is a field for storing the number of people who are watching, which is measured by the first method or the second method described above.

  The above fields are examples and may have other fields or may omit some of them. The number of people 303 and the number of seats 304 are used to set the total number of users as a parameter when measuring the degree of interest. For example, the number of people 303 can be used as a parameter as it is. Alternatively, when all seats are designated and tickets are sold out, the number of seats 304 can be a population. These are merely examples, and the parameter may be estimated by other methods. For example, when the information in the management database 300 is not used, it is also possible to perform pattern recognition of the human body and estimate the number of parameters, as in the case of face pattern recognition.

  FIG. 4 is a diagram showing an outline of processing executed by the computer system 101 according to the first embodiment. When the computer system 101 detects periodic or occurrence of an event, the computer system 101 acquires an image using the imaging device 204 (step S401).

  Next, the control module 211 of the computer system acquires, from the management database 300, parameters for switching the first method or the second method for interest degree measurement (step S402).

  The concept of switching the first method or the second method will be described with reference to FIG. In the present embodiment, the degree of interest is measured based on the face of the user facing in a predetermined direction. In this case, there are large venues (for example, 300 m square) such as a stadium or public hall and small venues (10 m square) such as a classroom or a conference room. In a large venue, the size of the face portion in the image captured by the imaging device 204 may be small, the number of faces may be large, the resolution may be insufficient, or the processing amount may be too large. In this case, in the first method using face pattern recognition, face recognition in a large venue can not be performed, and measurement may be difficult or inaccurate. Therefore, in the present embodiment, the first method and the second method are switched depending on the size of the hall. Here, the size can be reworded as the size, the area, and the number of people accommodated. These are sometimes referred to as "scales".

  For example, the number of people 303 and the number of seats 304 of the management database 300 can be used as the scale parameter for this purpose. For example, the threshold is set to 300 (person), and is compared with the parameter. If less than the threshold, the first method (face pattern recognition) is selected, and if it is greater than the threshold, the second method (skin color area detection) is set (step S403).

  That is, as shown in FIG. 4, if the parameter is equal to or greater than the threshold, it is determined that the venue is wide and the skin color area of the face is calculated based on the RGB signal. This can be done by counting the number of pixels having a predetermined RGB value (step S404). Furthermore, in order to calculate the number of face areas from the skin color area, the skin color area is divided by the area per face (step S405). The area per face may be determined in advance by setting a prescribed value or by estimating the number of persons 303 or the number of seats 304 and the imaging range of the imaging device 204.

  Although the size of the hall is determined based on the number of people 303 and the number of seats 304 in the above description, other methods may be used. For example, data on the size of the venue may be stored in advance in the management database 300, and the size of the venue may be compared with a threshold. Alternatively, it may be set for each venue whether to use the first method or the second method. For example, in the example of FIG. 3, it may be considered that the first method is used when the venue name 302 is “meeting room”, and the second method is used for “exhibition hall”.

  On the other hand, if the parameter is less than the threshold, it is determined that the venue is narrow, and the number of face areas is calculated by pattern recognition (step S406). The control module 311 detects a face area from the received image. The process of detecting the face area is known in the art, and can be performed, for example, by comparing the face pattern with the pattern in the image.

  As a result of step S405 or S406, it is possible to calculate the number of people who are facing a specific direction, that is, people who are paying attention. The control module 311 counts the number of notable people from the calculation result of the number of face areas (step S407).

  Next, the control module 311 acquires, from the management database 300, the number of people 303 and the number of seats 304 as parameters. Then, the degree of interest (%) is obtained by dividing the count result in step S407 by a parameter (step S408).

  In the first embodiment, the information in the management database 300 is used, and the case where the size of the hall is known in advance is taken as an example. However, for example, the venue may be a temporary venue, and the size of the venue may not be known in advance. In the second embodiment, an example which can cope with this case will be described. Although the basic configuration is the same as that of the first embodiment, it is assumed that the data of the management database 300 can not obtain a parameter indicating the size of the hall.

  FIG. 6 is a diagram showing an outline of processing executed by the computer system 101 of the second embodiment. The computer system 101 acquires an image using the imaging device 204 when detecting periodic or occurrence of an event.

  First, the control module 211 of the computer system calculates the resolution of the image using the received image (step S501).

  Next, the upper limit value (threshold value) of the number of detected faces is calculated (step S502). That is, detection of the number of faces by pattern recognition is generally restricted by the processing speed, and when the number of faces is too large, counting may not be possible. This mainly depends on the amount of processing by software. Therefore, when the upper limit is exceeded, the number of faces can not be counted. At this time, the control module 311 automatically calculates the upper limit value from the resolution of the image calculated in step S501. Also, the upper limit value may be given manually or manually.

  The control module 311 detects a face area from the received image (step S503). The process of detecting the face area is known in the art, and can be performed, for example, by comparing the pattern of the reference face with the pattern in the image.

  The control module 311 determines whether a face area can be detected. That is, it is determined whether pattern recognition has succeeded. If the venue is large and the size of the face in the image is too small, the resolution may be insufficient and pattern recognition may not be possible (step S504).

  If it is determined that the face area can not be detected, the control module 311 executes a process of calculating the skin color area of the face (step S404). In order to calculate the skin color area of the face, the area of the skin color area may be measured continuously or intermittently in time series. The control module 311 executes the process of calculating the number of face areas based on the calculation result of the skin color area of the face (step S405). The control module 311 counts the number of people from the calculation result of the number of face areas (step S407). These processes are the same as in the first embodiment.

  If it is determined in step S504 that the face area can be detected, the control module 311 executes the process of calculating the number of face areas based on the detection result of the face area (step S406). During counting of the number of faces, the control module 311 determines whether the calculated number of face areas is equal to or greater than a threshold (step S506). If the number of detected face areas exceeds a threshold (upper limit), it is determined that the subsequent counting is impossible or inaccurate.

  Therefore, when it is determined that the number of face areas is equal to or more than the threshold, the control module 311 executes a process of calculating the skin color area of the face (step S404). In order to calculate the skin color area of the face, the area of the skin color area may be measured continuously or intermittently in time series. The control module 311 executes the process of calculating the number of face areas based on the calculation result of the skin color area of the face (step S405).

  If it is determined that the number of face areas is less than the value, the control module 311 counts the number of people from the calculation result of the number of face areas (step S407).

  When the computer system of the second embodiment is applied to the core business system, the number of people is counted in the narrow hall whether or not the face can be detected even if there is no information about the hall. By measuring the area of the region continuously or intermittently in time series, it is possible to determine whether or not the user is interested in the contents of a lecture or the like even if the face is smaller than a certain size.

  The third embodiment differs from the first embodiment in that the situation of the user who is watching a presentation or a class is fed back to the performer. The third embodiment will be described below focusing on the difference from the first embodiment. The configuration of the computer system of the third embodiment is the same as the configuration of the computer system of the first embodiment. The configurations of the computer system 101 and the mother brain computer system 102 of the third embodiment are the same as the configurations of the respective devices of the first embodiment. Moreover, the data structure of the database which each apparatus hold | maintains is the same as the data structure of the database of Example 1. FIG. Except for parts particularly described in the third embodiment, it may be considered similar to the first or second embodiment.

  The number of watched people calculation process according to the third embodiment will be described with reference to FIG. The control module 311 acquires the number of users (the number of all attendees) serving as a parameter from the management database 300 (step S701). Then, by measuring continuously or intermittently in time series, the number of people of interest is calculated as in the first or second embodiment (step S407). Next, the degree of interest (degree of attention) is calculated from the ratio of the parameter and the calculated number of gazers (step S408).

Next, the parameter number, the number of attention persons, and the attention degree are displayed on the screen (step S702). Further, the feedback may not only be displayed on the screen, but may be audio feedback such as an alarm.
The other processes are the same as in the first or second embodiment. However, in the present embodiment, the number of gazers and the degree of interest are displayed in time series, so that temporal changes can be known.

  FIG. 8 shows an example in which the parameter number, the number of people watching, and the degree of attention are displayed on the screen 1010. The percentage (%) of the number of people focusing on the attention level is displayed.

  According to the third embodiment, in order to feed back the situation of the user who is watching a presentation or a class to the performer in real time, when the performer devises the content of the explanation, it is possible to expect improvement in the user's understanding level.

  The fourth embodiment is different from the first embodiment in that the situation of the user who is viewing a presentation or a product is recorded. Hereinafter, the fourth embodiment will be described focusing on the difference from the first embodiment.

  The configuration of the computer system of the fourth embodiment is the same as the configuration of the computer system of the first embodiment. The configurations of the computer system 101 and the mother brain computer system 102 of the fourth embodiment are the same as the configurations of the respective devices of the fourth embodiment. Also, the data structure of the database held by each device is the same as the data structure of the database of the fourth embodiment.

  The flow of the process of the fourth embodiment will be described with reference to FIG. Except for parts particularly described in the fourth embodiment, it may be considered similar to the first or second embodiment. First, the watched number of people calculation process of the fourth embodiment will be described. The control module 311 counts the number of people to be a parameter (step S701).

  Next, an attention target is specified. For example, it is assumed that slides are sequentially displayed on the screen in a lecture, and the user's interest in each slide is measured. In that case, the target of attention is set corresponding to the slide to be displayed. The target of interest is identified, for example, by the number of the slide currently displayed (step S1101). The target of attention may be real data such as presentation data and products such as electronic data.

  The attention number is calculated by measuring continuously or intermittently in time series for each attention target (step S407). The degree of interest is calculated from the calculated ratio of the number of attention persons and the number of attention for each attention object (step S408). The calculated degree of interest for each target of interest is recorded, for example, in the image DB 212 (step S1104). The other processes are the same as in Example 1 or 2.

  FIG. 10 is a conceptual view of data recorded in the image DB. The degree of interest is recorded corresponding to the slide number that is the target of interest. According to the fourth embodiment, since the degree of attention of the user who is viewing a presentation or a product is recorded, it is possible to expect display of a material or product with a high degree of attention.

  According to the embodiment described above, the number of people is counted as to whether or not a face can be detected in a narrow venue, and the face is measured continuously or intermittently in time series in a large venue with the area of the skin color luminance region. It is possible to determine whether you are interested in the contents of lectures etc.

100 Degree of Interest Measurement System 101 Computer System 102 Mother Brain Computer System 103 Network 104 Venue

Claims (11)

An interest measurement system for measuring a user's interest in an object of interest, comprising:
The arithmetic device is provided with an input device that obtains a color image from an imaging device that captures the state of the user.
The arithmetic device is
A first mode for recognizing the user's face in the color image and counting the number of faces;
A second mode for calculating an area of a pixel having a predetermined RGB value in the color image and estimating the number of faces based on the area;
Configured to be switchable,
Degree of interest measurement system. The arithmetic device is
Compare the input parameter with the threshold,
Switching between the first mode and the second mode based on a comparison result,
The interest measurement system according to claim 1. The arithmetic device is
Information from a management database storing at least one information of the size of the venue where the presence of the target and the user is assumed and the number of the users assumed to be at the venue is input,
The first mode and the second mode are switched using the size of the hall and at least one piece of information of the number of users as the parameter.
The interest measurement system according to claim 2. The arithmetic device is
Based on the recognition result of the face of the user in the color image,
Switching between the first mode and the second mode,
The interest measurement system according to claim 1. The arithmetic device is
Based on the count result of the number of faces of the user in the color image,
Switching between the first mode and the second mode,
The interest measurement system according to claim 1. An interest level measurement method for measuring a user's level of interest in an object of interest in a hall using a computer including an arithmetic unit, a storage unit, an input unit, and an output unit.
Acquiring a color image from an imaging device that captures the state of the user by the input device;
The arithmetic device is
A first mode for pattern-recognizing the face of the user in the color image, and calculating a degree of interest from the recognition result;
A second mode for calculating an area of a predetermined color in the color image and calculating a degree of interest based on the area;
Make it switchable,
Degree of interest measurement method. The first mode and the second mode are
Selected based on the size of the venue,
The interest level measurement method according to claim 6. If the size of the venue is less than a predetermined threshold, select the first mode,
The second mode is selected when the size of the venue is equal to or greater than a predetermined threshold.
The interest level measurement method according to claim 7. The first mode and the second mode are
It is selected based on the pattern recognition result of the user's face,
The interest level measurement method according to claim 6. If the pattern recognition of the user's face is successful, the first mode is selected,
If the pattern recognition of the face of the user is not successful, the second mode is selected.
The interest level measurement method according to claim 9. If the pattern recognition of the face of the user succeeds, the face area where the pattern recognition is performed is counted, and if the count result is less than a predetermined threshold, the first mode is selected, and if the count result is equal to or more than a predetermined threshold Select the second mode,
The method according to claim 9, wherein if the pattern recognition of the face of the user is not successful, the second mode is selected.

Images