JP7001429B2 - Interest measurement system and method - Google Patents

Description

The present invention relates to a degree of interest measurement technique for determining whether or not an person is interested in the content of a lecture or the like.

There is known a technique for detecting attention information (attention area, attention time) by detecting overlap of visual areas of conference participants.

As a method for creating attention information, for example, the technique described in Patent Document 1 is known. In Patent Document 1, "The information processing unit 108 of interest detects the overlap of the viewing areas of the conference participants. Reference numeral 240 in FIG. 3C indicates an overlapping area (overlapping area). The overlapping region 240 has a two-dimensional coordinate value (x, y) of the upper left origin and a width and height (w, h) of the region, and can specify the location and calculate the size. Creates attention information based on the detection result of the overlapping area. The attention information is information about a scene that attracts the attention of the conference participants.

Specifically, the attention information includes at least a region of interest and a time of attention (time of interest). Areas of interest are areas that have attracted the attention of conference participants. The region of interest is obtained by ORing the visible regions between the consecutive frames when the overlapping of the frames in which the overlap of the visual regions is detected is continuous in the conference data (OR region). The attention time corresponds to the number of consecutive frames in which the overlap of the visual recognition areas is detected in the conference data. "It is stated. The detection method described in Patent Document 1 can provide a mechanism for reproducing content focused on the content that attracted the attention of the participants at the conference.

Japanese Unexamined Patent Publication No. 2014-204411

In the technique of Patent Document 1, since the visual recognition area is extracted by predicting based on the position of the pupil of the participant and the direction of the face, the face cannot be detected unless the face is a certain size or larger, and the number of people. There was a problem that it could not be counted.

One aspect of the present invention is an interest level measuring system that measures a user's degree of interest in an object of interest, and includes an arithmetic device that obtains a color image from an image pickup device that captures a user's state as an input. The first mode that recognizes the user's face in the color image and counts the number of faces, calculates the area of the pixel having a predetermined RGB value in the color image, and estimates the number of faces based on the area. The second mode and the second mode can be switched.

Another aspect of the present invention is a method of measuring the degree of interest of a user with respect to an object of interest at a venue by using a computer including an arithmetic unit, a storage device, an input device, and an output device, wherein the input device is used. The first mode, which acquires a color image from an image pickup device that captures the user's state, recognizes the user's face in the color image as a pattern, and calculates the degree of interest based on the recognition result, and the color image. It is possible to switch between the second mode in which the area of a predetermined color is calculated and the degree of interest is calculated based on the area.

According to the present invention, it is possible to determine whether or not a person is interested in the content of a lecture or the like even if the face is smaller than a certain size. Issues, configurations and effects other than the above will be clarified by the description of the following examples.

It is a block diagram which shows an example of the structure of the computer system of Example 1. FIG. It is a functional block diagram which shows the hardware configuration and software configuration of the number of people counting apparatus of Example 1. FIG. It is a table diagram which shows an example of the data structure of the venue and the number of people included in the image DB of Example 1. FIG. It is a flowchart explaining an example of the number of people count processing of Example 1. FIG. It is a conceptual diagram which shows the concept of interest degree measurement of an Example. It is a flowchart explaining an example of the number of people count processing of Example 2. It is a flowchart explaining an example of the number of people count processing of Example 3. It is a conceptual diagram explaining an example of the display of Example 3. FIG. It is a flowchart explaining an example of the number of people count processing of Example 4. It is a conceptual diagram which shows an example of the data structure of Example 4. FIG.

Hereinafter, examples of the present invention will be described with reference to the drawings. However, the present invention is not limited to the description of the embodiments shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or purpose of the present invention.

In the configuration of the invention described below, the same or similar configurations or functions are designated by the same reference numerals, and duplicate description will be omitted.

The notations such as "first", "second", and "third" in the present specification are attached to identify the components, and do not necessarily limit the number or order.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

A typical example of the following embodiment is as follows. That is, it is a computer system that measures whether or not participants in lectures are interested in the content of the story. The computer system can detect faces in a small venue, and the brightness and color of the face in a large venue. It is equipped with an arithmetic unit that detects the layer of people who are interested in the information. Further, the computer system has an image pickup device, an arithmetic unit, a storage device connected to the arithmetic unit, and an interface connected to the arithmetic unit, and counts the number of people using the color image acquired by the image pickup device via the interface. Connect to the database that stores the images used for processing. The arithmetic unit counts the number of people depending on whether or not the face can be detected, and if the face can be detected, the number of people is counted from the number of face areas.

FIG. 1 is a diagram showing an example of the configuration of the interest level measurement system 100 of the first embodiment. The interest level measurement system 100 includes a computer system 101 and a mother brain computer system 102. Further, the computer system 101 is connected to the mother brain computer system 102 via the network 103. Although there is one computer system 101 shown in FIG. 1, there may be a plurality of computer systems 101. Both the computer system 101 and the mother brain computer system 102 can be configured by a normal computer.

In this embodiment, it is assumed that the computer system 101 is installed in one or more venues 104 where lectures and events are held. The computer system 101 acquires data representing the status of a presentation or exhibition at the venue 104. For example, various data acquired via an input device such as video, audio, or the number of visitors. The computer system 101 may also have a display function, an input function, and the like that assist the operation of the venue.

In addition, the Mother Brain computer system 102 manages information in a remote location and controls one or a plurality of venues 104 as a whole. However, it is also possible to integrally configure the computer system 101 and the mother brain computer system 102 and install them at the venue 104.

In addition, instead of the computer system 101, only a device consisting of a display, a sound collecting microphone, and a speaker is placed in the venue 104, and data is transmitted and received via the network 103 to bring the functions of the computer system 101 to the mother brain. It may be shared by the computer system 102.

The interest level measurement system 100 of the first embodiment measures the degree of interest of people (hereinafter referred to as users) who participated in the presentation or exhibition based on the data representing the situation of the presentation or exhibition acquired at the venue 104.

FIG. 2 is a diagram showing a hardware configuration and a software configuration of the computer system 101 of the first embodiment. The computer system 101 can be configured by, for example, a general computer, and includes an arithmetic unit 201, a storage device 202, a communication device 203, an image pickup device 204, a voice reproduction device 205, and a sound collection device 206. The image pickup device 204 and the sound collecting device 206 are a kind of input device, and the sound reproduction device 205 is a kind of output device. In addition to this, a known input / output device may be provided. The hardware is connected to each other via an internal bus or the like.

The arithmetic unit 201 is a device such as a CPU (Central Processing Unit). The arithmetic unit 201 executes a program stored in the storage device 202. The arithmetic unit 201 operates as a module that realizes a specific function by executing processing according to a program. In the following description, when the process is described with the module as the subject, it means that the arithmetic unit 201 is executing the program that realizes the module.

The storage device 202 is a device such as a memory, and stores a program executed by the arithmetic unit 201 and information necessary for executing the program. The storage device 202 also includes a work area used by the program. The computer system 101 may have a storage device such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

In this embodiment, the storage device 202 stores a program that realizes the control module 211, and also stores the image DB 212. The communication device 203 is a device such as a network interface, and is connected to another device via the network 103. The image pickup device 204 is a device such as a camera, and acquires a color image in a predetermined range. As will be described later, in this embodiment, the image pickup device is installed at the position of a lecturer or a screen, for example, and the state of the user is photographed. The voice reproduction device 205 is a device such as a speaker, and reproduces the utterance content as voice. The sound collecting device 206 is a device such as a sound collecting microphone and acquires sound.

In this embodiment, the degree of interest is measured using the image acquired by the image pickup apparatus 204. The degree of interest is measured by determining whether or not the user's face is facing a predetermined direction. For example, when 100 users are participating and 50 of them are facing a predetermined direction, it is assumed that 50 of them are paying attention to the predetermined direction, and the degree of interest is determined to be 50%. Will be done.

For example, when measuring the degree of interest of attendees in a lecture, a camera, which is an image pickup device 204, is installed at the position of the lecturer or the monitor screen, and the face of the attendee facing the camera is detected to calculate the degree of interest. do. That is, a person who is facing down, sideways, or in another direction is not looking at the camera (not paying attention), so it is judged that he / she is not interested. In this embodiment, as a face detection method, a first method (first mode) by counting the number of faces using pattern recognition and a second method based on the area ratio of the face portion using color information (skin color). By switching and using (second mode), it is possible to accurately measure the degree of interest regardless of the characteristics of the venue.

In the first method of counting the number of faces, the number of faces is counted by recognizing a face pattern known by a digital camera or the like. In the second method based on the area ratio of the color information face portion, it is determined whether or not the RGB element of each pixel of the color image signal is within a predetermined color range indicating the skin color, and the area of the skin color in the image is the face portion. It shall indicate the area. In the second method, not only the color but also the luminance information may be used.

FIG. 3 is a diagram showing an example of the data structure of the management database 300 used by the interest level measurement system 100 of the first embodiment. In the example of the data in FIG. 3, the data of the plurality of venues 104 are collectively managed, and the mother brain computer system 102 manages them. However, the data for each venue may be managed by the computer system 101 of each venue 104.

The information stored in the management database 300 will be described with reference to FIG. The management database 300 is a database used for holding the number of people when the user 110 is counted. The management database 300 includes an entry composed of a venue ID 301, a venue name 302, a number of people 303, a number of seats 304, a number of people of interest 305, and a date and time 306, which are IDs of the venue 104. One entry corresponds to one database group (subgroup).

The venue ID 301 is a field for storing identification information for each venue. The venue name 302 is a field for storing the venue name. The number of people 303 is a field for storing the number of visitors. The number of visitors is counted each time the user enters, and can be input from the input means of the computer system 101, for example. Alternatively, it may be estimated that the number of admission tickets sold is equal to the number of visitors, and the number of admission tickets may be used as the basis for input. The number of seats 304 is the number of seats when there are seats, and is a field for storing the number corresponding to the capacity of the venue. The number of people of interest 305 is a field for storing the number of people who are watching, as measured by the first method or the second method described above.

The above field is an example, and may have other fields or may omit some of them. The number of people 303 and the number of seats 304 are used to set the total number of users, which is a parameter when measuring the degree of interest. For example, the number of people 303 can be used as it is as a parameter. Alternatively, if all seats are designated and the ticket is sold out, the number of seats 304 can be used as a parameter. These are examples, and the population parameter may be estimated by other methods. For example, when the information in the management database 300 is not used, it is possible to estimate the population parameter by recognizing the pattern of the human body in the same manner as recognizing the pattern of the face.

FIG. 4 is a diagram showing an outline of processing executed by the computer system 101 of the first embodiment. When the computer system 101 detects the occurrence of a periodic event or an event, the computer system 101 acquires an image by using the image pickup apparatus 204 (step S401).

Next, the control module 211 of the computer system acquires a parameter for switching the first method or the second method for measuring the degree of interest from the management database 300 (step S402).

FIG. 5 illustrates the concept of switching between the first method and the second method. In this embodiment, the degree of interest is measured based on the face of the user facing a predetermined direction. In this case, the venue includes a large venue (for example, 300 m square) such as a stadium or a public hall, and a narrow venue (10 m square) such as a classroom or a conference room. In a large venue, the size of the face portion in the image captured by the image pickup apparatus 204 may be small, the number of faces may be large, the resolution may be insufficient, or the amount of processing may be too large. In this case, in the first method based on face pattern recognition, face recognition in a large venue cannot be performed, and measurement may be difficult or inaccurate. Therefore, in this embodiment, the first method and the second method are switched depending on the size of the venue. Here, the size can be rephrased as the size, the area, and the number of people that can be accommodated. Hereinafter, these are sometimes referred to as "scale".

As a parameter of the scale for this, for example, the number of people 303 and the number of seats 304 of the management database 300 can be used. For example, the threshold value is set to 300 (human) and compared with the parameters, and if it is less than the threshold value, the first method (face pattern recognition) is used, and if it is more than the threshold value, the second method (skin color area detection) is used (step S403).

That is, as shown in FIG. 4, if the parameter is equal to or greater than the threshold value, it is determined that the venue is large, and the skin color area of the face is calculated based on the RGB signal. This can be done by counting the number of pixels having a predetermined RGB value (step S404). Further, in order to calculate the number of face areas from the skin color area, the skin color area is divided by the area per face (step S405). The area per face is obtained by setting a predetermined value in advance or estimating from the number of people 303, the number of seats 304, and the imaging range of the imaging device 204.

In the above, the size of the venue is determined based on the number of people 303 and the number of seats 304, but other methods may be used. For example, data on the size of the venue may be stored in the management database 300 in advance, and the size of the venue may be compared with the threshold value. Alternatively, it may be set for each venue whether to use the first method or the second method. For example, in the example of FIG. 3, when the venue name 302 is the "meeting room", the first method may be used, and when the venue name 302 is the "exhibition hall", the second method may be used.

On the other hand, if the parameter is less than the threshold value, it is determined that the venue is narrow, and the number of face regions is calculated by pattern recognition (step S406). The control module 311 detects the face region from the received image. There is a known technique for the process of detecting a face region, and it is possible, for example, by collating a face pattern with a pattern in an image.

As a result of step S405 or S406, the number of people facing a specific direction, that is, the number of people paying attention can be calculated. The control module 311 counts the number of people of interest from the calculation result of the number of face regions (step S407).

Next, the control module 311 acquires the population parameter 303 and the number of seats 304 from the management database 300. Then, the degree of interest (%) is obtained by dividing the count result in step S407 by the parameter (step S408).

In the first embodiment, the case where the size of the venue is known in advance by using the information of the management database 300 is taken as an example. However, for example, the venue may be a temporary venue, and the size of the venue may be unknown in advance. In the second embodiment, an example in which this case can be dealt with will be described. The basic configuration is the same as that of the first embodiment, but it is assumed that the parameter indicating the size of the venue cannot be obtained from the data of the management database 300.

FIG. 6 is a diagram showing an outline of processing executed by the computer system 101 of the second embodiment. When the computer system 101 detects the occurrence of a periodic event or an event, the computer system 101 acquires an image by using the image pickup apparatus 204.

First, the control module 211 of the computer system calculates the resolution of the image using the received image (step S501).

Next, the upper limit value (threshold value) of the number of face detections is calculated (step S502). That is, in general, the detection of the number of faces by pattern recognition is limited by the processing speed, and if the number of faces is too large, counting may not be possible. This mainly depends on the amount of processing by the software. Therefore, if the upper limit is exceeded, the number of faces cannot be counted. At this time, the control module 311 automatically calculates the upper limit value from the image resolution calculated in step S501. Further, the upper limit value may be given as a specified value or manually.

The control module 311 detects a face region from the received image (step S503). There is a known technique for the process of detecting a face region, and it is possible, for example, by collating a reference face pattern with a pattern in an image.

The control module 311 determines whether or not the face region can be detected. That is, it is determined whether the pattern recognition is successful. If the venue is large and the size of the face in the image is too small, the resolution may be insufficient and pattern recognition may not be possible (step S504).

When it is determined that the face region cannot be detected, the control module 311 executes a process of calculating the skin color area of the face (step S404). In order to calculate the skin color area of the face, the area of the skin color area may be measured continuously or intermittently in time series. The control module 311 executes a process of calculating the number of face regions from the calculation result of the skin color area of the face (step S405). The control module 311 counts the number of people from the calculation result of the number of face areas (step S407). These processes are the same as in Example 1.

When it is determined in step S504 that the face region can be detected, the control module 311 executes the process of calculating the number of face regions from the detection result of the face region (step S406). The control module 311 determines whether or not the calculated number of face regions is equal to or greater than the threshold value during the counting of the number of faces (step S506). When the number of detected face areas exceeds the threshold value (upper limit), it is determined that the subsequent count cannot be performed or is inaccurate.

Therefore, when it is determined that the number of face regions is equal to or greater than the threshold value, the control module 311 executes a process of calculating the skin color area of the face (step S404). In order to calculate the skin color area of the face, the area of the skin color area may be measured continuously or intermittently in time series. The control module 311 executes a process of calculating the number of face regions from the calculation result of the skin color area of the face (step S405).

When it is determined that the number of face regions is less than the value, the control module 311 counts the number of people from the calculation result of the number of face regions (step S407).

When the computer system of Example 2 is applied to the mission-critical business system, even if there is no information about the venue, the number of people is counted based on whether or not the face can be detected in a narrow venue, and the skin color brightness in a large venue. By continuously or intermittently measuring the area of the area in time series, it becomes possible to determine whether or not the person is interested in the content of the lecture or the like even if the face is smaller than a certain size.

The third embodiment is different from the first embodiment in that the situation of the user who is watching the presentation or the lesson is fed back to the performer. Hereinafter, Example 3 will be described with a focus on the differences from Example 1. The configuration of the computer system of the third embodiment is the same as the configuration of the computer system of the first embodiment. The configurations of the computer system 101 and the mother brain computer system 102 of the third embodiment are the same as the configurations of the respective devices of the first embodiment. Further, the data structure of the database held by each device is the same as the data structure of the database of the first embodiment. Except for the part particularly described in Example 3, it may be considered to be the same as in Example 1 or 2.

FIG. 7 describes the gaze number calculation process of the third embodiment. The control module 311 acquires the number of users (total number of attendees) as a parameter from the management database 300 (step S701). Then, the number of people of interest is calculated in the same manner as in Example 1 or 2 by continuously or intermittently measuring in time series (step S407). Then, the degree of interest (attention) is calculated from the ratio of the population parameter and the calculated number of gazes (step S408).

Next, the population parameter, the number of people to pay attention to, and the degree of attention are displayed on the screen (step S702). Further, the feedback is not limited to being displayed on the screen, but may be fed back by voice such as an alarm.
Other processes are the same as in Example 1 or 2. However, in this embodiment, the number of people watching and the degree of interest are displayed in chronological order, and it is possible to know the change over time.

FIG. 8 shows an example of displaying the population parameter, the number of gazes, and the degree of attention on the screen 1010. The percentage of the number of people who are paying attention as an attention meter is displayed.

According to the third embodiment, since the situation of the user who is watching the presentation or the lesson is fed back to the performer in real time, it can be expected that the user's understanding level is improved by devising the explanation content by the performer.

The fourth embodiment differs from the first embodiment in that the situation of the user who is viewing the presentation or the product is recorded. Hereinafter, Example 4 will be described with a focus on the differences from Example 1.

The configuration of the computer system of the fourth embodiment is the same as the configuration of the computer system of the first embodiment. The configurations of the computer system 101 and the mother brain computer system 102 of the fourth embodiment are the same as the configurations of the respective devices of the fourth embodiment. Further, the data structure of the database held by each device is the same as the data structure of the database of the fourth embodiment.

The processing flow of the fourth embodiment will be described with reference to FIG. Except for the part particularly described in Example 4, it may be considered to be the same as in Example 1 or 2. First, the gaze number calculation process of Example 4 will be described. The control module 311 counts the number of people as a parameter (step S701).

Next, the target of interest is specified. For example, consider a case where slides are sequentially displayed on a screen in a lecture and the user's interest in each slide is measured. In that case, the attention target is set corresponding to the displayed slide. The object of interest is identified, for example, by the number of the currently displayed slide (step S1101). The object of interest may be electronic data such as presentation materials or the actual product.

The number of people of interest is calculated by continuously or intermittently measuring each object of interest in chronological order (step S407). The degree of interest is calculated from the calculated ratio between the number of people of interest and the population parameter for each object of interest (step S408). The calculated degree of interest for each object of interest is recorded in, for example, the image DB 212 (step S1104). Other processes are the same as in Example 1 or 2.

FIG. 10 is a conceptual diagram of the data recorded in the image DB. The degree of interest is recorded according to the number of the slide of interest. According to the fourth embodiment, since the degree of attention of the user who is viewing the presentation or the product is recorded, it can be expected that the material or the product having a high degree of attention will be displayed.

According to the above-described embodiment, in a narrow venue, the number of people is counted based on whether or not the face can be detected, and in a large venue, the area of the skin color luminance region is continuously or intermittently measured in chronological order. It is possible to judge whether or not you are interested in the content of a lecture, etc., even if the size is less than a certain size.

100 Interest level measurement system 101 Computer system 102 Mother Brain computer system 103 Network 104 Venue

Claims (11)

An interest level measurement system that measures the user's level of interest in an object of interest.
It is equipped with an arithmetic unit that obtains a color image from an image pickup device that captures the state of the user as an input.
The arithmetic unit is
A first mode that recognizes the user's face in the color image and counts the number of faces, and
A second mode in which the area of a pixel having a predetermined RGB value in the color image is calculated and the number of faces is estimated based on the area.
Is configured to be switchable,
Interest level measurement system. The arithmetic unit is
Using at least one of the size of the venue stored in the management database and the number of detected face areas as parameters,
Compare the input parameters with the threshold and
Switching between the first mode and the second mode based on the comparison result.
The degree of interest measurement system according to claim 1. The arithmetic unit is
Compare the input parameters with the threshold
The mode is switched between the first mode and the second mode based on the comparison result.
Information from the management database that stores information on at least one of the attention target, the size of the venue where the user is assumed to exist, and the number of the users who are assumed to be in the venue is input.
Switching between the first mode and the second mode using at least one piece of information on the size of the venue and the number of users as the parameters.
The degree of interest measurement system according to claim 1 . The arithmetic unit is
Based on the recognition result of the user's face in the color image,
Switching between the first mode and the second mode,
The degree of interest measurement system according to claim 1. The arithmetic unit is
Based on the count result of the number of faces of the user in the color image
Switching between the first mode and the second mode,
The degree of interest measurement system according to claim 1. It is a method of measuring the degree of interest of a user in a venue by using a computer equipped with an arithmetic unit, a storage device, an input device, and an output device.
A color image from an image pickup device that captures the state of the user is acquired by the input device, and the color image is acquired.
The arithmetic unit is
A first mode that recognizes the user's face in the color image as a pattern and calculates the degree of interest based on the recognition result.
A second mode in which the area of a predetermined color in the color image is calculated and the degree of interest is calculated based on the area.
Can be switched,
Interest level measurement method. The first mode and the second mode are
Selected based on the size of the venue,
The method for measuring the degree of interest according to claim 6. If the size of the venue is less than the predetermined threshold, the first mode is selected.
When the scale of the venue is equal to or larger than a predetermined threshold value, the second mode is selected.
The method for measuring the degree of interest according to claim 7. The first mode and the second mode are
Selected based on the user's face pattern recognition result.
The method for measuring the degree of interest according to claim 6. If the user's face pattern recognition is successful, the first mode is selected.
If the user's face pattern recognition is not successful, the second mode is selected.
The method for measuring the degree of interest according to claim 9. When the pattern recognition of the user's face is successful, the face area where the pattern is recognized is counted, the first mode is selected when the count result is less than the predetermined threshold value, and the first mode is selected when the count result is the predetermined threshold value or more. Select the second mode,
The interest level measuring method according to claim 9, wherein when the user's face pattern recognition is not successful, the second mode is selected.

Images