JP2006268395A - Information processor, information processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processor allowing looking-back upon contents including a feeling of a conference participant or the like. <P>SOLUTION: This information processor has: a synchronization part 8 synchronizing bioinformation of a person with video information; a state estimation processing part 10 finding a state estimation value showing a psychological state of the person by use of a prescribed evaluation function on the basis of the bioinformation of the person or voice information in time of photography; an index file storage part 12 storing the state estimation value correspondingly to a scene of a video; a retrieval means retrieving the scene of the video having the state estimation value according to movement of an object; and a display control part 16 displaying a screen wherein the object showing the person is disposed inside a coordinate space expressed by use of the psychological state of the person according to the state estimation value. Thereby, because the object showing the person is disposed inside the coordinate space expressed by use of the psychological state of the person, the contents including the feeling can be looked back upon. The psychological state can be converted into a distance and be disposed to express subjective information as positional information. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  Conventionally, decisions made at meetings and the like can be reviewed by reviewing the minutes issued later. However, since the details of the decision are not recorded, it is not possible to look back appropriately.

  A method of using a moving image has been proposed as a technology for supporting such reflection. For example, it is a method of supporting recall by taking a video of a meeting and playing back a scene that you want to remember later. For this purpose, technology for quickly searching for a desired scene is indispensable.

  For example, by presenting a scene (still image) extracted from a moving image at a certain time interval as a representative image to the user, the user guesses the temporal context based on the representative image and traces to the desired scene. Can arrive. However, in this case, the scene presented to the user is a scene cut out at regular intervals, and it is unclear how useful the scene is for time estimation. For example, when a plurality of similar scenes are extracted, it is difficult to estimate the temporal context from them.

  In order to solve such a problem, there is a method of presenting a still image to the user as a representative image when the content of the moving image changes greatly. In this case, unlike the representative image obtained at simple time intervals, the point in time when the scene changes is used as the representative image, so that similar scenes do not continuously become the representative image. However, in this method, there is a case where a representative image cannot be obtained because there are few changes in the image. Conversely, when the camera pans, blurs, zooms, or someone passes in front of the camera, the scene is considered to have changed significantly, and the scene may be presented as a representative image. They are useless because they are meaningless (unremembered) scenes for users looking back at the meeting.

  In addition, a method has been proposed in which which speaker speaks and when it is displayed, which is visually displayed as a timeline, and a part of the video to be reproduced is accessed from the timeline. However, the user needs to look at the displayed timeline and infer a scene that he / she wants to see where the scene he / she is looking for corresponds. For this reason, the longer the meeting is, the wider the viewing range becomes, and the user is burdened.

  In addition, there is a method in which a user searches a scene similar to the given image when the user gives the system a scene to be viewed as an image. In this case, the user gives an image as a search request, but in the case of recording a conference, it is necessary to assume the position of the installed camera and to create an image that can be seen from that position and give it to the system. Since the user usually does not consider the camera position or the like, it is difficult to create and provide an image from the camera position.

  In addition, the above method has a problem that it is impossible to detect a meaningful scene. The following are proposed as conventional techniques for solving such problems.

  For example, it detects the degree of interest of participants for multiple slides displayed sequentially at the beginning of the presentation, and keeps them in the form of a slide number and interest level table, corresponding to the slide number with the highest interest level. A method has been proposed in which a slide is taken out and displayed according to a slide progress pattern (see Patent Document 1).

  In addition, a method has been proposed in which the speech of the presenter is input at the time of presentation, the volume and inflection of the input speech is analyzed, and the screen during presentation is highlighted when the speech is high or the emphasis is high. (See Patent Document 2).

  In addition, a method in which the photographer determines the importance and tags the video (see Patent Document 3), a method in which the degree of interest is calculated from the facial skin temperature and the facial distance (see Patent Document 4), when the terminal is operated A method for supporting the operation by monitoring the operation status of the operator (see Patent Document 5) has been proposed.

JP 2002-149145 A JP 2002-23716 A JP 2000-197002 A Japanese Patent Laid-Open No. 2001-100808 JP-A-5-176325

  However, when the content of the conference including the emotions of the conference participants is later reviewed, the information related to emotions has few clues and is difficult to reflect. In remote conferences, the participants are not close to each other, so it is difficult to express subtle emotions (personal space problem).

  Further, in the apparatus described in Patent Document 1, the object is not the content itself but the slide, and at the time of a meeting or presentation, the participant looks at the speaker and the material while complementing each other. Because it is not connected, it is not possible to look back on the contents including the emotions of the participants. Also, the interest level of each participant cannot be determined.

  In the apparatus described in Patent Document 2, there are many situations where speakers are biased in a conference, and in a presentation, there are extremely few opportunities for viewers to speak, and it is difficult to specify a meaningful scene based on the speech volume. In the device described in Patent Document 3, the intention of the viewer cannot be reflected depending on the determination of one photographer. Furthermore, the photographer must intentionally perform tagging work.

  The device described in Patent Document 4 is an emotional input support method in a TV game or the like, and it is impossible to look back on an important scene by a link between the recording and the video. Depending on the device described in Patent Document 5, it is not possible to look back on the content including emotions.

  Therefore, the present invention has been made in view of the above-described problems, and an object thereof is to provide an information processing apparatus, an information processing method, and a program that can look back on the contents including the feelings of conference participants and the like.

  In order to solve the above problems, the present invention relates a person's mental state estimated using at least one of the biological information and audio information of the person obtained at the time of shooting to the video information at the time of shooting. An information processing apparatus comprising: storage means for storing; and display means for displaying a screen in which an object representing the person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis.

  According to the present invention, it is possible to look back on the contents including emotions by displaying a screen in which an object representing a person is arranged in a coordinate space in which the mental state of the person is at least one coordinate axis. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized. Moreover, it becomes easy to search later by remembering a mental state as a pattern by making a mental state into positional information. Here, the coordinate space includes a coordinate plane in the case of a two-dimensional space.

  The present invention further includes search means for searching the storage means for a video corresponding to the position of the object in the coordinate space, and the display means displays the video searched by the search means. According to the present invention, it is possible to intuitively search for a scene in a target psychological state using a person object as a search key.

  The display means displays a thumbnail of a screen on which the object is arranged in a coordinate space having the mental state of the person as at least one coordinate axis. As a result, the psychological state can be confirmed at a glance. The display means displays an image corresponding to the selected thumbnail. The display means displays the transition of the object. The coordinate space has a time axis as one coordinate axis. The present invention further includes estimation means for obtaining a state estimation value representing a degree of the mental state of the person using a predetermined evaluation function based on the biological information of the person or the sound information at the time of photographing.

  The present invention provides a storage means for storing a mental state of a person estimated using at least one of the biological information and audio information of the person obtained at the time of photographing in correspondence with the video information at the time of photographing, and the person And a display means for displaying a coordinate space having the mental state of the person as at least one coordinate axis, and a search means for searching a video corresponding to the position of the object in the coordinate space from the storage means. Information processing apparatus. According to the present invention, it is possible to intuitively search for a scene in a target psychological state by moving a person object. The mental state of the person includes the person's interest degree, person's excitement degree, person's comfort degree, person's understanding degree, person's memory degree, person's support degree, person's shared sense degree, person's subjectivity degree, person Including at least one of an objective degree, a person's dislike degree, and a person's fatigue degree.

  The present invention obtains a state estimated value indicating a mental state of a person calculated using at least one of the biological information and audio information of the person obtained at the time of shooting, and according to the state estimated value, And displaying a screen on which an object representing the person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis. According to the present invention, since a screen on which an object representing a person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis is displayed, the contents including emotion can be looked back on. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized. Moreover, it becomes easy to search later by remembering a mental state as a pattern by making a mental state into positional information.

  The present invention further includes a step of displaying a thumbnail of a screen on which an object representing the person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis. The present invention includes a step of displaying a screen arranged in a coordinate space having an object representing a person and a mental state of the person as at least one coordinate axis, and a video scene having a state estimated value corresponding to the position of the object Information processing method.

  The present invention includes a step of obtaining a state estimated value indicating a person's mental state calculated using at least one of the biological information and audio information of the person obtained at the time of photographing, according to the state estimated value, A program for causing a computer to execute a step of generating information for displaying a screen in which an object representing a person is arranged in a coordinate space having a mental state of the person as at least one coordinate axis. According to the present invention, since a screen on which an object representing a person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis is displayed, the contents including emotion can be looked back on. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized. Moreover, it becomes easy to search later by remembering a mental state as a pattern by making a mental state into positional information.

  According to the present invention, it is an object to provide an information processing apparatus, an information processing method, and a program capable of looking back on the contents including feelings of conference participants and the like.

  Hereinafter, the best mode for carrying out the present invention will be described. FIG. 1 is a block diagram of an information search system 1 in an embodiment of the present invention. As shown in FIG. 1, the information retrieval system 1 includes a biological information detection unit 2, a voice information detection unit 3, a conference video detection unit 4, a biological information storage unit 5, a voice information storage unit 6, a conference video storage unit 7, and a synchronization. Unit 8, conference video storage unit 9, state estimation processing unit 10, state rank determination unit 11, index file storage unit 12, search request input unit 13, search request recording unit 14, search unit 15, display control unit 16 and display unit 17.

  This information retrieval system 1 enables a desired scene to be retrieved from a conference video. The biometric information detection unit 2 detects information related to the eyeball such as blinks, pupil diameters, gaze targets, and gaze times of conference viewers, and biometric information such as facial skin temperature. It is preferable that this biometric information can be obtained without wearing a measuring device particularly for a conference viewer.

  The biological information detection unit 2 extracts, for example, information relating to blinking and pupil diameter from a face image obtained by capturing a viewer's face with a camera using a known method, further specifies an eye region, and blinks. Is obtained by counting the number of pupils or measuring the pupil diameter. In the gaze target and the gaze time, the eye area is identified by the above method using an image captured by a camera or the like arranged on the gaze target candidate side, the gaze target is identified from the position of the captured camera, and the captured eye area It can be acquired by specifying the gaze time from the shooting time. The facial skin temperature can be obtained without wearing a measuring device for the viewer by using an infrared camera, thermography or the like.

  The voice information detection unit 3 includes, for example, a sound collecting microphone and detects the voices of conference viewers and speakers. The conference video detection unit 4 is composed of a video camera or the like, and detects video information related to the conference. The conference video detection unit 4 may be a camera that can capture presentation materials and viewers used in a conference at a wide angle.

  The biological information storage unit 5 stores the biological information detected by the biological information detection unit 2 as a tabular data sheet. FIG. 2 is a diagram illustrating a data sheet stored in the biometric information storage unit 5. In FIG. 2, time t, pupil diameter x, y, gaze target, gaze time, blink, and facial skin temperature are biological information. This biometric information is stored for each viewer in association with time t. The gaze target is set as the gaze target by specifying the camera position that can be imaged from the camera installed on the gaze target side. The gaze time is calculated and recorded for each subject.

  The voice information storage unit 6 stores the voice detected by the voice information detection unit 22 as a tabular data sheet. FIG. 3 is a view showing a data sheet stored in the audio information storage unit 6. In FIG. 3, utterance presence / absence, volume, speaker utterance presence / absence, speaker volume, and environmental sound are voice information. This biometric information is stored for each viewer in association with time t. In particular, the environmental sound includes acoustic information generated accompanying the video.

  The conference video storage unit 7 stores the conference video detected by the conference video detection unit 4 as a data sheet in a list format. Next, the conference video stored in the conference video storage unit 7 will be described. FIG. 4 is a diagram illustrating a data sheet stored in the conference video storage unit 7. As shown in FIG. 4, the conference video is stored in correspondence with the identification symbol ID.

  The synchronization unit 8 synchronizes the biological information of the person and the audio information at the time of shooting with the video information. The conference information storage unit 9 is configured by a recording device such as a semiconductor memory, a hard disk, or a flexible disk, and holds information synchronized by the synchronization unit 8 as a conference index file. FIG. 5 is a view showing a data sheet stored in the conference information storage unit 9. As shown in FIG. 5, the conference information includes biological information such as time, pupil diameter, gaze target, gaze time, blink, and facial skin temperature, utterance presence / absence, volume, speaker utterance presence / absence, speaker volume, and environmental sound. Such as sound information and conference video information data ID. The meeting information is associated with the time for each viewer.

  The state estimation processing unit 10 obtains a state estimation value using a predetermined evaluation function based on the biological information and voice information of the person, and associates the obtained state estimation value with the scene ID of the conference video to store the index file storage unit. 12. In this way, the state estimation processing unit 10 assigns an index to the conference video. The state estimated value indicates a mental state of the person and is calculated using at least one of the biological information and voice information of the person obtained at the time of photographing.

  The mental state of the conference viewer includes, for example, the cognitive state and mental information of the conference viewer. Here, the mental state of the person includes the person's interest degree, person's excitement degree, person's comfort degree, person's understanding degree, person's memory degree, person's support degree, person's shared sense degree, person's subjectivity Degree, objective degree of person, degree of disgust of person, degree of fatigue of person, and the like. This evaluation function weights and adds the biological information of the viewer and the sound information obtained when the viewer is photographed. The function of the state estimation processing unit 10 is realized by executing a predetermined program.

  The state rank determination unit 11 ranks the viewer state based on the viewer state estimation value calculated by the state estimation processing unit 10 and performs a process of reducing the number of estimation results of the state estimation processing unit 10. Thus, by making the number of scenes appropriate as landmarks by rank determination, it is possible to prevent the number of candidates from becoming enormous and the search performance from deteriorating significantly.

  The index file storage unit 12 stores the estimated state value of the person associated with the time and the conference video ID as an index file. The index file storage unit 12 is stored, for example, in a tabular data sheet for each viewer.

  FIG. 6 is a diagram showing a data sheet stored in the index file storage unit 12. As shown in FIG. 6, items of the viewer's mental state such as time t, conference video information ID, interest level, excitement level, comfort level, understanding level, and memory level are provided from the left side. In the column of time t, times corresponding to the conference video information are respectively input. As the conference video information ID, a number for identifying the conference video information is input. In each column of the viewer's mental state, such as the degree of interest, the degree of excitement, the degree of comfort, the degree of understanding, and the degree of memory, state estimation values obtained by the evaluation function are input. In the figure, * indicates a scene deleted by the state order determination unit 11. This reduces the number of scenes to be searched. In the figure, the interest level, the excitement level, the comfort level, the understanding level, and the memory level are displayed as the viewer's mental state, but the viewer's mental state is not limited to these.

  The search request input unit 13 includes, for example, a touch panel, a mouse, or a keyboard, and the user can input a specific search request by operating the search request input unit 13. The user can specify, for example, a specific viewer from the search request input unit 13, and can further specify the mental state of the viewer such as a state in which the viewer is interested. The search request recording unit 14 is configured by a recording device such as a semiconductor memory, a hard disk, or a flexible disk, and records the search condition input by the search request input unit 13.

  Based on the search request input from the search request input unit 13, the search unit 15 stores a video scene having a state estimation value corresponding to the position of the object in the coordinate space (video corresponding to the position of the object) as an index file. Search from part 12. The display control unit 16 acquires the conference video data ID and the state estimated value from the index file storage unit 12, converts the state estimated value into position information, and uses the mental state of the person as at least one coordinate axis in the coordinate space. Displays a screen on which objects representing persons are arranged. At this time, the display control unit 16 acquires video information from the conference video storage unit 7 with reference to the conference video data ID, and displays the video on the display unit 17 in association with the object.

  Next, the operation of the state estimation processing unit 10 will be described. According to eye movement psychology (experimental psychology of eye movement, psychology of blinking, psychology of pupil movement), interest is related to pupil diameter, comprehension and memory are related to blink, and excitement Degree and comfort are considered to be related to blink and facial skin temperature. However, these simple substances cannot maintain accuracy such as an increase in facial skin temperature due to an indoor temperature environment. Therefore, the state is identified by an evaluation function obtained by weighting and adding these data, speech volume, environmental sound, and the like.

  Next, the evaluation function used in the state estimation processing unit 10 is shown below. The evaluation functions for obtaining state estimation values of person interest, person excitement, person comfort, person understanding, and person memory are as follows. Similarly, it is possible to obtain state estimation values of a person's support level, a person's shared sense level, a person's subjectivity level, a person's objectivity level, a person's disgust level, and a person's fatigue level.

Interest degree f1 = w11 * pupil diameter (change amount, change speed) + w12 * gaze (time, frequency) + W13 * blink (rate, frequency, cluster blink) + w14 * facial skin temperature change + w15 * speech volume + w16 * Speaker volume + w16 * Environmental sound (1)

Exciting degree f2 = w21 * pupil diameter (change amount, change speed) + w22 * gaze (time, frequency) + w23 * blink (rate, frequency, number of cluster blinks) + w24 * facial skin temperature change + w25 * speech volume + w26 * Speaker volume + w26 * Environmental sound (2)

Comfort level f3 = w31 * pupil diameter (change amount, change speed) + w32 * gaze (time, frequency) + w33 * blink (rate, frequency, cluster blink) + w34 * facial skin temperature change + w35 * speech volume + w36 * Speaker volume + w36 * Environmental sound (3)

・ Understanding f4 = w41 * pupil diameter (change amount, change speed) + w42 * gaze (time, number of times) + w43 * blink (rate, number of times, cluster blink number) + w44 * face skin temperature change amount + w45 * utterance volume + w46 * Speaker volume + w46 * Environmental sound (4)

Memory degree f5 = w51 * pupil diameter (change amount, change speed) + w52 * gaze (time, frequency) + w53 * blink (rate, frequency, cluster blink) + w54 * facial skin temperature change + w55 * speech volume + w56 * Speaker volume + w56 * Environmental sound (5)

  Next, the evaluation function will be described. In the degree of interest f in Formula 1, the video is specified as a scene of interest where the pupil variation is large, the gaze time is long, the number of blinks is small, the facial skin temperature variation is large, and the speech volume is large. be able to. Therefore, the degree of interest f in the expression (1) is the pupil diameter (change amount, change speed), gaze (time, number of times), blink (rate, number of times, number of cluster blinks), facial skin temperature change amount, speech volume. The estimated state value can be calculated by weighting and adding the speaker utterance volume, the environmental sound, and the like by weighting factors w11 to w1n, respectively.

  The degree of excitement f2 in equation (2) is the pupil diameter (change amount, change speed), gaze (time, number of times), blink (rate, number of times, number of cluster blinks), facial skin temperature change amount, speech volume, talk The estimated state value can be calculated by adding the weights w21 to w2n to the person's utterance volume, the environmental sound, and the like. The comfort level f3 in Equation (3) is the pupil diameter (change amount, change speed), gaze (time, number of times), blink (rate, number of times, number of cluster blinks), facial skin temperature change amount, speech volume, talk The estimated state value can be calculated by weighting and adding to each person's speech volume, environmental sound, etc. with weighting coefficients w31 to w3n.

  In the degree of understanding of equation (4), it is considered that blinking increases when it is difficult to understand (psychology of blinking). In general, when it is difficult to understand, viewers tend to acquire supplementary information and the gaze time tends to increase, so it is difficult to understand the part where the gaze time is long and the number of blinks is large. The video is specified as a certain scene. Therefore, the comprehension degree f4 of the equation (4) is the pupil diameter (change amount, change speed), gaze (time, number of times), blink (rate, number of times, number of cluster blinks), facial skin temperature change amount, speech The estimated state value can be calculated by weighting and adding the volume, speaker utterance volume, environmental sound, etc. with weighting factors w41 to w4n.

  In addition, the memory f5 of Expression (5) is the pupil diameter (change amount, change speed), gaze (time, number of times), blink (rate, number of times, number of cluster blinks), facial skin temperature change amount, speech volume. The estimated state value can be calculated by weighting the speaker utterance volume, the environmental sound, and the like with weighting coefficients w51 to w5n and adding them.

  FIG. 7 is a diagram for explaining detection and recording of biometric information, audio information, and conference video information. Reference numeral 200 indicates a conference room. In the example illustrated in FIG. 7, the above-described biological information detection unit 2 includes infrared cameras 211 and 212. Thereby, for example, when the viewer A gazes at the slide display unit 205, the eyeball image is detected by the infrared camera 211. When the speaker 206 is watched, it is picked up by the infrared camera 212. Moreover, the audio | voice information detection part 3 mentioned above is comprised by the sound pressure sensor and the directional microphone 202, for example. Thereby, for example, the utterance information of the viewer A is detected. The conference video information detection unit 4 includes a conference video detection camera 203.

  FIG. 8 is an operation flowchart of the index assigning process. In step S <b> 101, the state estimation processing unit 10 reads the conference video index file in the conference information storage unit 9. In step S102, the state estimation processing unit 10 reads out the parameters of the conference video index file for each scene ID, calculates the estimated state value of the person using the evaluation function, and uses the estimated state value of the person as the index file storage unit 12. To store. In step S103, the state ranking determination unit 11 ranks the viewer state based on the viewer state estimation value calculated by the state estimation processing unit 10, and reduces the number of estimation results of the state estimation processing unit 10. Process. Thus, by making the number of scenes appropriate as landmarks by rank determination, it is possible to prevent the number of candidates from becoming enormous and the search performance from deteriorating significantly. In this way, an index corresponding to the mental state of the person can be added to the video information. Note that the state order determination process in step S103 may be selectively performed.

  Next, display processing will be described. FIG. 9 is an operation flowchart of the display process. The user operates the search request input unit 13 to input an interest level and an understanding level as search conditions as the viewer's mental state. In step S201, the state order determination unit 11 refers to the landmark selection criteria based on the above-described evaluation function, and refers to, for example, the top five places. In step S202, the conference video ID is acquired by referring to the index file in FIG. 6 based on the information in step S1.

  In step S203, the display control unit 16 performs a coordinate space selection process. That is, the display control unit 16 creates the coordinate space in FIG. 10 based on the request information obtained by the search request input unit 13, for example, a request to search for the degree of interest and the degree of understanding. In step S204, the display control unit 16 refers to each evaluation function for the selected scene ID for each conference participant and obtains numerical values of the degree of interest and the degree of understanding. In step S205, the display control unit 16 places the object in the coordinate space of FIG. 10 based on the numerical values of FIG.

  FIG. 10 is a diagram showing a user interface in the present embodiment. Reference numeral 21 denotes a conference video timeline, 22 denotes a conference video, and 23 denotes a search result screen. The user operates the search request input unit 13 to input an interest level and an understanding level as search conditions as the viewer's mental state. Based on the search condition from the search request input unit 13, the search unit 15 acquires the interest level and the understanding level estimated value in the index file storage unit 12 and the corresponding conference video information ID, and sends them to the display control unit 16. send. The display control unit 16 arranges the objects (avatars) A to D representing the viewer (person) in a coordinate plane having the mental state (degree of understanding and interest) of the person as coordinate axes according to the state estimation value. The search result screen 23 is displayed. Thereby, a person with low understanding and low interest is arranged in the lower left, a person with high understanding and high interest is arranged in the upper right, and the subjective state can be grasped at a glance. The avatar is a doll icon that schematically represents the individual arranged in FIG. 10, and the expression format may be, for example, an actual person image or a figure such as ○ or □.

  Referring to the search result screen 23, the viewer A is interested but not understood, the viewer B is not interested or understood, the viewer C is interested and understood, and the viewer D is I understand that I understand but I am not interested. In this way, by displaying a screen in which an object representing a person is arranged in a coordinate plane having the mental state of the person as at least one coordinate axis, it is possible to look back on the content of the conference including the viewer's emotions. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized. In addition, it becomes a clue to search for a time zone when the mental state is changed.

  FIG. 11 is a diagram illustrating another example of the user interface in the present embodiment. Reference numeral 21 denotes a conference video timeline, 22 a conference video, and 24 a search result screen 24. The user operates the search request input unit 13 to input the degree of concentration and the comfort level as search conditions as the mental state of the viewer. Based on the search condition from the search request input unit 13, the search unit 15 acquires the concentration level and comfort level estimated values in the index file storage unit 12 and the corresponding conference video information ID, and sends them to the display control unit 16. send. The display control unit 16 arranges the objects (avatars) A to D representing the viewer (person) in a coordinate plane having the mental state (concentration level, comfort level) of the person as coordinate axes according to the state estimation value. The search result screen 23 is displayed.

  Referring to the search result screen 24, the viewer A is less concentrated, the viewer B is not comfortable, the concentration is low, the viewer C is comfortable and high, and the viewer D is concentrated. It turns out that it is not comfortable. In this way, by displaying a screen in which an object representing a person is arranged in a coordinate plane having the mental state of the person as at least one coordinate axis, it is possible to look back on the content of the conference including the viewer's emotions. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized.

  FIG. 12 is a diagram illustrating another example of the user interface in the present embodiment. Reference numeral 21 denotes a conference video timeline, 22 a conference video, and 25 a search result screen 25. The coordinates indicate the viewer X (degree of understanding, degree of interest, degree of comfort). The user operates the search request input unit 13 to input the understanding level, the interest level, and the comfort level as search conditions as the viewer's mental state. Based on the search condition from the search request input unit 13, the search unit 15 acquires the understanding level, the interest level and the comfort level state estimation value in the index file storage unit 12 and the corresponding conference video information ID, and performs display control. Send to part 16. In accordance with the state estimation value, the display control unit 16 uses objects (avatars) A to D representing viewers (persons) in a coordinate space with a person's mental state (degree of understanding, interest, and comfort) as coordinate axes. The search result screen 23 arranged in is displayed. Here, the coordinate space may have a time axis as one coordinate axis.

  Referring to the search result screen 25, it can be seen that the viewer A is interested and comfortable, but does not understand. It is understood that the viewer B is not interested, does not understand, and is not in a comfortable state. It is understood that the viewer C is interested and understands and is in a comfortable state. Viewer D is interested and understands, but understands that it is not comfortable. In this way, by displaying a screen in which an object representing a person is arranged in a coordinate space in which the mental state of the person is at least one coordinate axis, it is possible to look back on the content of the conference including the viewer's emotions. Moreover, subjective information can be expressed as position information by converting a mental state into a distance. That is, the personal space can be visualized.

  FIG. 13 is a diagram illustrating another example of the user interface in the present embodiment. Reference numeral 21 denotes a conference video timeline, 22 denotes a conference video, and 26 denotes a search result screen. The user operates the search request input unit 13 to input the degree of interest as a search condition as the viewer's mental state. Based on the search condition from the search request input unit 13, the search unit 15 acquires the state of interest state estimated value and the corresponding conference video information ID in the index file storage unit 12, and sends them to the display control unit 16. The display control unit 16 arranges objects (avatars) A and B representing viewers (persons) in a coordinate space with the mental state (degree of interest) of the person as a coordinate axis according to the state estimation value. 26 is displayed. The coordinate plane has a time axis as one coordinate axis.

  Referring to the search result screen 26, it can be seen that the viewer A has the highest degree of interest in (2) and the degree of interest after that is low. Viewer B was initially less interested, but when (2), the degree of interest slightly increased, but after that, the degree of interest decreased, and when (5), the degree of interest increased greatly. Thus, by displaying the transition of the object, it is possible to grasp the change of the mental state.

  FIG. 14 is a diagram illustrating a representation example of a search result as a thumbnail. Reference numeral 21 is a meeting video timeline, 22 is a meeting video, and 271 to 274 are search result screen thumbnails. The user operates the search request input unit 13 to input an interest level and an understanding level as search conditions as the viewer's mental state. Based on the search condition from the search request input unit 13, the search unit 15 acquires the interest level and the understanding level estimated value in the index file storage unit 12 and the corresponding conference video information ID, and sends them to the display control unit 16. send. The display control unit 16 displays the thumbnails 271 to 274 of the search result screen in which the objects A to D representing the viewer are arranged in the coordinate plane having the mental state (degree of understanding and interest) of the person as at least one coordinate axis. To do. Here, a coordinate space may be used instead of the coordinate plane.

  By referring to the thumbnail 272 of the search result, it can be seen that the understanding level of the viewer A has increased, and by referring to the thumbnail 273 of the search result, it has been understood that the interest level of the viewer D has increased. With reference to 274, it can be seen that the understanding level of the viewer A decreases and the interest level of the viewer D decreases. By displaying the search results as thumbnails in this way, it is possible to check the psychological state at a glance. You can immediately find out where you were interested. Moreover, since the psychological state is used as position information, it can be remembered as a pattern. Therefore, it becomes easy to search later, and it becomes possible to provide a clue to search for a time zone in which a person has moved (a psychological state has changed). When the thumbnail is clicked, the display control unit 16 links to the original video and displays the video of that portion.

  FIG. 15 is a diagram for explaining a search user interface. Reference numeral 21 is a conference video timeline, 28 is a search user interface, 291 and 292 are search result thumbnails, and 30 is a conference video. The display control unit 16 displays an object representing a person in the participant area 281 as the search user interface 28, and displays a screen 282 arranged in a coordinate plane having the mental state of the person as at least one coordinate axis. Here, not a coordinate plane but a coordinate space may be used. The user selects an arbitrary viewer in the participant area 281 and moves it to the coordinate space 282. In the figure, an object indicating the viewer B is arranged at a position where the degree of interest and the degree of understanding are low.

  The display control unit 16 sends the position information of the object of the viewer B in the coordinate space 282 to the search unit 15. The search unit 15 searches the index file storage unit 12 for a video scene ID having a state estimation value corresponding to the position of the viewer B object in the coordinate plane, and sends the video scene ID to the display control unit 16. The display control unit 16 acquires a video scene corresponding to the video scene ID from the conference video storage unit 7 and displays the conference video scene. In addition, the display control unit 16 displays thumbnails 291 and 292 of search results suitable for the video scene 30. In this way, by regarding the participant's object (avatar) as a search key and arranging it in the two-dimensional virtual conference, it is possible to intuitively search for a target psychological state. In addition, by moving the object, a change in the psychological state is shown, and the original video in that state can be quickly accessed. The search result is not presented at the end of arranging all the conference participants, but takes a form in which candidate videos are displayed as thumbnails during the arrangement.

  FIG. 16 is a diagram for explaining an example in which the present invention is applied to a remote conference. Reference numeral 21 is a timeline of the conference video, 30 is the conference video, and 311 and 312 are thumbnails of the search results. The user operates the search request input unit 13 to set the understanding level and the interest level as the viewer's mental state, and the conference participants A and B participating in the conference at the remote location X as the conference participants, The conference participants C and D participating in the conference at the ground Y are input as search conditions. Based on the search condition from the search request input unit 13, the search unit 15 acquires the understanding level and the interest level estimated value in the index file storage unit 12 and the corresponding conference video information ID, and sends them to the display control unit 16. send. The display control unit 16 displays a search result screen in which objects A to D representing participants in remote locations X and Y are arranged in a coordinate plane having a person's mental state (degree of understanding and interest) as at least one coordinate axis. Thumbnail 311 and 312 are displayed.

  A person may express a psychological state depending on the distance to others (social psychology: personal space). However, in a remote conference, the physical distance is so far that it cannot be expressed. Therefore, in the conventional technology, in the remote conference at the remote location X and the remote location Y, the psychological state of the conference participants A to D cannot be expressed. Therefore, by applying the present invention to a virtual conference room, it is possible to express a psychological state even in a remote conference by arranging objects in consideration of the psychological state.

  As described above, the psychological state of the person is specified using the biological information, and this psychological state is converted into a distance and placed in the virtual conference space. In addition, avatar position information is used as a search key for searching by subjective information. Thereby, for example, the virtual conference space is displayed in two dimensions of the understanding level and the interest level, and by placing the object on the coordinates, the understanding level is low, and the person with low interest is in the lower left, the understanding level is high, Persons with high interest are placed in the upper right, and their subjective state can be grasped at a glance. Subjective information during the meeting is expressed as position information, supports pattern recognition, and can be a landmark for storage. Moreover, the subjective information during the meeting can be expressed as position information, and a personal space can be visualized and provided.

  The information retrieval system 1 is realized using a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk device, a display device, and the like. The information processing method according to the present invention is realized by the information search system 1, and the program is installed from a hard disk device, a portable storage medium such as a CD-ROM, a DVD, or a flexible disk, or downloaded from a communication circuit, Each step is realized by the CPU executing this program.

  The program obtains a state estimated value indicating the mental state of the person calculated using at least one of the biological information and voice information of the person obtained at the time of shooting, and the person is selected according to the state estimated value. The computer is caused to execute a step of displaying a screen in which an object representing the person is arranged in a coordinate space having at least one coordinate axis as the mental state of. The information search system 1 corresponds to the information processing apparatus, the index file storage unit 12 corresponds to the storage unit, the display control unit 16 corresponds to the display unit, the search unit 15 corresponds to the search unit, and the state estimation processing unit 10 corresponds to the estimation unit. To do.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed. For example, in each of the embodiments described above, the conference video is used as the video, but the video of the present invention is not limited to the conference video.

It is a block diagram of the information search system in the embodiment of the present invention. It is a figure which shows the data sheet stored in the biometric information storage part. It is a figure which shows the data sheet stored in the audio | voice information storage part. It is a figure which shows the data sheet stored in the meeting image | video storage part. It is a figure which shows the data sheet stored in the meeting information storage part. It is a figure which shows the data sheet which an index file storage part stores. It is a figure for demonstrating the detection and recording of biometric information, audio | voice information, and conference video information. It is an operation | movement flowchart of an index provision process. It is an operation | movement flowchart of a display process. It is a figure which shows the user interface in this embodiment. It is a figure which shows the other example of the user interface in this embodiment. It is a figure which shows the other example of the user interface in this embodiment. It is a figure which shows the other example of the user interface in this embodiment. It is a figure which shows the example of a display by the thumbnail of a search result. It is a figure for demonstrating the user interface for a search. It is a figure explaining the example which applied this invention by the remote conference.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information search system 11 State order | rank determination part 2 Biometric information detection part 12 Index file storage part 3 Audio | voice information detection part 13 Search request input part 4 Conference image | video detection part 15 Search part 8 Synchronization part 16 Display control part 9 Conference information storage part 17 Display unit 10 State estimation processing unit

Claims (13)

Storage means for storing the mental state of the person estimated using at least one of the biological information and audio information of the person obtained at the time of shooting in association with the video information at the time of shooting;
An information processing apparatus comprising: a display unit configured to display a screen in which an object representing the person is arranged in a coordinate space in which the mental state of the person is at least one coordinate axis. Search means for searching the storage means for a video according to the position of the object in the coordinate space,
The information processing apparatus according to claim 1, wherein the display unit displays the video searched by the search unit. The information processing apparatus according to claim 1, wherein the display unit displays a thumbnail of a screen on which the object is arranged in a coordinate space having the mental state of the person as at least one coordinate axis. The information processing apparatus according to claim 3, wherein the display unit displays an image corresponding to the selected thumbnail. The information processing apparatus according to claim 1, wherein the display unit displays a transition of the object. The information processing apparatus according to claim 1, wherein the coordinate space has a time axis as one coordinate axis. The apparatus further comprises an estimation unit that obtains a state estimation value representing a degree of the mental state of the person using a predetermined evaluation function based on the biological information of the person or the voice information at the time of shooting. The information processing apparatus according to any one of claims 1 to 6. Storage means for storing the mental state of the person estimated using at least one of the biological information and audio information of the person obtained at the time of shooting in association with the video information at the time of shooting;
Display means for displaying a coordinate space having the object representing the person and the mental state of the person as at least one coordinate axis;
An information processing apparatus comprising: a search unit that searches the storage unit for a video corresponding to the position of the object in the coordinate space. The mental state of the person includes the person's interest degree, person's excitement degree, person's comfort degree, person's understanding degree, person's memory degree, person's support degree, person's shared sense degree, person's subjectivity degree, person The information processing apparatus according to claim 1, wherein the information processing apparatus includes at least one of an objectivity level, a person's disgust level, and a person's fatigue level. Obtaining a state estimation value indicating the mental state of the person calculated using at least one of the biological information and voice information of the person obtained at the time of shooting;
And displaying a screen on which an object representing the person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis in accordance with the state estimation value. The information processing method according to claim 10, further comprising: displaying a thumbnail of a screen on which an object representing the person is arranged in a coordinate space in which the mental state of the person is at least one coordinate axis. Displaying a screen arranged in a coordinate space having an object representing a person and a mental state of the person as at least one coordinate axis;
And a step of searching for a video scene having a state estimated value corresponding to the position of the object. Obtaining a state estimation value indicating the mental state of the person calculated using at least one of the biological information and voice information of the person obtained at the time of shooting;
In order to cause the computer to execute a step of generating information for displaying a screen in which an object representing the person is arranged in a coordinate space having the mental state of the person as at least one coordinate axis according to the state estimation value Program.

Images