JP5937829B2 - Viewing situation recognition device and viewing situation recognition program - Google Patents

Description

  The present invention relates to a viewing status recognition device and a viewing status recognition program, and more particularly to a viewing status recognition device and a viewing status recognition program for recognizing a viewer's viewing status with high accuracy.

  Conventionally, there is a technique for recognizing and estimating the viewing situation. Such a technique is used, for example, in an information providing system that performs advanced program recommendation function of television, effective presentation of information related to the content being viewed, and the like.

  As a technique used in the above-described system, for example, there is a method of recognizing a viewing situation by analyzing an operation history of a device (see, for example, Patent Document 1 and Patent Document 2). In the technique disclosed in Patent Document 1, viewing status recognition output is performed based on whether or not the operation information of the remote controller of the device matches the rule determined by the system designer. The technique disclosed in Patent Document 2 handles a wide range of events such as a PC (personal computer), a mobile phone, a car navigation system, and home appliances instead of remote control operation for input information.

  Conventionally, there is known a method for estimating a viewing situation by analyzing data obtained by observing a user's behavior such as an image or sound (for example, see Patent Document 3). In the method disclosed in Patent Document 3, the viewing situation is estimated through image recognition processing such as facial expression recognition, gaze detection, emotion expression detection, and voice recognition processing from video and audio data observed with a camera or a microphone. .

  Further, conventionally, data obtained from a sensor such as a viewer's image and sound and the interaction data directly input by the viewer such as operation history and feedback input information to the system are integrated to estimate the viewing situation. A technique is known (see, for example, Patent Document 4). In the method disclosed in Patent Document 4, the data obtained from the sensor is called reaction data, the interaction data is called viewing data, and a linear sum output in which the weighting variables of the respective signals are changed in synchronization with each other in time. Is output as a result of situation recognition.

JP 2002-369090 A JP 2005-190421 A JP 2006-260275 A JP 2005-142975 A

  However, in the method as shown in Patent Document 1, since the viewing status is recognized and output based on whether or not the operation information of the remote controller matches the rule determined by the system designer, this is the basis for the determination. Rules tend to be subjective. In addition, the technique disclosed in Patent Document 1 is based on a rule-based determination, and thus there is a problem that it is difficult to deal with complicated events and at the same time, the expandability is poor.

  In addition, in the technique as shown in Patent Document 2, a wider range of events is handled as information input. However, since situation recognition is collated with a rule, the rule base described above is used. Has a problem associated with judgment. In addition, since both Patent Document 1 and Patent Document 2 are processed in units of events, there is a problem that the temporal relationship with time-series continuous events such as TV viewing becomes ambiguous. Furthermore, both Patent Literature 1 and Patent Literature 2 are processing only with data that can be input to the system consciously by the user, such as device operation history and behavior history.

  Further, the technique as disclosed in Patent Document 3 does not process the recognition output result, but assigns a viewing situation ID preset by the system designer, and the type of viewing situation becomes too large. There are challenges. Further, in the method as disclosed in Patent Document 3, only the image recognition and voice recognition results are used, and therefore high-accuracy recognition processing cannot be realized.

  Furthermore, in the method as disclosed in Patent Document 4, temporal synchronization is performed by a time stamp associated with the integrated signal, and a synchronization method that takes into account the temporal deviation of individual signals that are input. It's hard to say. In addition, the technique as shown in Patent Document 4 is composed of an integration method itself, input reaction data, weighting variables set in an ad hoc manner for viewing data, and threshold processing. It is not an objective method considering the nature of the input data, but often depends on the subjectivity of the system designer.

  In other words, in the conventional method, the situation recognition is performed using the rules determined by the system designer's subjectivity centering on the rule base, and it becomes a response in ad hoc program design.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide a viewing status recognition apparatus and a viewing status recognition program for recognizing a viewer's viewing status with high accuracy.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

The present invention relates to a histogram generation means for generating histogram data from a plurality of different data obtained as the viewer information in a viewing status recognition apparatus for recognizing a viewing status based on viewer information obtained when the viewer views the program. Synchronizing the plurality of different data using representative value selection means for selecting a representative value for each unit time from histogram data obtained by the histogram generation means, and representative values obtained by the representative value selection means Synchronization symbol sequence generation means for generating a symbol sequence, conversion means for converting a synchronization symbol sequence obtained by the synchronization symbol sequence generation means into a time-series feature vector signal using a preset feature function, learning path showing the time series feature vector signal obtained by converting means, the weighting previously set Integrated with the meter, and having a recognizing integrating means the viewing condition of the viewer.

According to the present invention, in a viewing situation recognition program for recognizing a viewing situation based on viewer information obtained when a viewer views a program, the computer generates histogram data from a plurality of different data obtained as the viewer information. Histogram generation means, representative value selection means for selecting a representative value for each unit time from histogram data obtained by the histogram generation means, and the representative value obtained by the representative value selection means to synchronize the plurality of different data. A synchronization symbol sequence generating means for generating a synchronized symbol sequence, a conversion means for converting a synchronization symbol sequence obtained by the synchronization symbol sequence generation means into a time series feature vector signal using a preset feature function, and The time-series feature vector signal obtained by the conversion means is set in advance. Integrated with the learning parameter indicating the weighting, to function as recognizing integrating means the viewing condition of the viewer.

  According to the present invention, a viewer's viewing situation can be recognized with high accuracy.

It is a figure which shows an example of a function structure of the viewing condition recognition apparatus in this embodiment. It is a flowchart which shows an example of the learning process in this embodiment. It is a flowchart which shows an example of the recognition process in this embodiment. It is a figure for demonstrating the flow of the recognition process using the machine learning process in this embodiment. It is a figure for demonstrating the production | generation procedure of the time-sequential feature vector signal using several different data. It is a figure for demonstrating the production | generation example of learning data. It is a figure which shows an example of a logical function group. It is a figure which shows an example of a function structure of the viewing condition recognition apparatus in other embodiment. It is a figure for demonstrating the specific content of other embodiment. It is a figure for demonstrating the correspondence of a viewing condition label and foot operation.

<About the present invention>
The present invention, for example, in a method for recognizing a viewer's viewing situation, statistically integrates a plurality of data having different properties related to viewing situation recognition, including their respective interactions, and effectively recognizes the viewing situation. . In addition, the present invention constructs a highly scalable mechanism that can objectively and automatically capture the nature of data related to viewing situation recognition, regardless of the subjectivity of the system designer.

  Here, the type of data related to viewing status recognition varies depending on the viewing environment for each viewer. Therefore, in the present invention, in the viewing situation recognition method, a mechanism that can easily change the mechanism of the recognition process is constructed. For example, in the viewing situation recognition method of the present invention, first, a learning parameter for recognition is obtained by learning processing using correct answer data of a viewing situation obtained in advance in an environment close to the real environment, and then the learning parameter is used. A viewing state recognition process is performed based on an arbitrary input signal for each viewer, and a viewing state corresponding to the recognition result is output.

  Specifically, in the present invention, for example, a plurality of time series heterogeneous signals having different time units (time resolutions) are symbolized in consideration of the characteristics of the signals for each unit time, and the time series symbols are unified. Convert to vector sequence signal. In addition, the present invention provides a plurality of feature functions that describe the interaction between each element of the symbol vector sequence generated in the above process, and outputs each function as a new time-series feature vector signal for machine learning input. Signal. Here, the input variable of the above-described feature function is configured to be able to take an arbitrary element at an arbitrary time in the time-series symbol vector sequence obtained above, for example.

  In the machine learning process described above, the learning process is performed using the time series feature vector signal described above as the input signal and the target viewing situation as the output signal, and the relationship between the input and the output is formulated and obtained from the learning result. Store learning parameters, etc. Also, in the present invention, the learning parameters and the like are applied to an unknown time-series feature vector signal obtained by the same processing as described above from an arbitrary input signal for each viewer, and the values of the viewing situation are integrated. decide. Thereby, in this invention, the interest degree etc. with respect to a television program of a viewer can be acquired with high precision, for example.

  Note that, in the present invention, when learning data is collected in the learning process, for example, foot data is collected in real time and with high accuracy while collecting the interaction data in a natural environment while collecting the correct data of the viewing situation. It is also possible to label the correct viewing situation while setting the interest level and type (direction) of interest for viewing using foot operation means such as a pedal.

  In the following, a preferred embodiment of the viewing situation recognition apparatus and viewing situation recognition program according to the present invention having the above-described features will be described in detail with reference to the drawings.

<Viewing situation recognition device: functional configuration example>
A functional configuration example of the viewing status recognition apparatus according to the present embodiment will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a functional configuration of a viewing status recognition apparatus according to the present embodiment. 1 includes an information acquisition unit 11, an analysis unit 12, a histogram generation unit 13, a representative value selection unit 14, a synchronization symbol string generation unit 15, a conversion unit 16, and a feature function. The group database 17, the statistical learning unit 18, the learning database 19, the learning parameter storage unit 20, and the integration unit 21 are configured.

  The information acquisition unit 11 acquires an input signal for learning and an arbitrary input signal for recognition for recognizing a viewing situation for each viewer. The input signal is acquired as viewer information. Examples of input signals include observation information such as camera video from a camera (imaging means) or sound information from sound acquisition means such as a microphone, operation by a remote operation means (hereinafter referred to as “remote control”) for a television or the like. There is heterogeneous data such as operation history information.

  In the present embodiment, the input signal is not limited to the type described above. For example, a portable information terminal (for example, a remote control for a television or the like, an Internet browsing function, a mail transmission / reception function, etc.) Operation information or the like of a smartphone, a tablet terminal, a notebook PC, a mobile phone, a game machine, or the like can be used as an input signal.

  Here, it is preferable that time information is given to each of the input signals described above. However, the present invention is not limited to this. For example, when the information acquisition unit 11 acquires each input signal. Thus, the time information may be added to the input signal. The information acquisition unit 11 outputs the above-described one or more input signals to the analysis unit 12.

  The analysis unit 12 analyzes each input signal obtained by the information acquisition unit 11. Specifically, when acquiring the camera image, the analysis unit 12 analyzes at least one of the viewer's face recognition, face orientation, facial expression, posture, action (motion), and the like. Note that the viewer in the present embodiment corresponds to an experimenter or administrator for learning correct answer data, for example, when the learning process is performed in the viewing status recognition device 10, and the recognition process is performed. If so, a general audience or the like is applicable.

  For example, with regard to the above-described face orientation, posture, and the like, for example, by matching template information of a plurality of faces set in advance to a face detection area from a camera video or the like, When there is, it recognizes which direction the face is facing. As for the above-described facial expressions, for example, the degree of change in the facial expression of the viewer when watching TV is detected from a camera image or the like, and whether or not the facial expression has changed is output based on a preset threshold or the like. Further, with respect to the above-described behavior, for example, it is determined whether the viewer is moving or stationary when viewing the television from the camera image, and the result is binary output. The determination is that the viewer is stationary when the amount of time difference in the viewer's video is small over a certain period of time.

  Further, the analysis means 12 classifies operation information such as switching programs or selecting recommended programs with a remote controller or a portable information terminal and outputs the information as symbol information. In addition, about the analysis method mentioned above, it is not limited to this, You may acquire the same information using another method.

  Here, for example, when an input signal is acquired from a remote controller, a portable information terminal, or the like, the analysis unit 12 determines whether the viewer's face is facing the direction of the remote controller or the portable information terminal from the camera image. Is analyzed. In other words, the analysis unit 12 performs analysis that incorporates each interaction using a plurality of data having different properties. The analysis unit 12 can also analyze the ambient environment of the viewer (for example, night, noon, etc.) by turning on / off the brightness information of the camera video and the power of the lighting device included in the video.

  In addition, when the voice information is acquired, the analysis unit 12 performs, for example, voice analysis such as sound intensity and sentence (word) extraction, morphological analysis on sentences obtained by sound tone analysis, voice analysis, and the like. , Syntactic analysis, semantic analysis, etc., for example, at least one of the meaning of words spoken by the viewer, emotion information by recognition of laughter or cry, personality information by recognition of speaking speed, loudness, etc. Analyze one. The analysis unit 12 can also analyze the surrounding environment (for example, presence or absence of noise) of the viewer.

  The analysis unit 12 acquires information such as channel switching content and volume adjustment from operation history information obtained from a remote controller or the like.

  Further, the analysis unit 12 analyzes, for example, whether or not browsing on the Internet, whether or not mailing, the contents of television operation, and the like from the operation information obtained from the portable information terminal. Note that the analysis unit 12 may output a corresponding analysis result by referring to, for example, preset table information based on operation history information, operation information, or the like of a remote controller or a portable information terminal.

  That is, the analysis unit 12 generates a time-series symbol sequence for each input signal by the above-described analysis, and outputs the information to the histogram generation unit 13.

  The histogram generation means 13 generates histogram data corresponding to each information for each preset unit time based on one or a plurality of data having different properties obtained by the analysis means 12, and quantifies the input signal ( (Numericalization).

  Specifically, when there are a plurality of different pieces of data as viewing information, the histogram generation unit 13 generates histogram data representing the frequency of the different data in a preset time unit. As a result, in the subsequent processing, the symbol strings synchronized in time can be changed and handled uniformly.

  Further, the histogram generation means 13 may set a plurality of unit times in advance and generate histogram data for each set unit time. Thereby, in the present embodiment, it is possible to perform a viewing situation learning process or a recognition process in consideration of an interaction between data expressed by symbol sequences having different unit times. The histogram generation unit 13 outputs the generated histogram data to the representative value selection unit 14 and the synchronization symbol string generation unit 15.

  The representative value selection unit 14 selects a representative value based on the histogram data obtained by the histogram generation unit 13. Note that the representative value in the present embodiment may be, for example, a value of maximum frequency (maximum value) or minimum frequency (minimum value) of histogram data for the same data, but is not limited thereto. It may be a total value or an average value of histogram data.

  Specifically, for example, when the input signal is a symbol string indicating the type of operation such as program channel switching or volume adjustment, the symbol of the operation most frequently performed in that unit time is the representative value output. The representative value selection unit 14 outputs the selected representative value to the synchronization symbol string generation unit 15.

  The synchronization symbol string generation unit 15 generates a synchronization symbol string based on the histogram data obtained by the histogram generation unit 13 and the representative value data obtained by the representative value selection unit 14. A plurality of different data (elements) acquired when the viewer is watching a television program or the like has different reference time intervals. Therefore, the synchronization symbol string generation means 15 generates a symbol string of representative values obtained by synchronizing each data in order to synchronize the time between the data. In addition, the synchronization symbol string generation unit 15 outputs the generated synchronization symbol string to the conversion unit 16.

  The conversion unit 16 performs a conversion process of the symbol sequence obtained by the synchronous symbol sequence generation unit 15 based on one or a plurality of feature functions stored in advance in the feature function group database 17, and a synchronized time-series feature vector signal Is generated. Note that a feature function is a function that determines in advance what kind of output is to be output with respect to a certain input. For example, by using a plurality of feature functions, further symbolization is performed from a symbolized data sequence. A plurality of data strings can be generated.

  Here, the conversion means 16 outputs the generated time series feature vector signal to the statistical learning means 18 when the learning process based on the learning input signal is being performed. Further, the conversion means 16 outputs the generated time-series feature vector signal to the integration means 21 when performing recognition processing based on an arbitrary input signal for recognition.

  The statistical learning unit 18 performs machine learning processing based on the time-series feature vector signal obtained by the conversion unit 16 and the correct viewing status label data (correct viewing status data) set in the learning database 19 in advance. The learning parameter is acquired by For example, CRF (Conditional Random Field) can be used for the machine learning process in the present embodiment, but the present embodiment is not limited to the CRF, and handles time-series vector sequences. Other machine learning processes that can be used can be used.

  Specifically, the statistical learning means 18 describes a relationship between time-synchronized symbol sequences obtained in the above-described process, and newly adds a plurality of time-series symbol sequences (for example, feature symbol sequences). ) And a learning parameter for the viewing state recognition process can be acquired from the generated symbol string using a machine learning process such as CRF. The statistical learning unit 18 outputs the acquired learning parameter to the learning parameter storage unit 20.

  The learning parameter storage unit 20 stores the learning parameters obtained from the statistical learning unit 18.

  The integration unit 21 performs integration processing on the time-series feature vector signal obtained by the conversion unit 16 based on the learning parameter obtained by the learning parameter storage unit 20, and the user's viewing situation with respect to the input arbitrary input signal Recognition results (for example, interest level, interest type (direction), etc.) are output.

  In the present embodiment, the learning database 19, the feature function group database 17, and the learning parameter storage unit 20 described above may be configured separately or may be provided in one storage unit.

  As described above, in this embodiment, time series viewing status data is output as a recognition result by integrating time series data of user interaction information with devices such as viewer video data, audio data, and operation history. To do. As a result, in this embodiment, for example, when watching TV, the viewer interacts with the program by integrating device interaction log data such as video, audio, and operation history that is measured as an arbitrary input signal. It is possible to recognize the viewing situation including mental information such as whether or not the user has. Note that the above-described viewing status recognition device 10 may be provided integrally within a program viewing device such as a television, for example, or may be provided separately.

  In this embodiment, for example, the learning process is performed as an offline process before the viewing state recognition process, and the recognition process is performed as an online process after the learning process. In the present embodiment, for example, the analysis unit 12, the histogram generation unit 13, the representative value selection unit 14, the synchronization symbol sequence generation unit 15, and the conversion unit 16 described above are recognition processing for viewing status even in the learning process. Even if it exists, the same processing is performed on the input data. Here, the learning process and the recognition process described above will be specifically described with reference to flowcharts.

<Example of learning process>
FIG. 2 is a flowchart illustrating an example of the learning process in the present embodiment. In the learning process shown in FIG. 2, first, a plurality of input signals for learning are acquired (S01), the above-described analysis process is performed on the acquired input signals, and time-series symbolization of each signal is performed ( S02). In the learning process, a histogram is generated for the symbol string obtained by the process of S02 (S03).

  Next, the learning process selects a representative value based on a predetermined condition for the generated histogram data (S04), and synchronizes using data having different properties for each time series based on the selected representative value. A symbol string is generated (S05). Further, in the learning process, the synchronization symbol sequence obtained in the process of S05 is converted by a preset feature function (S06), and the statistical data using the correct viewing status label data for the converted data. Learning is performed (S07). In the learning process, the learning parameter obtained by the statistical learning in S07 is stored in a storage unit or the like for use in the recognition process (S08). The learning process described above is preferably executed as an offline process before the recognition process is executed, but is not limited to this.

<Example of recognition processing>
Next, an example of recognition processing in this embodiment will be described. FIG. 3 is a flowchart showing an example of recognition processing in the present embodiment. In the recognition process shown in FIG. 3, for example, an arbitrary input signal is acquired from a viewer who is watching a program or the like (S11). The arbitrary input signal preferably has the same signal format as the learning and input signal, but is not limited to this.

  Next, in the recognition processing, the above-described analysis processing is performed on the input signal obtained by the processing of S11, and time-series symbolization is performed on each signal (S12). In the learning process, a histogram is generated for the symbol string obtained by the process of S12 (S13).

  Next, the recognition process selects a representative value from the generated histogram data based on a predetermined condition (S14), and generates a synchronization symbol sequence using data having different properties for each time series from the selected representative value ( S15) The generated synchronization symbol string is converted by a preset feature function (S16). In addition, the process of S12-S16 mentioned above performs the process similar to the process of S02-S06 in the learning process mentioned above.

  Next, in the recognition process, the time series feature vector signal converted by the process of S16 is integrated by the learning parameter obtained by the learning process described above (S17), and the integrated result is obtained for each viewer. Is output as the recognition result of the viewing status (S18), and the viewing status (for example, interest level) for each viewer corresponding to the input signal is output from the recognition result (S19).

  As described above, the present embodiment can collect learning data for parameter prediction and perform recognition processing using a predicted parameter by using the machine learning processing framework as described above. .

<Statistical learning method in learning process>
Here, the statistical learning method in the learning process described above will be specifically described. In the statistical learning method according to the present embodiment, for example, a CRF excellent in handling a time-series signal labeling problem is used as the machine learning process, but the present invention is not limited to this.

  The CRF is a machine learning process for obtaining the most probable labeling time series data y corresponding to the time series observation vector sequence x, and the basic ones are expressed by the following formulas (1) to (3). It becomes.

Here, in the above-described expression (1), p () indicates a probability distribution, Z () indicates a normalization term (normalization function), and exp () is a function for obtaining a power of a numerical value with e as the base. Is shown. In (1), x represents an observation signal sequence (time-series symbol sequence), y represents an output value of a hidden state at each x and t (for example, determination result of interest), and t represents time. Yes. The hidden state means, for example, a state invisible to the viewer (for example, interested / not interested), but is not limited thereto. In the above-described equation (1), Z () is defined by equation (2), and F () is defined by equation (3).

  Regarding y and y ′ in the above formulas (1) to (3), for convenience of expression, y represents one state vector, and y ′ represents all state vectors summed by Σ. Only different expressions are used to emphasize the expression, and the same y may be used.

Further, f _i () and g _i () in the expression (3) are called feature functions, and include time-series observation data (for example, including the above-described observation information, operation history information, operation information, etc.) and labeling data. Is a function with Specifically, f _i () is a feature that expresses the interaction of x as a result of the synchronization processing. For example, in what state, the observed symbol string in the synchronized state is in what state Shows the function to find out what you are most interested in. G _i () is a feature that depends on the transition of y, which is a state variable. For example, g _i () indicates how easily a state transitions from one state to the next over time. The function shown is shown.

  In the learning process using the CRF, first, learning parameters are obtained from observation data obtained from the viewer and correct labeling data. Next, it consists of a procedure that outputs the recognition result of the viewing situation using the learning parameter obtained for the observation data whose labeling data is unknown and the output data of the feature function calculated from the observation data obtained from the viewer Is done.

In this embodiment, in addition to the above-described equations (1) to (3), there is a structure having a hidden layer set in advance between the observation data sequence and the labeling data sequence. This method can also be applied to the present embodiment. For the above-mentioned CRF, for example, "J.Lafferty, A.McCallum, and F.Pereira.. " Conditional random fields: probabilistic models for segmenting and labeling sequence data, "Proc.18 th International conf on Machine Learning, 2001 ", etc. Is shown in

  Here, FIG. 4 is a diagram for explaining the flow of recognition processing using machine learning processing in the present embodiment.

  In FIG. 4, the collection of learning data for parameter prediction indicates the procedure as the learning process described above.

  In the present embodiment, the synchronization processing (synchronization symbol sequence generation) and feature function processing (synchronization symbol sequence conversion processing using a plurality of feature functions) described above from a plurality of data (input signals) having different properties existing in time series. And a process for generating a time-series feature vector signal and a process for giving a viewing status label value corresponding to the generated time-series feature vector signal by an administrator or the like. Note that the labeling described above may be set in advance.

  In the present embodiment, a plurality of combination data groups of observation data for CRF learning processing and corresponding correct data are acquired by the above-described processing. Thereafter, in this embodiment, the learning parameters are predicted using these data. This learning parameter corresponds to {θ} in the case of CRF.

  In the recognition process shown in FIG. 4, a time-series feature vector signal is obtained from a plurality of data (arbitrary input signals) having different properties existing in a time series in the same process as when learning data is collected for parameter prediction. Generate. At this time, the viewing status label corresponding to this time-series feature vector signal is unknown. Therefore, a viewing status label is assigned using the obtained learning parameter. Note that the viewing status label is assigned by selecting the most probable labeling sequence, for example, as shown in equation (4) below.

Here, in the above-described equation (4), Θ represents a learning parameter.

<Procedure for generating time-series feature vector signal>
Next, a procedure for generating the above-described time series feature vector signal will be described with reference to the drawings. FIG. 5 is a diagram for explaining a procedure for generating a time-series feature vector signal using a plurality of different data.

  In the example of FIG. 5, first, a histogram representing the frequency of a signal corresponding to a preset unit time interval is generated from each input heterogeneous data. Thereafter, in the representative value selection process, a representative symbol value is determined from the histogram shape, and the determined symbol value is set as an output value at the time interval.

  As a process of selecting a representative symbol value from the histogram, for example, a process of selecting a maximum frequency value or a minimum frequency value can be used, but the present invention is not limited to this. An average value or the like of each symbol value obtained from a plurality of histogram data at intervals may be obtained, and the obtained average value may be used as an output value.

  For example, in the present embodiment, when the input signal is a symbol string indicating device interaction such as program channel switching or volume adjustment, the symbol of the interaction most frequently performed in the unit time is the output of the representative value selection process.

  As described above, in this embodiment, by performing histogram generation and representative value selection processing from observation data in each section of a preset time interval (for example, time Tp shown in FIG. 5), the time resolution and the like are set. Different types of signals can be synchronized, and the interaction between the signals can be expressed by the following feature function processing.

  In the feature function processing in the present embodiment, an arbitrary function that expresses a relationship between a plurality of signals can be set, and a plurality of relational expressions are defined according to the type of viewing situation to be recognized, and each output is defined. Can be grouped into a time-series feature vector signal. In particular, in the learning process using CRF, a signal at any time of the input observation time series data can be handled as a variable of the feature function, and a flexible feature function can be assembled.

  Here, FIG. 6 is a diagram for explaining an example of generation of learning data. In the example of FIG. 6, for example, the viewing situation is the degree of interest when watching a TV program, and face direction information (Dir), body rest state (Mov), facial expression change (Exp), 3 shows a set of learning data in the case of operating information (Tab) of a portable information terminal such as a tablet. Note that the type of information serving as learning data is not limited to this.

  The table of FIG. 6 represents a time series signal converted from observation data at the time of program viewing into data for each unit time of a representative value obtained from a histogram for each unit time as an input signal. In the table of FIG. 6, the interest level (Curi) of interest (Curi) / none (Non) at that time is represented as the interest level of the viewing situation. The interest determination is an interest marker subjectively added by an administrator or the like to generate learning data.

Here, the feature function may describe an interaction related to interest from these input signals. For example, one or more logical functions (for example, {f _i (x, y _t )}, {g _i ( y _t−1 , y _t )}) can be generated.

  FIG. 7 is a diagram illustrating an example of a logical function group. In FIG. 7, logic function groups f0 to f20 are shown as examples of feature functions. In the logical function shown in FIG. 7, Dir represents “face orientation information”, Mov represents “body still state”, Exp represents “expression change”, and Tab represents “portable information terminal such as a tablet” Operation information ”.

  Furthermore, symbols such as Front, TB, Pos, Neg, Ch, and Non shown in FIG. 7 represent symbols individually representing states such as face orientation information, expression changes, and operation information, for example. Specifically, Front is “forward”, TB is “normal terminal operation”, Pos is “active operation”, Neg is “reactive terminal operation”, and Ch is “channel switching by terminal operation” “Yes” and Non are symbols for “do nothing”. T represents time. Here, TB, Pos, and Neg, which are examples of the above-described operation information symbols, will be described more specifically. These symbols are subjectively added to the operation of a portable information terminal such as a tablet terminal at the stage of system design, and how much the operation relates to the program content currently being viewed. Is a symbol that expresses

  In general, when watching a television while operating a portable information terminal or the like, it is assumed that various functions on the portable information terminal are used. Therefore, when the operation is related to a TV program, for example, when the operation is related to a TV program, Pos is input. Neg is added when the terminal is in operation, and TB is added when the terminal is operated casually. In the present embodiment, the user's degree of interest in a program and the like can be estimated with higher accuracy by the terminal operation as described above.

  For example, Dir (t) = Front represents a state that “when the face is at the time t, it is facing the front”. In the present embodiment, as described above, the state is individually identified by each symbol, and the logic shown in FIG. 7 can be formed by combining these states. Note that the types, number, order, contents, and the like of logic functions are not limited to this.

  In the present embodiment, the output of the function group selected as shown in FIG. 7 is a constituent element of the time-series feature vector signal, and is a part f in the equation (1). Further, in the feature function from f0 to f20 shown in FIG. 7, the labeling state (output value) y is omitted as a variable, but in reality, the feature function output from f0 to f20 is calculated for each labeling state. It will be.

  Here, f0, f17, and f18 shown in FIG. 7 will be described as examples of specific processing of the feature function in the present embodiment. The feature function of f0 is, for example, a logic that outputs “1” as an output at time t when the face is facing forward at the time t, there is a change in facial expression, and it is stationary.

  Also, the feature function of f17 is a logic that outputs “1” as the output at time t when the channel is switched by terminal operation at time t, for example. The feature function of f18 is, for example, logic that takes into account all input states, and there is a channel switching at time t-1, and the face turns to the front at time t, and the face turns to the front at time t + 1. Or when it is still, the logic is to output “1” as the output at time t.

  In the present embodiment, such weighting parameters θi are calculated for each feature function element by assembling such one or more feature functions from the input and performing learning processing.

<CRF learning>
Here, in the above-described CRF learning, Z (x) in the above-described equation (1) can be calculated using a forward-backward algorithm widely used in HMM (Hidden Markov Model) learning and the like. In the present embodiment, learning can be efficiently performed in combination with an optimal algorithm such as the L-BFGS (Limited-memory Broden Fletcher Goldfarb Shanno) method which is a quasi-Newton method.

  Further, the processing of the above-described expression (4) can also be obtained efficiently by using the Viterbi algorithm in this embodiment. Furthermore, in the present embodiment, a plurality of hidden states h are provided between the output time-series labeling data y and the observed vector sequence x for the basic CRF of the above-described equation (1), and the respective hidden states and outputs are provided. LDCRF (Lent Dynamic Conditional Random Field) for obtaining y from the relationship between labeling can be used. Regarding the above-mentioned LDCRF, for example, “L.-P. Morency, A. Quattoni and Trevor Darnell, etc. It is shown.

  Here, in the case of the above-described LDCRF, what is obtained in the framework of the CRF is a probability to h that represents each hidden state, and the probability of y is obtained by equation (5).

In addition, when there is no overlap between hidden states associated with individual output labels, it is known that a simple sum is obtained as shown in the following equation (6), and the basic CRF framework is maintained as it is. Calculations to follow are possible.

In the case of the above-described LDCRF, g _i () in the above-described equation (3) is a feature that depends on the transition of h, which is a state variable, and for example, from a hidden state to the next hidden state, The function which shows the ease of transfer of how to change according to progress is shown. In the present embodiment, by using LDCRF, it is possible to improve the identification performance more than CRF.

<Specific signal flow from input signal to learning parameter output>
Here, a specific signal flow from the input signal to the learning parameter output will be described. In the following description, as an example of an input signal, a process from input to viewing status recognition output will be described using, for example, an observation video signal when the viewer watches the television and operation channel (Ch) switching information.

[1. Time-series symbol sequence from original signal]
First, the face orientation is calculated as an analysis process for the video signal obtained as observation data. In addition, the operation history signal is summed up as to whether or not there is an operation event every certain time, and a symbol is assigned and recorded for each operation information when each operation is performed. The symbol when the channel (Ch) is switched is “Ch”.

[2. Histogram generation from time series symbol sequence data]
Next, a histogram is generated based on the face orientation and operation history information within a certain period of time (for example, 5 seconds). In the case of face orientation, how often each face orientation occurred within a certain time, and for the operation history, how often each operation occurred within a certain time. Reflected by histogram generation.

[3. Generation of time series symbol data and representative value selection processing]
As for the face direction, the face direction indicating the maximum frequency of the histogram data is set as the output of the histogram generation section. For the operation history information, the type of operation information indicating the maximum frequency of the operation information histogram is output as the histogram generation section. For channel switching, the symbol “Ch” is assigned to the section with the most channel switching. If no operation is performed, the symbol “Non” is assigned.

[4. Generation of time-synchronized time series symbol sequence]
The face direction signal sequence and the channel switching symbol sequence signal synchronized in time are obtained by the above processing.

[5. Example of signal generation of time series feature vector signal sequence with feature function]
Next, a plurality of time-series feature vector signal sequences are generated from a plurality of different feature function outputs using a plurality of time-synchronized time-series symbol sequences as input variables. Here, description will be made using an example of a feature function that handles time-series signals of channel switching and face orientation as inputs.

  First, the current time is T, and the previous time is T-1. If the operation information Tab (T-1) at time T-1 and the face orientation information Dir (T) at time T are input signals, a function such as the following equation (7) is given as an example of a feature function. is assumed.

This is a function that performs channel switching at time T-1 regardless of the hidden state Y (T) at time T, outputs 1 when facing the front at time T, and outputs 0 otherwise. It is.

  As a result of this feature function, a new time-series signal that outputs 1 only when the condition is satisfied is generated from the time-series signals of channel switching and face orientation.

  The time series feature vector signal is configured as a vector time series signal in which a plurality of feature function outputs described for each of a plurality of assumed states as described above are combined.

[6. Learning weighting parameters for integration of feature functions]
In the present embodiment, weighting parameters for integrating a plurality of feature functions are learned and stored in a machine learning framework. The stored weighting parameters are stored as learning parameters and used during recognition processing.

  That is, in the present embodiment, for example, the weighting parameter is included in the learning parameter, and the value of the weighting parameter obtained by learning is used in the recognition process, for example, based on the above-described equation (3). Then, by multiplying the feature function value calculated from the state of the recognition target, the probability value of the recognition target is obtained by the above equation (1). In recognition, the state y, that is, the degree of interest is estimated by selecting y having the highest probability of the expression (1) according to the above-described expression (4).

<Other embodiments>
In the viewing status recognition device 10 described above, learning parameters are predicted based on the collected learning data, and the labeling of the viewing status is calculated according to the above-described formula (4). It may be influenced by whether the same high quality learning data can be acquired.

  For example, correct labeling data for viewing status such as “I watched with interest” is suitable to be performed simultaneously with measurement of observation data, but in this embodiment, interaction information with the device is also included in the observation data. ing. Therefore, it is difficult for the user to input correct labeling data while operating the device.

  Therefore, in such a case, in the conventional method, after viewing the whole program, a method of giving correct labeling data while remembering the situation was used. However, this approach tends to be inaccurate due to memory ambiguity. Therefore, in the present invention, a configuration for providing correct labeling data in real time while viewing a program is provided. Specifically, the above-described configuration of the viewing status recognition apparatus 10 is provided with foot operation means such as a foot pedal, for example, and correct data of data to be labeled such as the degree of interest and the type of interest (direction) by the foot operation of the viewer Input. Here, the above-described content will be described below as another embodiment.

  FIG. 8 is a diagram illustrating an example of a functional configuration of a viewing status recognition apparatus according to another embodiment. In the viewing situation recognition device 30 shown in FIG. 8, the same components as those in the above-described viewing situation recognition device 10 are denoted by the same names and reference numerals, and detailed description thereof is omitted here.

  8 includes an information acquisition unit 11, an analysis unit 12, a histogram generation unit 13, a representative value selection unit 14, a synchronization symbol string generation unit 15, a conversion unit 16, and a feature function. The group database 17, the statistical learning means 18, the learning database 19, the learning parameter storage means 20, the integration means 21, the foot operation means 31, and the correct data conversion means 32 are configured.

  In the configuration shown in FIG. 8, a foot operation unit 31 and a correct data conversion unit 32 are newly provided as compared with the above-described viewing state recognition device 10. As the foot operation means 31, for example, a foot pedal used in a flight simulation game, a driving game, or the like can be used.

  The foot operation means 31 inputs an interest level, an interest genre (kind of interest), and the like by a viewer's foot operation when, for example, content of interest is viewed with respect to a program being viewed by the viewer. The interest level is obtained, for example, by setting evaluation criteria such as 3 levels or 10 levels in advance, and acquiring each set level based on how the foot pedal is depressed. For the genre of interest, the content of the genre set in advance according to how the foot pedal is depressed is acquired. The genre of interest includes at least one of, for example, actors and actresses appearing in the program, clothes worn by the characters, scenery, buildings, music, program contents, etc. It is not limited to. In addition, about an interest level and an interest genre, each information can be input, for example using a right foot or a left foot.

  Note that the foot operation means 31 is not limited to the above-described foot pedal. For example, a camera image of a viewer's foot is acquired, and the interest described above is obtained from the positional relationship of both feet analyzed from the acquired camera image. A level, an interest genre, etc. may be input. In addition, the information input by the foot operation means 31 is not limited to the above-described interest level or interest genre.

  The foot operation means 31 outputs the acquired interest level and interest genre to the correct data conversion means 32. At this time, the foot operation means 31 adds and outputs time information corresponding to program viewing. Thereby, it is possible to easily synchronize with the viewer information obtained when the viewer views the program.

  The correct answer data conversion means 32 generates correct viewing status label data from the interest level and interest genre obtained from the foot operation means 31 and stores the generated label data in the learning database 19. The data obtained by the foot operation means 31 and the correct answer data conversion means 32 is acquired as, for example, the interest determination (Cur) shown in the table of FIG. 6 described above, but is not limited thereto.

  In the embodiment shown in FIG. 8, the correct viewing status label data can be set in real time under the same environment as that of a general viewer by inputting with the foot without using hands or voices.

  Since the foot operation means 31 and the correct answer data conversion means 32 shown in FIG. 8 are used in the learning process, they are mainly used by experimenters and managers for learning correct answer data, but are not limited thereto. For example, a general viewer may use it.

  Here, FIG. 9 is a diagram for explaining specific contents of another embodiment. In the example of FIG. 9, for example, the observation data or user interaction data obtained from the information acquisition unit 11 is converted into a time-series feature vector signal by the conversion unit 16, and the left and right foot pedals 40 as the foot operation unit 31 are further processed. -1 and 40-2 conversion of pedal depression position information ([left: x, y, z], [right: x, y, z]) into correct data of viewing situation by correct data conversion means 32 Process. Specifically, as shown in FIG. 9, the xyz position positions of the left and right foot pedals 40-1 and 40-2 are output by the operation of the foot operation means 31 by the viewer. By assigning the viewing status label value, it is possible to easily register the correct answer data only by operating the foot pedal.

  In FIG. 9, the learning data described above is collected based on the results of the respective conversion processes, and learning parameters are acquired.

  Here, the correspondence between the label of the viewing situation and the xyz position position from the foot pedals 40-1 and 40-2 is defined in advance in, for example, an xml (Extensible Markup Language) file or the like. Corresponding labeling can be performed.

  Here, FIG. 10 is a diagram for explaining the correspondence between the viewing status label and the foot operation. Note that the example of FIG. 10 shows an example of an xml file in the case where, for example, the level of interest (degree) as the viewing situation and the genre that interests in (the type of interest) are input simultaneously.

  In the example of FIG. 10, as the definition of the foot pedal, for example, the degree of interest is input in three stages (represented by 0, 1 and 2) using the right foot foot pedal (right). In the example of FIG. 10, for example, the genre that interests the user is input in three types (represented by 0, 1, and 2) with the left foot pedal (left).

  Here, the level of interest on the right foot is input corresponding to the amount of stepping in the range by dividing the maximum value 255 of the input value into three equal parts in any of the three-dimensional x, y and z directions. Is done. Specifically, numerical data of 0 is output when the amount of depression is 0 to 80, numerical data of 1 is output when 80 to 120, and numerical data of 2 is output when 120 to 255. Is output. In other words, the maximum interest level (for example, interest level 2) is output if you make a thorough decision in any direction.

  For the genre of interest, only the amount of depression of the left foot in the Z direction is supported, and a value corresponding to a range obtained by dividing the maximum value 255 of the input value into three equal parts is input. Specifically, in the example of FIG. 10, there are numerical data of {0, 1, 2} as input values, but by making each numerical value correspond to the genre of interest, it is possible to input genre input. . Through such processing, the viewer can easily input the viewing status at that time while interacting with the device as usual.

  In addition, when there are a plurality of genres of interest at a certain point in time, data for each genre may be input in a short time, and the depression amount of the foot pedal 40 combining a plurality of genres is set in advance. You may make it step on that position.

  As described above, according to another embodiment, the label data string of the recognition result of the viewing situation for machine learning can be created without impairing the television viewing situation, for example. Specifically, by making the labeled viewing situation results correspond to the amount of foot pedal depression used in, for example, a flight simulation game, a correct viewing situation data string for learning is created in real time while operating with the feet. can do.

  In the above-described embodiment, the degree of interest of the program obtained from the recognition result has been described as an example of the output content of the viewing situation. However, the present invention is not limited to this. For example, the content of interest and the preference of the viewer May be estimated.

<Execution program (viewing situation recognition program)>
Here, the above-described viewing status recognition devices 10 and 30 are, for example, volatile storage media such as a CPU (Central Processing Unit), RAM (Random Access Memory), and non-volatile storage media such as a ROM (Read Only Memory). In addition, it can be configured by a computer having an input device such as a mouse, a keyboard, and a pointing device, a display unit for displaying images and data, and an interface for communicating with the outside.

  Accordingly, the functions of the viewing status recognition devices 10 and 30 can be realized by causing the CPU to execute a program describing these functions. The program can also be stored and distributed on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

  That is, in the present embodiment, an execution program (viewing situation recognition program) for causing a computer (hardware) to execute the processing in each configuration described above is generated, and the program is installed in, for example, a general-purpose personal computer or server. Accordingly, it is possible to realize the viewing state recognition process having the above-described learning process, recognition process, and the like by cooperating the above-described hardware and software including a program or the like.

  As described above, according to the present invention, the viewing status of the viewer can be recognized with high accuracy. Specifically, according to the present invention, it is possible to construct a mechanism for describing the relationship between a plurality of signals having different properties that allow the viewer's viewing status to be obtained and automatically outputting the relationship by machine learning processing. For example, in the present invention, an estimation of an interest interval when watching TV is performed, and a plurality of pieces of information such as an operation history of a remote control or a portable information terminal, a person recognition result by a camera image, etc. are integrated in a CRF and labeled in time series. Treat as information. As a result, in the present invention, when watching TV, etc., the viewing situation such as which scene the viewer is interested in is estimated, and the program and information are adaptively adapted according to the interest state of the program. It can be used for information recommendation services such as UTAN (User Technology Assisted Navigation) to be recommended. Specifically, for example, using UTAN to which the present invention is applied, the interest level of a viewer that changes during a program is estimated from recognition technology, and program recommendation is performed using subtitles and program information in that time period. And related keywords can be presented on a portable information terminal or a television screen.

  Further, according to the present invention, it is possible to construct an action support system that reflects a situation viewed with interest by assigning a label not only to the level of interest in the viewing situation but also to the activity content after the interest viewing, for example. it can.

  Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10,30 Viewing condition recognition apparatus 11 Information acquisition means 12 Analysis means 13 Histogram generation means 14 Representative value selection means 15 Synchronization symbol sequence generation means 16 Conversion means 17 Feature function group database 18 Statistical learning means 19 Learning database 20 Learning parameter storage means 21 Integration means 31 Foot operation means 32 Correct data conversion means 40 Foot pedal

Claims (6)

In a viewing status recognition device for recognizing a viewing status based on viewer information obtained when a viewer views a program,
Histogram generating means for generating histogram data from a plurality of different data obtained as the viewer information;
Representative value selection means for selecting a representative value for each unit time from histogram data obtained by the histogram generation means;
Using a representative value obtained by the representative value selection means, a synchronization symbol string generation means for generating a synchronization symbol string by synchronizing the plurality of different data;
Conversion means for converting a synchronization symbol sequence obtained by the synchronization symbol sequence generation means into a time-series feature vector signal using a preset feature function;
A viewing situation recognition comprising: a time series feature vector signal obtained by the converting means, using a learning parameter indicating a preset weight, and integrating a means for recognizing the viewing situation of the viewer. apparatus. 2. A statistical learning means for generating the learning parameter by statistical learning based on a time-series feature vector signal obtained by the conversion means and preset correct viewing situation data. The viewing situation recognition device according to 1. The statistical learning means includes
A plurality of time series symbol sequences are newly generated based on the relationship between the plurality of synchronization symbol sequences obtained by the synchronization symbol sequence generation means, and the learning is performed by a machine learning process set in advance from the time series symbol sequences. The viewing condition recognition apparatus according to claim 2, wherein a parameter is generated. The histogram generation means includes
4. The viewing status recognition apparatus according to claim 1, wherein a histogram is generated based on a plurality of different unit times.   4. The viewing status recognition apparatus according to claim 2, further comprising foot operation means for inputting information that is a basis of the correct viewing status data by operating the viewer's foot. In the viewing status recognition program for recognizing the viewing status based on the viewer information obtained when the viewer views the program,
Computer
Histogram generating means for generating histogram data from a plurality of different data obtained as the viewer information;
Representative value selecting means for selecting a representative value for each unit time from histogram data obtained by the histogram generating means;
Using a representative value obtained by the representative value selection means, a synchronization symbol string generation means for generating a synchronization symbol string by synchronizing the plurality of different data;
Conversion means for converting a synchronization symbol sequence obtained by the synchronization symbol sequence generation means into a time-series feature vector signal using a preset feature function; and
A viewing situation recognition program for integrating time series feature vector signals obtained by the conversion means using a learning parameter indicating a preset weight and functioning as an integration means for recognizing the viewing situation of the viewer.