JP2002073634A - Device for processing behavioral pattern - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a behavioral pattern processor capable of proposing a behavioral pattern obtained by taking the feelings of a user into consideration. SOLUTION: A characteristic association processing part 34 stores characteristic data obtained by making respective pieces of information which are respectively outputted from an image characteristic extracting part 12, a timing information extracting part 18, a music characteristic extracting part 22, a location information extracting part 28, a room temperature information extracting part 32 correspond to the feelings information of the user, outputted from a feelings recognizing device 14 in a characteristic DB 36. An inclination extraction processing part 38 acquires the respective pieces of the information outputted from the part 12, etc., and the feelings information outputted from the device 14, extracts the characteristic data showing a similar behavioral pattern by collating these information with the characteristic data stored in the DB 36 and retrieves movement contents that do not coincide or are omitted. A movement contents setting part 40 sets movement contents to be proposed to the user on the basis of movement contents retrieved by the processing part 38.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a behavior pattern processing apparatus for proposing a behavior pattern desired by a user.

[0002]

2. Description of the Related Art In recent years, various animal-type robots using various sensors have been attracting attention. These animal-type robots are programmed so that the response content changes while learning the user's daily behavior patterns and conversation patterns, so that the user can enjoy the feeling of raising an animal-type robot as a pet. Is available.

[0003]

By the way, the above-mentioned animal-type robot has a behavior (for example,
"Stroking,""talking," etc.) can be detected by using various sensors, and the corresponding reaction behavior can be performed. Could not do.

[0004] Further, the conventional animal-type robot cannot take a reactive action in consideration of the emotion and personality of the user. However, considering the reaction behavior of actual pets, when a person is in a bad mood, he looks like a complexion, and when a person is smiling and touching, he gets excited to respond to it. You cannot take a natural reaction without considering emotions and personalities.

[0005] In addition to the animal type robot, a similar problem arises when a future type system that proposes various behavior patterns to a user is considered in a vehicle-mounted device or a home device. That is, if a user cannot predict a desired behavior pattern, an accurate proposal cannot be made, but there has been no such system in the past.

[0006] The behavior pattern desired by the user is considered to be closely related to the emotional state and the like of the user at that time, but the behavior pattern desired by the user is determined in consideration of the emotion. There was no system that could be proposed. The present invention has been made in view of such a point, and an object of the present invention is to provide a behavior pattern processing device capable of proposing an accurate behavior pattern.

It is another object of the present invention to provide a behavior pattern processing device capable of proposing a behavior pattern in consideration of a user's emotion.

[0008]

In order to solve the above-mentioned problems, the action pattern processing apparatus of the present invention uses a plurality of features of each action content for a user's action pattern including a plurality of action contents. And associated data associated with the extracted features are stored in a feature database. Then, when a feature of the user's action content is newly extracted by the feature extraction means, the action suggestion means proposes at least one of the plurality of action contents based on the association data stored in the feature database. It is carried out.

[0009] By storing association data in which the characteristics of the respective operation contents are extracted and associated with the behavior pattern of the user, data relating to the behavior pattern unique to the user is stored in correspondence with various situations. be able to. Therefore, when a feature of the user's action content is newly extracted, an action pattern desired by the user can be predicted based on the accumulated association data, so that an accurate proposal can be made.

In another behavior pattern processing apparatus of the present invention, the emotion of the user is determined by the emotion determination means, and the behavior pattern of the user including a plurality of operation contents is analyzed by the behavior pattern analysis means. In the feature database, association data that associates the user's emotion determined by the emotion determination unit with the user's behavior pattern analyzed by the behavior pattern analysis unit is stored. When the emotion of the user is newly determined by the emotion determination unit and the behavior pattern of the user is analyzed by the behavior pattern analysis unit, based on the association data stored in the feature database,
The motion suggestion unit proposes at least one of the plurality of operation contents.

[0011] By storing association data that associates a user's emotions and behavioral patterns, data relating to user-specific emotional changes and behavioral patterns can be accumulated in accordance with various situations. Therefore, when the user's emotion is newly determined and the behavior pattern is analyzed, the action content is proposed based on the accumulated association data, thereby proposing the behavior pattern in consideration of the user's emotion. It can be performed.

Further, the above-mentioned operation suggesting means is provided when a part of the plurality of operation contents included in the new action pattern does not match or is missing from the plurality of operation contents included in the association data. It is desirable to make a proposal. Based on the association data already stored,
By providing a mismatched or missing operation content, it is possible to reliably propose an operation content that is likely to be desired by the user.

It is desirable to store the association data in the feature database when the contents of the association data repeatedly acquired change. By storing the acquired association data only when the content of the acquired association data changes, the data amount of the association data stored in the feature database can be reduced.

[0014] It is preferable that the apparatus further comprises environment determining means for determining the surrounding environment of the user, and that the surrounding data determined by the environment determining means be associated with the association data stored in the above-mentioned feature database. Since the behavior patterns or emotions of the user are considered to be correlated with the state of the surrounding environment, by including the surrounding environment of the user in the association data, the behavior patterns or feelings of the user can be grasped more accurately. Can be.

[0015] Further, there is further provided a personality determining means for determining the personality of the user and extracting a specific user from predetermined candidates, and the above-mentioned action proposing means provides a suggestion for each user. It is desirable to set the content. As a result, it is possible to make proposals having different contents for each user, reflecting differences in behavior patterns and the like of each of the plurality of users.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A vehicle-mounted system according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram illustrating an overall configuration of a vehicle-mounted system according to the present embodiment. The in-vehicle system shown in FIG. 1 includes a camera 10, an image feature extraction unit 12, an emotion recognition device 14, and a timepiece 1.
6. Time information extraction unit 18, audio device 20, music feature extraction unit 22, navigation device 24, point information extraction unit 28, air conditioner 30, room temperature information extraction unit 32, feature association processing unit 34, feature DB (database) 36 , A trend extraction processing section 38, and an operation content setting section 40.

The camera 10 is installed at a predetermined position in a vehicle compartment, and captures a landscape in the lateral direction of the vehicle. The image feature extraction unit 12 performs various processes on the image captured by the camera 10 to extract features related to the image (hereinafter, referred to as “image features”). More specifically, for example, features relating to landscapes such as a landscape in which mountains can be seen, a landscape in which the sea can be seen, and a landscape in a downtown area, and features relating to the color tone (brightness and darkness) of an image are extracted as image features. The extracted image features are output to the feature association processing unit 34 together with the data of the image itself.

The emotion recognizing device 14 performs processing for recognizing the user's emotion, and as a result of the emotion recognition, "happy",
One of "angry", "sorrow", and "easy" is output. Also,
The emotion recognition device 14 of the present embodiment also has a function of extracting a specific user from a plurality of users (candidates). The detailed configuration of the emotion recognition device 14 will be described later.

The clock 16 measures the date and the current time.
The time information extraction unit 18 extracts information about time.
Specifically, in the present embodiment, one day (24 hours) is divided into several time zones as information on time, and the time zone to which the current time belongs is extracted. For example,
In the present embodiment, (1) 4:00 to
8:00, (2) 8:00 to 12:00, (3) 12:00 to 16:00,
(4) 16: 00-20: 00, (5) 20-24: 00, (6) 2
There is a section from 4:00 to 4:00. In this embodiment, for the sake of convenience, the above-mentioned category (1) is referred to as “early morning”,
The category of (2) is "AM", the category of (3) is "PM",
The category of (4) is "evening", the category of (5) is "night",
The category (6) will be referred to as “midnight”.
The method of dividing these time zones is an example, and the present invention is not limited to this.

The audio device 20 includes, for example, a CD changer which can load a plurality of music CDs and selectively reproduce any one of the music CDs, and reproduces music desired by the user. I do. Music feature extraction unit 22
Extracts features related to music (hereinafter, referred to as “music features”) corresponding to the music being reproduced by the audio device 20. Specifically, as the music feature, for example, a "music genre" such as classical music, pop music and jazz, and a "musical tone" such as up tempo and slow tempo are extracted.

The navigation device 24 performs processing such as display of map information around the position of the vehicle, route search processing, and route guidance processing, and has a display device 26 for displaying a map image and the like. . The point information extraction unit 28 extracts point information corresponding to the current position of the vehicle from the navigation device 24. Specifically, the point information extraction unit 28
As the point information, the road type indicating whether the road on which the vehicle is traveling is a general road or an expressway, and whether the position of the vehicle is near a mountain, near the sea, or near a downtown area. Information such as a location type indicating the status of the location corresponding to the vehicle position is extracted as location information. When a predetermined destination is set and the route guidance processing is being performed, coordinate information (longitude and latitude) corresponding to the destination is extracted as point information.

The air conditioner 30 is for adjusting the room temperature in the passenger compartment. The room temperature information extraction unit 32 includes the air conditioner 3
The information about the room temperature in the vehicle compartment (hereinafter, referred to as "room temperature information") is acquired from a room temperature sensor (not shown) provided in the vehicle. The feature association processing unit 34 includes information output from each of the above-described image feature extraction unit 12, time information extraction unit 18, music feature extraction unit 22, point information extraction unit 28, room temperature information extraction unit 32, and emotion recognition. Apparatus 1
4 is associated with information about the emotion of the user (hereinafter, this information is referred to as “emotional information”), and the characteristic DB 36 is used as characteristic data (association data) about the user.
A predetermined feature associating process is performed.

The feature association processing unit 34 also performs a process of counting the frequency of appearance of feature data having the same content, and the feature data having the same content as the feature data already stored in the feature DB 36 is obtained. If extracted, the appearance frequency of the already stored feature data is updated.

The feature DB 36 is a feature association processing unit 34
The feature data output from is stored. FIG. 2 shows the feature D
It is a figure explaining the contents of the characteristic data stored in B36. As shown in the figure, the feature associating processing unit 34 includes the image feature extracting unit 12, the time information extracting unit 18, the music feature extracting unit 22, the point information extracting unit 28, and the room temperature information extracting unit 32, Information and emotion recognition device 14
By adding a common ID (identification number) to the emotion information output from, the information is associated with each other and stored in the feature DB 36 as a set of feature data. For example, the feature data with the ID “1” will be described. As the image features, “Image # 1” which is data of the image itself, “Dark” which is a feature related to color tone, and “Sea” which is a feature related to landscape・ Emotion information stores “sorrow”. ・ Time information stores “night”. ・ Music characteristics include “pops” related to genre and “slow” related to tune. Tempo) is stored.-As the point information, "X1, Y1" of the coordinate information, "Expressway" of the road type, and "Sea" of the point type are stored.-As the room temperature information, "25" is stored. ° C ”is stored. Also, “3” is stored as the appearance frequency, which indicates that feature data having the same content as the feature data with ID “1” has appeared three times so far.

In the feature data with the ID “3”, no information is stored on the music feature, which indicates a state in which the user is not listening to music.
In the present embodiment, "not listening to music"
Is also treated as one of the music features.

As described above, the emotion recognition device 14 of the present embodiment has a function of extracting a specific user from a plurality of users (candidates) (details will be described later). When storing the feature data in the feature DB 36, the association processing unit 34 stores the feature data corresponding to each of the plurality of users specified by the emotion recognition device 14. By storing the feature data corresponding to each of a plurality of users in this way, it is possible to reflect different behavior patterns and the like of each of the plurality of users and make a proposal having different contents for each user. it can.

The trend extraction processing unit 38 includes the image feature extraction unit 1
2. The information output from each of the time information extraction unit 18, the music feature extraction unit 22, the point information extraction unit 28, and the room temperature information extraction unit 32, and the emotion information output from the emotion recognition device 14 are acquired. Is compared with the feature data stored in the feature DB 36 to extract feature data indicating a similar behavior pattern, and to search for mismatched or missing action content.

The operation content setting unit 40 includes the trend extraction processing unit 3
8 to set a proposed operation content based on the operation content searched by the user, and propose this to the user. next,
The detailed configuration of the above-described emotion recognition device 14 will be described. FIG.
FIG. 2 is a diagram showing a detailed configuration of the emotion recognition device 14. The emotion recognition device 14 shown in the figure includes a microphone 50, a voice recognition unit 52, a camera 54, an image recognition unit 56, a sensor 58,
Motion recognition unit 60, personality DB 62, personality recognition unit 64, emotion recognition units 66, 70, 74, voice DB 68, image DB 7
2, an operation DB 76, a weighting processing unit 78, and an individuality data correcting unit 80.

The microphone 50 collects the voice uttered by the user and converts it into an electric signal. The voice recognition unit 52
The voice signal of the user collected by the microphone 50 is analyzed to perform voice recognition, and a word uttered by the user is specified. Further, the voice recognition unit 52 extracts a feature amount of the user's voice based on the result of the voice recognition. As the feature amount extracted by the voice recognition unit 52, for example, the pitch and loudness of a voice, the utterance interval of each character when speaking, and the like can be considered.

The camera 54 photographs the facial expression of the user.
The image recognition unit 56 analyzes a facial expression of the user photographed by the camera 54 and extracts a feature amount. Image recognition unit 56
For example, the position and size of each element such as the eyes, eyebrows, and mouth included in the face, or the movement of these elements can be considered as the feature amount extracted by the. Specifically, the contour of the eye, the angle of the eye, the angle of the eyebrows, the degree of opening of the mouth, the number of blinks per unit time, and the like can be considered.

The sensor 58 detects a user's operation and converts it into an electric signal. Here, in this specification, in addition to the position and movement of each part of the body such as the user's limbs and head, the body temperature, the amount of perspiration, and the like, include all physical quantities that can be measured with respect to the state of the user's body. It shall be expressed as “operation”. Therefore, the sensor 58 of the present embodiment includes various sensors such as a temperature sensor, a pressure sensor, and a position sensor. As a specific installation location of the sensor 58 in the present embodiment, a handle or a seat of a driver's seat, and other spaces around the driver's seat can be considered.

The motion recognition section 60 analyzes the motion of the user detected by using the sensor 58 and extracts a feature amount. Specifically, for example, by installing a pressure sensor on the handle and examining the change in pressure applied to it, it is possible to check the movement of the hand, whether it is moving the hand more frequently, or holding the handle strongly Information such as whether or not it is available. In addition, by installing a position sensor on a seat or other place, information such as the user's sitting height, head position, and shoulder position can be obtained.

The personality DB 62 stores personal identification data for specifying an individual based on the personality of the user and a personality data group for use as a criterion for recognizing the emotion of the user. . FIG. 4 is a diagram showing an example of personal identification data stored in the personality DB 62. As shown in the figure, the personal identification data includes, for example, a waveform pattern of a voice when a user utters a predetermined word, an outline of a chin, an element such as sitting height, that is, regardless of a change in emotion. An element whose value is substantially constant is considered, and data corresponding to each of the users A, B,... Is registered. These data are given as values having a predetermined width as shown in FIG. For example, user A
Are defined by the upper limit value a1 and the lower limit value a1 ', and are represented as a1 to a1'. However, as described above, since an element whose value is substantially constant regardless of a change in the user's emotion is considered as an element in the personal identification data, the upper limit and the lower limit in the personal identification data are considered. The value specified by the value has a narrow width that indicates an error range. The same applies to other elements.

In addition, the personality DB 62 stores the voice in a state where the user does not have a specific emotion, in other words, in a state where the possibility that the user has any emotion has the same probability. , A facial expression and an action are stored. This data group corresponds to the “individuality data group” described above. Each feature amount included in the individuality data group is referred to as “individuality data”.

FIG. 5 is a diagram showing an example of the personality data group stored in the personality DB 62. For example, voice characteristics include voice pitch, loudness, utterance interval, etc., and individuality data corresponding to these elements is given as a value having a predetermined width defined by an upper limit and a lower limit. I have. For example, to describe a personality data group defined as “standard”, an upper limit value s4 and a lower limit value s4 ′ are defined as personality data for a voice element “height”, and the width of the personality data is s4 to s4. '. The same applies to other elements included in audio, images, and actions.

In FIG. 5, the personality data group designated as “standard” includes, for example, personality data statistically obtained from an unspecified number of subjects for each feature of voice, image, and motion. Is stored. Further, in FIG. 5, the individuality data groups corresponding to “A” and “B” respectively represent the individuality data groups generated corresponding to the users A and B described in FIG. . As described above, in the present embodiment, personality data groups corresponding to a plurality of users are generated and the personality DB 62
Registered in.

The individuality recognizing section 64 is provided with the voice recognizing section 5 described above.
2. Obtain a part or all of each feature amount extracted by each of the image recognition unit 56 and the motion recognition unit 60, search the personal identification data stored in the personality DB 62, and match the obtained feature amount. Check whether there is any personal identification data to be performed. If personal identification data that matches the acquired feature amount is found, the individuality recognition unit 64 reads out the individuality data group corresponding to the user having the individual identification data and reads out the emotion recognition units 66, 70, Output to 74. For example, as a result of searching personal identification data,
If it matches the user A, the personality data group corresponding to the user A is read.

If no personal identification data matching the acquired feature amount is found, the personality recognizing unit 64
The personal identification data is newly generated using the acquired feature amount, and a personality data group corresponding to the generated personal identification data is also newly generated and registered in the personality DB 62.

The emotion recognition unit 66 determines the state of the user's voice based on the characteristic amount of the voice extracted by the voice recognition unit 52 and the personality data group input from the personality recognition unit 64, and determines the state of the determined voice. The emotion of the user is recognized based on the state and the data table stored in the voice DB 68. As described above, in the present embodiment, the emotion recognition unit 6
6 (similarly for the emotion recognition units 70 and 74 described later) include “joy”, “anger”, “sad”,
Consider the four feelings of "easy".

The emotion recognition result by the emotion recognition unit 66 is, for example, "happy" is 30%, "anger" is 55%, and "sad".
Is 10% and "easy" is 5%, and the likelihood is given by the probability that the user has each emotion. In the example described above, there is a 30% possibility that the user has the emotion “happy”, and a 55% possibility that the user has the emotion “anger”.
%, The possibility of having the emotion “sorrow” is 10%, and the possibility of having the emotion “easy” is 5%, and the emotion that the user is likely to have is “ Anger ”.

The voice DB 68 stores a data table indicating the relationship between the user's voice state determined by the emotion recognition unit 66 and the above four emotions (happy, angry, sad, and easy). FIG. 6 is a diagram illustrating an example of a data table stored in the audio DB 68. As shown in FIG. 6, the state of the voice of the user is represented by a difference between the feature amount extracted by the voice recognition unit 52 and the personality data, and the appearance probabilities of the four emotions are corresponding to the difference. Given in percentage. For example, a description will be given of the “utterance interval” which is a voice element.
In the case where 0 ms is given, if the feature amount is within the range of the individuality data, “normal” is set, and the upper limit value (1
If the difference is larger than 00 ms), the utterance interval is long, so that it is "long". If it is smaller than the lower limit (80 ms) of the individuality data, it is short, so that the utterance interval is short. For each difference (long, normal, short), the probability of appearance of four emotions (happy, angry, sad, easy) is set.

Further, the emotion recognition unit 70 includes the image recognition unit 56
Feature and individuality recognition unit 6 related to facial expression extracted by
The state of the user's facial expression is determined based on the personality data group input from Step 4, and the emotion of the user is recognized based on the determined facial expression state and the data table stored in the image DB 72. The image DB 72 stores a data table indicating the relationship between the state of the user's facial expression determined by the emotion recognition unit 70 and the four emotions described above.

FIG. 7 is a diagram showing an example of a data table stored in the image DB 72. As shown in FIG. 7, the state of the facial expression of the user is represented by a difference between the feature amount extracted by the image recognition unit 56 and the personality data, and the appearance probabilities of the four emotions are corresponding to the difference. Given in percentage. For example, when the “eye angle” which is an element of the image is described, when the personality data for the “eye angle” included in the personality data group is given as 1 ° to 2 °, the characteristic amount becomes If it is within the range of the data, it is "normal", and if it is larger than the upper limit (2 °) of the personality data, it means that the eye is raised. Since it is falling, the difference from the personality data is set, such as "below", and the probability of appearance of four emotions is set for each difference (upper, normal, lower). .

The emotion recognition section 74 includes the motion recognition section 60.
Of feature and individuality recognizing unit 6 relating to motion extracted by
The state of the user's operation is determined based on the personality data group input from Step 4, and the emotion of the user is recognized based on the determined state of the operation and the data table stored in the operation DB 76. The motion DB 76 stores a data table indicating the relationship between the state of the user's facial expression determined by the emotion recognition unit 74 and the above-described four emotions (happy, angry, sad, and easy).

FIG. 8 is a diagram showing an example of a data table stored in the operation DB 76. As shown in FIG. 8, the state of the user's motion is represented by the difference between the feature amount extracted by the motion recognition unit 60 and the personality data, and the appearance probabilities of the four emotions are corresponding to this difference. Given in percentage. For example, to explain the “body temperature” which is an element of the motion, when the individuality data of “body temperature” included in the individuality data group is given as 35.8 ° C. to 36.4 ° C., the characteristic amount is If it is within the range of the data, it is "normal", and if it is larger than the upper limit of the personality data (36.4 ° C.), it means that the body temperature is high, so it is “high”; if it is smaller than the lower limit of the personality data (35.8 ° C.), the body temperature is Is low, so the difference from personality data is set, such as "low", and each difference (high, normal, low)
On the other hand, the probability of appearance of four emotions (happy, angry, sad, and easy) is set.

The weighting processing section 78 is provided for each of the emotion recognition sections 6.
A predetermined weighting process is performed on the emotion recognition results output from 6, 70, and 74 to integrate them, and a final emotion recognition result is obtained. Specifically, the emotion recognition result obtained based on the voice information is {Ah, Aa, As, Ae}, and the emotion recognition result obtained based on the image information is {Vh, Va, V
s, Ve}, the emotion recognition result obtained based on the motion information is {Mh, Ma, Ms, Me}, and the weighting coefficients corresponding to these are {Ph, Pa, Ps, P, respectively.
e｝, ｛Qh, Qa, Qs, Qe｝, ｛Rh, Ra, R
s, Re｝, the weighting process for each emotion recognition result is obtained by the following equations (1) to (4). Here, in the suffix added to each emotion recognition result and each weighting coefficient, “h” corresponds to “pleasure” and “a”
Corresponds to “anger”, “s” corresponds to “sad”, and “e” corresponds to “easy”.

Joy = Ph · Ah + Qh · Vh + Rh · Mh (1) Fury = Pa · Aa + Qa · Va + Ra · Ma (2) Sad = Ps · As + Qs · Vs + Rs · Ms (3) Easy = Pe · Ae + Qe · Ve + Re · Me (4) Each weighting coefficient is experimentally determined so as to obtain an appropriate recognition result, and is set to the following value, for example.

Ph = 41, Qh = 0, Rh = 5 Pa = 23, Qh = 0, Rh = 35 Ps = 0, Qs = 21, Rs = 10 Pe = 0, Qe = 30, Re = 7 Based on the expressions (1) to (4), an emotion recognition result obtained by integrating emotion recognition results obtained from each of voice, image, and motion is obtained. After that, the weighting processing unit 78
From the emotion recognition results obtained by the equations (1) to (4), the emotion having the largest value is selected, and this is output as the final emotion recognition result. The emotion recognition result obtained by the weighting processing unit 78 is output to the feature association processing unit 34 and the like together with the user identification result as an output from the emotion recognition device 14. Specifically, for example, information such as “the user is“ A ”and the emotion recognition result is“ happy ”” is output from the weighting processing unit 78.

The personality data correcting section 80 includes a personality recognizing section 64
The personality data group output from and the voice recognition unit 52,
Based on the respective feature amounts extracted by the image recognition unit 56 and the motion recognition unit 60, a process of correcting each individuality data included in the individuality data group in accordance with the individuality of the user is performed.

The above-described emotion recognition device 14 is used as an emotion determination unit, and the image feature extraction unit 12, time information extraction unit 18, music feature extraction unit 22, and point information extraction unit 28 are used as a plurality of feature extraction units and behavior pattern analysis units. , Trend extraction processing unit 3
8. The operation content setting unit 40 corresponds to the operation proposal unit, the image feature extraction unit 12, the point information extraction unit 28, and the room temperature information extraction unit 32 correspond to the environment determination unit, and the personality DB 62 and the personality recognition unit 64 correspond to the personality determination unit. are doing.

The on-vehicle system according to the present embodiment has such a configuration. Next, an operation in the case of proposing an action pattern according to the user's emotion and the like will be described. It should be noted that the result of extracting a specific user (for example, that the user is “A”) has already been notified from the emotion recognition device 14 to the feature association processing unit 34 and the tendency extraction processing unit 38. .

FIGS. 9 and 10 are flowcharts showing the operation procedure of the vehicle-mounted system. The image feature extraction unit 12 performs a process of extracting an image feature based on an image captured by the camera 10 (Step 100). The emotion recognition device 14 performs a predetermined emotion recognition process and outputs emotion information (step 101). Further, the time information extracting unit 18
Performs a process of extracting time information based on the current time acquired from the clock 16 (step 102).

[0053] The music feature extraction unit 22 is provided with the audio device 2.
A process for extracting music features based on the music being reproduced by 0 is performed (step 103). As described above, when music is not listened to, the state is extracted as a music feature.

The point information extracting unit 28 performs a process of extracting point information from the navigation device 24 (step 1).
04). Further, the room temperature information extracting unit 32 extracts information (room temperature information) on the room temperature in the vehicle from the air conditioner 30 (step 105). The order of the processes shown in steps 100 to 105 can be arbitrarily changed, and a plurality of processes can be performed in parallel.

When the predetermined feature extraction processing shown in steps 100 to 105 is performed and various information related to the user is extracted, the feature association processing unit 34 converts the information into a set of feature data. A predetermined feature association process is performed (step 106).

Next, the feature association processing unit 34
Check the number of stored feature data stored in B36,
It is determined whether the number of stored cases is 0 (step 10).
7). If one or more pieces of feature data have already been stored, a negative determination is made in step 107, and then the feature association processing unit 34 determines whether the feature data obtained in the processes shown in steps 100 to 105 (hereinafter referred to as feature data) This is referred to as “current feature data” and the feature data obtained in the previous processing (hereinafter, referred to as “previous feature data”)
Is determined (step 108).
If the current feature data matches the previous feature data, that is, if there is no change in the content of the feature data, an affirmative determination is made in step 108, and in this case, the process returns to step 100 and the predetermined feature is determined. The extraction process is continued.

If the current feature data does not match the previous feature data, that is, if the content of the feature data has changed, a negative determination is made in step 108, and then the feature association processing unit 34 is a feature DB 36
Then, it is determined whether or not the number of stored feature data already stored in is equal to or greater than a predetermined number (step 109). Specifically, the process shown in step 109 is a process for determining whether or not a certain amount of feature data is stored in the feature DB 36. For example, whether or not 100 or more feature data items are stored It is determined whether or not.

If the number of stored feature data is equal to or greater than the predetermined number, an affirmative determination is made in step 109, and the feature association processing unit 34 outputs the current feature data to the trend extraction processing unit 38. The trend extraction processing unit 38 that has acquired the current feature data sets the current feature data and the feature DB 36
Is compared with all the feature data stored in (step 110), and it is determined whether or not there is feature data whose contents match all the items with the current feature data (step 111). The "items" referred to here are "image characteristics", "emotional information", "time information", "music characteristics",
It corresponds to the six “point information” and “room temperature information” (see FIG. 2).

If there is no feature data in which the contents of all the items match, a negative determination is made in step 111, and then the tendency extraction processing unit 38 determines that the contents other than the "emotional information" are one. It is determined whether there is any matching feature data (step 112).

If there is no feature data in which all items other than “emotion information” match, a negative determination is made in step 112, and then the tendency extraction processing unit 38 determines the number of items whose contents match. Is greater than or equal to a predetermined number (step 113). Here, for example, it is determined whether or not five or more of the six items described above match. In other words, in step 113, it is determined whether there is feature data whose content is similar to the current feature data.

If the number of items whose contents match is equal to or greater than the predetermined number, an affirmative determination is made in step 113, and the tendency extraction processing unit 38 determines whether the items are inconsistent or missing (items for which no information is obtained). ), The corresponding feature data is read from the feature DB 36 and output to the operation content setting unit 40. For example, the current feature data is similar to the feature data of ID “1” shown in FIG.
If the contents of the point information only do not match, the point information (coordinate information “X1, Y1”, road type “expressway”, point type “sea”) stored in the feature data of ID “1” is used. Is read.

The operation content setting section 40 includes the trend extraction processing section 3
Based on the feature data output from step 8, a process for proposing an operation content corresponding to a mismatched item or a missing item is performed (step 114). Specifically, for example, if the spot information is output as the feature data corresponding to the mismatched item or the missing item, the operation content setting unit 40
Proposes to the user using the display device 26 provided in the navigation device 24 to perform route guidance using the point corresponding to the given point information as the destination. Similarly, the operation content setting unit 40 makes a suggestion on the music to be played when the music feature is output, and makes a suggestion on the set temperature of the room temperature in the vehicle compartment when the room temperature information is output. When a user gives a positive instruction using an operation unit (not shown), the operation content setting unit 40 sends a predetermined operation instruction to the audio device 20, the navigation device 24, or the like. As a result, an operation corresponding to the content of the proposal is realized. The processing shown in steps 110 to 114 described above is processing for proposing the operation content (proposing processing).

In the case where the proposal of the predetermined operation content shown in step 114 is made, in the case where a negative decision is made in step 109 described above, in the case where an affirmative decision is made in step 111, or when a positive decision is made in step 111, an affirmative decision is made in step 112 If
In any of the cases where a negative determination is made in step 113, the feature association processing unit 34 determines whether or not feature data having the same content in all items as the current feature data is already stored in the feature DB 36. (Step 115).

If the feature data having the same content has not been stored yet, a negative determination is made in step 115, and the feature association processing unit 34 newly adds this feature data to the feature D.
B36 (step 116), and the number of stored feature data is updated (step 117). Also, feature DB3
Even if no feature data has been stored yet in step 6 and a positive determination is made in step 107 described above, the processing of step 116 and thereafter is also performed.

If feature data having the same content as the current feature data has already been stored, an affirmative determination is made in step 115 described above, and the feature association processing unit 34 determines whether the feature data already stored is present. Then, for those having the same contents as the current feature data, the value of the appearance frequency is updated (step 118). Step 117
Or, after the processing shown in step 118 is performed,
Returning to step 100 described above, the subsequent processing is repeated. The processing shown in steps 115 to 118 described above is a learning processing for updating the contents of the feature data stored in the feature DB 36.

As described above, in the on-vehicle system according to the present embodiment, each of the user's emotions, behavior patterns, and surrounding environment is analyzed, and the analysis results are stored in the feature DB 36 as characteristic data relating to the user. When each of the user's emotions, behavior patterns, and surrounding environment is newly analyzed, the operation data corresponding to the mismatched or missing item is referred to by referring to the characteristic data stored in the characteristic DB 36. Has been proposed. Therefore, it is possible to predict an action pattern desired by the user and make an accurate proposal, and to propose an action pattern in consideration of the user's emotions and personality.

The present invention is not limited to the above embodiment, and various modifications can be made within the scope of the present invention. For example, in the embodiment described above,
Image features, emotion information, time information, music features, point information, and room temperature information were extracted as items of user-related feature data, but the contents of the extracted feature data items are limited to these. Rather, it may be any characteristic relating to the user himself or the surrounding environment where the user is located.

In the above-described embodiment, when it is determined whether or not to newly store the feature data in the feature DB 36 (see step 115 in FIG. 10), the content of the feature data differs even for one item. In such a case, the characteristic data is newly stored. However, when characteristic data having very similar contents (for example, data that differs only by one item) is already stored, the similar data is stored. The appearance frequency of a certain feature data may be increased by one. As described above, in the case where feature data having very similar contents are regarded as the same and only the appearance frequency is updated, the number of pieces of feature data stored in the feature DB 36 can be reduced. The storage capacity of the hard disk device or the like constituting the feature DB 36 can be reduced.

Further, in the above-described embodiment, the case where the behavior pattern processing apparatus of the present invention is applied to an in-vehicle system has been described. However, the application range of the present invention is not limited to this, and other various The present invention can be applied to a system. Specifically, for example, if the present invention is applied to an animal-type robot, it is possible to realize an animal-type robot that takes a more natural reaction behavior in consideration of the emotions and behavior patterns of the user. it can.

[0070]

As described above, according to the present invention, it is possible to cope with various situations by storing the association data obtained by extracting and associating the characteristics of each action content with the user's action pattern. As a result, data relating to a user-specific behavior pattern can be accumulated.
Therefore, when a feature of the user's action content is newly extracted, an action pattern desired by the user can be predicted based on the accumulated association data, so that an accurate proposal can be made.

Also, by storing association data in which the user's emotions and behavior patterns are associated, it is possible to accumulate data relating to user-specific emotion changes and behavior patterns corresponding to various situations. Therefore, when the user's emotion is newly determined and the behavior pattern is analyzed, the action content is proposed based on the accumulated association data, thereby proposing the behavior pattern in consideration of the user's emotion. It can be performed.

[Brief description of the drawings]

FIG. 1 is a diagram showing an overall configuration of a vehicle-mounted system according to an embodiment.

FIG. 2 is a diagram illustrating the contents of feature data stored in a feature DB.

FIG. 3 is a diagram showing a detailed configuration of an emotion recognition device.

FIG. 4 is a diagram showing an example of personal identification data stored in a personality DB.

FIG. 5 is a diagram showing an example of a personality data group stored in a personality DB.

FIG. 6 is a diagram illustrating an example of a data table stored in a voice DB.

FIG. 7 is a diagram illustrating an example of a data table stored in an image DB.

FIG. 8 is a diagram showing an example of a data table stored in an operation DB.

FIG. 9 is a flowchart showing an operation procedure of the vehicle-mounted system.

FIG. 10 is a flowchart showing an operation procedure of the vehicle-mounted system.

[Explanation of symbols]

 Reference Signs List 10 camera 12 image feature extraction unit 14 emotion recognition device 16 clock 18 time information extraction unit 20 audio device 22 music feature extraction unit 24 navigation device 26 display device 28 point information extraction unit 30 air conditioner 32 room temperature information extraction unit 34 feature association processing unit 36 Feature database (DB) 38 Trend extraction processing unit 40 Operation content setting unit 62 Individuality database (DB) 64 Individuality recognition unit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) B25J 13/00 G06F 3/00 601 G06F 3/00 601 G10L 3/00 531N F term (Reference) 2C150 CA01 CA02 CA04 DA04 DA05 DA24 DA25 DA26 DA27 DA28 DF03 DF04 DF06 DF33 ED42 ED52 EF07 EF16 EF23 EF29 EF33 EF36 3C007 KS11 KS39 LW03 MT14 WB16 WB17 WB19 WB22 5B075 ND20 NK06 PP28 AU30 A30A30 A501 A501 A30A30 A501 A501A30

Claims (6)

[Claims] 1. A plurality of feature extracting means for extracting a feature of each action content from a user's action pattern including a plurality of action content, and a feature extracted by each of the plurality of feature extracting means. A feature database that stores the associated data; and a new feature extraction unit that extracts a feature of the user's operation based on the association data stored in the feature database. And a motion suggesting means for proposing at least one of the motion contents of the above (1). 2. An emotion determining means for determining an emotion of a user, an action pattern analyzing means for analyzing an action pattern of the user including a plurality of operation contents, and an action pattern of the user determined by the emotion determining means. A feature database that stores association data in which the emotion and the user's behavior pattern analyzed by the behavior pattern analysis unit are stored; and a new emotion determination unit determines the user's emotion, When the behavior pattern of the user is analyzed by the behavior pattern analysis means, based on the association data stored in the feature database, an operation proposal means for proposing at least one of the plurality of operation contents. A behavior pattern processing device comprising: 3. The operation suggestion unit according to claim 1, wherein the operation suggestion unit does not match a part of the plurality of operation contents included in the new action pattern with the plurality of operation contents included in the association data. Alternatively, the behavior pattern processing device is characterized in that the proposal is made when the information is missing. 4. The behavior pattern processing according to claim 1, wherein the storage of the association data in the feature database is performed when the content of the association data repeatedly acquired changes. apparatus. 5. The apparatus according to claim 1, further comprising an environment determination unit configured to determine a surrounding environment of the user, wherein the association data stored in the feature database includes the environment determination unit. A behavior pattern processing device, wherein the surrounding environment determined by the above is associated. 6. The method according to claim 1, further comprising: a personality determination unit configured to determine a personality of the user and extract a specific user from predetermined candidates. The action pattern processing device, wherein the proposal means sets the content of the proposal for each of the specific users extracted by the personality determination means.