CN104244824A

CN104244824A - Affect-monitoring system

Info

Publication number: CN104244824A
Application number: CN201380019356.3A
Authority: CN
Inventors: 蜂须贺知理; 野添聪司; 盐谷武司; 坂井诚
Original assignee: Denso Corp
Current assignee: Denso Corp
Priority date: 2012-04-10
Filing date: 2013-04-04
Publication date: 2014-12-24
Anticipated expiration: 2033-04-04
Also published as: JP2013216241A; CN104244824B; DE112013001979T5; US20150078632A1; US9465978B2; WO2013153781A1; JP5729345B2

Abstract

An affect-monitoring system (1) characterized in being provided with: a facial image-acquiring unit (3) capable of acquiring facial images of occupants (101) seated at two or more seating positions of a vehicle; an affect information-acquiring unit (13), which acquires affect information representing the affect of the occupants on the basis of the facial images acquired by the facial image-acquiring unit; and an affect display-generating unit (15), which generates an affect display that corresponds to the affect information acquired by the affect information-acquiring unit.

Description

Emotion surveillance

The cross reference of related application

The Japanese patent application 2012-89493 that the application submitted to based on April 10th, 2012, quotes its contents at this.

Technical field

The application relates to a kind of emotion surveillance.

Background technology

In patent documentation 1, disclose following technology.Obtain the facial expression of the multidigit passenger in vehicle, the good degree of the expression of each passenger is learnt with car external environment with being associated, and detects according to sequential.Further, propose that passenger's expression remains good travel route (car external environment).

At first technical literature

Patent documentation

Patent documentation 1: JP 2011 – No. 117905 publications

Summary of the invention

In the technology described in patent documentation 1, the information of the facial expression of regarding passengers only supports process for route selection, and do not show, therefore each passenger cannot understand the emotion of other passengers.The application puts to propose in view of the above problems, its object is to provide a kind of emotion surveillance that each passenger can be made to understand the emotion of other passengers.

The emotion surveillance of the one side of the application possesses: face-image obtaining section, for the plural seating position of vehicle, can obtain the face-image of the passenger occupying this seating position; Emotional information obtaining section, obtains the emotional information of the emotion representing passenger based on the face-image obtained by face-image obtaining section; And emotion display preparing department, make the emotion corresponding with the emotional information obtained by emotional information obtaining section and show.

When the emotion surveillance of this aspect such as has a multidigit passenger in vehicle, the emotion display of the emotion representing each passenger can be made.Therefore, even if passenger such as occupies the position that other passengers cannot directly see, the abnormal conditions etc. of that passenger also can be noticed like a cork.

Accompanying drawing explanation

By while with reference to accompanying drawing while carry out following detailed description, about the above-mentioned purpose of the application and other objects, feature or advantage will become more clear and definite.

Fig. 1 is the block diagram of the formation representing emotion surveillance,

The flow chart of the main process of Fig. 2 performed by expression emotion surveillance,

The flow chart of the emotion determination processing of Fig. 3 performed by expression emotion surveillance,

The flow chart that Fig. 4 processes for the emotion display making represented performed by emotion surveillance,

Fig. 5 is the key diagram of the AU (Action Unit: motor unit) representing passenger's face,

Fig. 6 is the key diagram of the indication example representing emotion display,

Fig. 7 is the key diagram of the indication example representing emotion display,

Fig. 8 is the key diagram of the indication example representing emotion display,

Fig. 9 is the key diagram of the indication example representing emotion display.

Detailed description of the invention

1. the formation of emotion surveillance 1

The formation of emotion surveillance 1 is described based on Fig. 1.Emotion surveillance 1 is for being equipped on the onboard system of vehicle, possess camera (camera, face-image obtaining section) 3, seating position test section 5, passenger identification part 7, storage part 9, feature amount extraction module 11, emotion detection unit (emotional information obtaining section) 13, emotion display preparing department 15, storage part (the 1st storage part, the 2nd storage part) 17 and interface (interface) portion 19.Seating position test section 5, passenger identification part 7, feature amount extraction module 11, emotion detection unit 13 and emotion display preparing department 15 are made up of known computer.

Camera 3 is arranged at the ceiling of car indoor, all seating positions (the right seat of driver's seat, co-pilot seat, rear side, the left seat of rear side etc.) of its image pickup scope covering car indoor.Thus, camera 3 can occupy face and the health of the passenger 101 of this seating position for all seating position shootings of car indoor.In addition, also camera 3 can be set one by one for each seating position.Now, each camera 3 shooting occupies face and the health of a passenger 101 of corresponding seating position.In addition, as camera 3, the camera at the camera in the front (driver's seat and co-pilot seat) of shooting car indoor and the rear (rear side seat) of shooting car indoor also can be set.

Seating position test section 5 detects the seating position of passenger 101 according to the image taken by camera 3.Namely, by image recognition, judge in the image taken by camera 3, at each seating position, whether passenger 101 takes a seat.

Passenger identification part 7 uses the image taken by camera 3, identifies passenger 101.Be described in detail later.

Storage part 9 stores the identifier for identifying passenger 101.This identifier is the intrinsic feature of each passenger that can be extracted from image by image recognition, such as, the pattern (shape of the facial key element of eye, nose, mouth etc. or size, facial key element position relationship etc. each other) of face, the shape of clothes or color, build etc. can be listed.Storage part 9 stores the identifier of the passenger 101 in the past registered for each passenger.In addition, in storage part 9, the title etc. of seating position, emotional information described later and passenger can be stored with being associated with identifier.In addition, storage part 9 stores emotion detection unit 13 the data base judging the reference of emotion time institute.

In addition, in this application, " information " both can be used as uncountable noun use and also can be used as countable noun use.

Feature amount extraction module 11 according to the face of the image recognition passenger 101 taken by camera 3, and then, in its face, extracted the characteristic quantity (AU) preset by image procossing.If identify the face of multidigit passenger 101 in the image taken by camera 3, then extract characteristic quantity (AU) for individual passenger.In addition, the details of Characteristic Extraction is aftermentioned.

Emotion detection unit 13 judges the emotion of passenger 101 according to the characteristic quantity extracted by feature amount extraction module 11 (AU), makes emotional information.If identify the face of multidigit passenger 101 in the image taken by camera 3, then make emotional information for individual passenger.In addition, the details of the making of emotional information is aftermentioned.

Emotion display preparing department 15 uses the emotional information made by emotion detection unit 13, makes emotion display.In addition, emotion display preparing department 15 also can make notification voice.The details of the making of emotion display and notification voice is aftermentioned.

Storage part 17 stores the mapping (corresponding relation of the form that the benchmark relevant with the kind of emotional information, the intensity of emotion and emotion show) that emotion display preparing department 15 uses when making emotion display and notification voice.In this mapping, there are two kinds of mappings, namely (i) stores by every passenger 101 and establishes with the identifier of this passenger 101 mapping associated, and the mapping that (ii) is general.Content about mapping can describe in detail later.

Interface portion 19 possess the display 19a that can show image, the speaker 19b that can export sound, can sound import mike 19c and accept operating portion (keyboard, various switch, the contact panel etc.) 19d of input operation of passenger 101.Interface portion 19 shows the emotion display of emotion display made by preparing department 15 on display 19a.In addition, the notification voice made by emotion display preparing department 15 is sent by speaker 19.In addition, by for the Speech input of mike 19c or the input operation for operating portion 19d, can the mapping being stored in storage part 17 be edited (adding, for the additional or renewal of the content of existing mapping, the deletion etc. of mapping of new mapping).

The display (such as other passengers are difficult to the display watched, maybe cannot see) that the driver that the display 19a of interface portion 19 can be set to vehicle can observe.As such display 19a, such as, the dual screen display frame etc. of the display frame of instrument, navigation screen can be listed.In addition, display 19a also can be the display (such as driver is difficult to the display watched, maybe cannot see) that the some or all passengers beyond driver can watch.As such display 19a, such as, can list and be arranged on the rear side of seat or the display of ceiling.In addition, display 19a also can be the display that driver and other passenger both sides all can watch.

2. the process performed by emotion surveillance 1

Based on Fig. 2 ~ Fig. 8, the process performed by emotion surveillance 1 is described.

At this, the flow chart described in this application or the process of flow chart are made up of multiple sections (or perhaps steps), and each section such as shows as S1.And then each section can be divided into multiple subsegment, on the other hand, also multistage merging a section can be formed.And then each section that forms like this can as device, module, means.

Emotion surveillance 1 is " opening " period in the ignition switch of the vehicle being equipped with this system, repeatedly performs the process shown in Fig. 2 in required time.In the S1 of Fig. 2, camera 3 obtains the image of the scope of all seating positions covering vehicle.If passenger occupies the arbitrary seating position in vehicle, then include face and the health of this passenger in described image.In addition, if passenger occupies the plural seating position in vehicle respectively, then comprise face and the health of all passengers in described image.

In S2, by each seating position of vehicle, seating position test section 5, by carrying out image analysis to the image obtained at described S1, judges whether passenger takes a seat.

In S3, passenger identification part 7, for being judged as all seating positions that passenger takes a seat in described S2, performs the identification of the passenger taken a seat as follows.First, for being judged as the seating position α that passenger takes a seat in described S2, be used in the identifier that image zooming-out that described S1 obtains occupies the passenger of this seating position α.Then, the identifier of extraction and the identifier being stored in storage part 9 are contrasted.With regard to the result of contrast, if among the identifier being stored in storage part 9, there is identical with the identifier of the passenger occupying seating position α or akin identifier, then this identifier is stored in storage part 9 with being associated with seating position α.On the other hand, with regard to the result of contrast, if among the identifier being stored in storage part 9, there is not identical with the identifier of the passenger occupying seating position α or akin identifier, then the identifier occupying the passenger of seating position α is newly stored in storage part 9 with being associated with seating position α.

Not only for seating position α, for being judged as that in described S2 all seating position β, γ that passenger takes a seat perform above-mentioned process similarly.

In S4, for being judged as that in described S2 the seating position that passenger takes a seat, execution obtain the emotion determination processing of the emotional information of the emotion representing passenger.The seating position that passenger takes a seat if be judged as in described S2 exists multiple, then obtain the emotional information of passenger for this all multiple seating position.

Based on Fig. 3, this emotion determination processing is described.In the S11 of Fig. 3, repeatedly obtained the image of all seating positions of covering car indoor in required time by camera 3.

In S12, feature amount extraction module 11 is by carrying out image analysis for each image repeatedly obtained in described S11 (comprising the face-image of the face of passenger), extract the features of the face of passenger (such as, eyebrow, eyelid, mouth), and then, extract the action (hereinafter referred to as AU (Action Unit: motor unit)) of the features with time process.As AU, there is the unit shown in table 1 and Fig. 5.

In addition, R, L in Fig. 5 refer to the right side, the left side of face.

[table 1]

It is widely known that there is dependency relation (the facial behavior coding system that Ai Keman proposes) between the emotion and AU of people.

In S13, emotion detection unit 13 judges the emotion of passenger according to the AU extracted in described S12.Specifically, carry out as following.Storage part 9 prestores the data base of the corresponding relation showing AU and emotion.This data base such as stores as following corresponding relation.

It is glad: 6R,

6L，12R，12L，25

Be taken aback: 1R,

1L，2R，2L，5R，5L，25，27

It is sad: 1R,

1L，4R，4L，15R，15L，17

By the AU extracted in described S12 is applicable to this data base, the kind (glad, sad etc.) of the emotion of passenger can be judged.In addition, the intensity of emotion can be judged by the size of AU.Using the information of the intensity of the kind and emotion that comprise this emotion as emotional information.For being judged as all seating positions that passenger takes a seat in described S2, make emotional information.In addition, emotional information is stored in storage part 9 with being associated with corresponding seating position and identifier.Such as, the emotional information occupying passenger's (its identifier is set to identifier X) of seating position α is stored in storage part 9 with being associated with seating position α and identifier X.As more than, feature amount extraction module 11 obtains the emotional information representing passenger emotion based on the face-image obtained by camera 3.

Be back to Fig. 2, in S5, emotion display preparing department 15 carries out making the emotion that shows of the emotion corresponding with the emotional information obtained in described S4 and shows to make and process.Illustrate that the display of this emotion makes process based on Fig. 4.In the S21 of Fig. 4, be invoked at the emotional information being stored in storage part 9 in described S13.If emotional information exists multiple, call all.

In S22, call mapping from storage part 17.Carry out as following in detail.First, for being judged as the seating position α that passenger takes a seat in described S2, from storage part 9, read the identifier stored with being associated with this seating position α.Then, search in storage part 17 and establish with the identifier of this reading the mapping associated, if there is the mapping establishing association, read this mapping.On the other hand, if there is not the mapping establishing association, general mapping is read.If be judged as in described S2, the seating position that passenger takes a seat also has other except seating position α, then call mapping similarly for all seating positions beyond seating position α.

At this, map as shown in table 2 ~ table 4.

[table 2]

The mapping of passenger (identifier X)

Categories of emotions	The making of emotion display	Show Color	?	Notification voice
					Smiling face's (happiness)	○	Pink	Below L5 bright light, more than L6 glimmers	More than L6
Angry (irritability)	○	Red	Below L5 bright light, more than L6 glimmers	More than L6
					Be taken aback (error)	○	Blue	Below L5 bright light, more than L6 glimmers	More than L6
Sad (face of crying)	×	―	―	―
					Be weary of	○	Green	Below L5 bright light, more than L6 glimmers	More than L6
Puzzled	×	―	―	―
					Sleepy	○	Purple	Below L5 bright light, more than L6 glimmers	More than L6

[table 3]

The mapping of passenger (identifier Y)

Categories of emotions	The making of emotion display	Show Color	Bright light or flicker	Notification voice
					Smiling face's (happiness)	○	Pink	Below L6 bright light, more than L7 glimmers	More than L8
Angry (irritability)	○	Red	Below L6 bright light, more than L7 glimmers	More than L8
					Be taken aback (error)	○	Blue	Below L6 bright light, more than L7 glimmers	More than L8
Sad (face of crying)	○	Orange	Below L6 bright light, more than L7 glimmers	More than L8
					Be weary of	○	Green	Below L6 bright light, more than L7 glimmers	More than L8
Puzzled	○	Yellow	Below L6 bright light, more than L7 glimmers	More than L8
					Sleepy	○	Purple	Below L6 bright light, more than L7 glimmers	More than L8

[table 4]

General mapping

Categories of emotions	The making of emotion display	Show Color	Bright light or flicker	Notification voice
					Smiling face's (happiness)	○	Pink	Below L7 bright light, more than L8 glimmers	More than L8
Angry (irritability)	○	Red	Below L7 bright light, more than L8 glimmers	More than L8
					Be taken aback (error)	×	―	―	―
Sad (face of crying)	×	―	―	―
					Be weary of	×	―	―	―
Puzzled	×	―	―	―
					Sleepy	○	Purple	Below L7 bright light, more than L8 glimmers	More than L8

Being mapped as of table 2 establishes with certain identifier X the mapping associated, and being mapped as of table 3 establishes with the identifier Y being different from identifier X the mapping associated, table 4 be mapped as the general mapping not establishing with any identifier and associate.

Each mapping comprises the benchmark relevant with the kind that should make the emotional information that emotion shows.Namely, each emotional information mapping part categories of emotions among by all kinds with emotion is defined as the emotional information making emotion display, the emotional information with other categories of emotions is defined as the emotional information (" making of the emotion display " row in table 2 ~ 4) not making emotion display.Content in " making of emotion display " row both can be different by each mapping (namely by each identifier), also can be common.

Such as, in the mapping of table 2, have the smiling face's (happiness) indicating "○" in " emotion display making " row, angry (irritability), be taken aback (error), be weary of, the emotional information of any one sleepy categories of emotions is the emotional information making emotion display, have during " making of emotion display " arranges indicate the sadness (face of crying) of "×", the emotional information of any one categories of emotions of puzzlement is do not make the emotional information that emotion shows.The mapping of table 3, table 4 too.

In addition, each mapping defines the display form (Show Color, bright light or flicker) of emotion display.Namely, in each mapping, each categories of emotions for emotional information in " Show Color " row specifies the color that emotion shows in advance, the relation of the form of bright light/flicker that the intensity (represent with 10 stages of L1 ~ L10, the larger then emotion of numerical value is stronger) that " bright light or flicker " row define the emotion in emotional information shows with emotion.Such as, in the mapping of table 2, be defined as, based on categories of emotions be smiling face's (happiness) emotional information make emotion display color be pink, intensity emotion display bright light (brightness is fixed) within the scope of L1 ~ L5 of emotion, emotion display flicker in L6 ~ L10.The mapping of table 3, table 4 too.

In S23, make emotion display and notification voice in the following manner.For being judged as the seating position α that passenger takes a seat in described S2, according to the mapping read in described S22 and establish with seating position α the emotional information associated and make emotion and show.For this reason, first, judge whether the categories of emotions in emotional information is the kind (namely, meeting the benchmark relevant with the kind of emotional information) indicating "○" in " making of the emotion display " hurdle mapped.If indicate the kind of "○", then make emotion display for this emotional information.On the other hand, if indicate the kind of "×", then do not make emotion display for this emotional information.

When making emotion display, determine the design of emotion display according to the categories of emotions in emotional information.In addition, the design that categories of emotions and emotion show is established in advance to be associated, and is stored in storage part 17.Such as, the design being designed to the emotion display 217 in Fig. 6 of emotion display when categories of emotions is smiling face (happiness), the design being designed to the emotion display 219 in Fig. 6 that categories of emotions is emotion display time angry (irritability), the design being designed to the emotion display 221 in Fig. 6 of emotion display when categories of emotions is sleepy.In addition, when making emotion shows, according to the form (form that emotion shows) mapping the color of decision emotion display and the intensity decision bright light/flicker according to the emotion in emotional information.

In addition, when making emotion display for seating position α, the position based on the seating position α of car indoor determines the display position of emotion display.Specifically, if seating position α is driver's seat, the display position that emotion shows is set to the position corresponding with driver's seat, if seating position α is co-pilot seat, the display position that emotion shows is set to the position corresponding with co-pilot seat, if seating position α is the right seat of rear side, the display position that emotion shows is set to position corresponding to seat right with rear side, if seating position α is the left seat of rear side, the display position that emotion shows is set to position corresponding to seat left with rear side.

According to more than, determine emotion display (form of design, color, bright light/flicker) and its display position of seating position α.If be judged as in described S2, the seating position that passenger takes a seat also has other except seating position α, then perform the making process of emotion display and its display position similarly for all seating positions beyond seating position α.

In addition, in this S23, the presence or absence of the sounding of notification voice is also determined according to emotional information.As long as among the emotional information making emotion display, the condition that " notification voice " that have the intensity of the emotion of to meet mapping arranges, just determines the sounding of notification voice.Notification voice can be set as according to meeting the kind of emotion of the condition that " notification voice " arranges and different.On the other hand, if among the emotional information making emotion display, the condition that " notification voice " that do not have the intensity of the emotion of to meet mapping arranges, then determine sound of not giving notice.The identifier of the passenger of the condition that the kind of " notification voice " or volume arrange according to " notification voice " that the intensity of emotion meets mapping set also can, set according to the seating position of this passenger and also can.

Be back to Fig. 2, in S6, emotion display preparing department 15 will represent that the signal of the content (emotion shows the presence or absence with the sounding of its display position, notification voice) determined in described S23 exports interface portion 19 to.If the emotion display made exists multiple, then export all emotion displays.

In addition, in S6, there is the seating position that passenger does not take a seat if judge in described S2, then export the positional information of this seating position to interface portion 19.The seating position that this positional information does not take a seat for showing passenger in interface portion 19.In addition, in described S23, be judged as that if exist passenger takes a seat but do not make the seating position of emotion display, then export the positional information of this seating position to interface portion 19.This positional information is used for showing passenger in interface portion 19 and takes a seat but the seating position not making emotion display.

In addition, in interface portion 19, the signal according to inputting from emotion display preparing department 15 carries out following display or sounding.Namely, the display 19a of interface portion 19 such as shown in Figure 6, display emotion display 215,217,219,221.In addition, Fig. 6, for detect passenger at all 4 seating positions, makes indication example when emotion shows at all seating positions.

The design of emotion display 215,217,219,221, the form of color and bright light/flicker determine in described S23.In addition, emotion display 215,217,219,221 is based on the display position determined in described S23, and the seating position corresponding to showing with each emotion shows with being associated.Such as, emotion display 215 makes according to the emotional information of the passenger (driver) occupying the driver's seat among seating position, and this emotion display 215 is shown in the position 205 schematically showing driver's seat.Similarly, according to occupy co-pilot seat passenger emotional information and the emotion display that makes 217 is shown in the position 207 schematically showing co-pilot seat, according to occupy rear side right seat passenger emotional information and make emotion display 219 be shown in schematically show rear side right seat position 209, according to occupy rear side left seat passenger emotional information and make emotion display 221 be shown in schematically show rear side left seat position 211.

In addition, interface portion 19 display 19a by represent the line 203 of the profile of car room, the arrow 213 representing the direct of travel of vehicle, represent steering wheel mark 215 etc. respectively performance represent the position 205 of driver's seat, vice driver's seat position 207, represent the position 209 of the right seat of rear side and represent the position 211 of the left seat of rear side.

In addition, Fig. 7 for detect passenger at all 4 seating positions, but only makes emotion at driver's seat and co-pilot seat and shows, and the indication example when rear side seat does not make emotional information.Now, schematically showing the position 209 of the right seat of rear side and schematically showing position 211 place of the left seat of rear side among the display 19a of interface portion 19, although distinctive display when carrying out detecting that passenger does not make emotional information.This is shown as inside is blank circle 225,227 (showing all different and also different with display when not detecting passenger displays from any emotion).

In addition, Fig. 8, for detect passenger at driver's seat and co-pilot seat, makes emotion display at this place, and the indication example when rear side seat does not detect passenger.Now, schematically showing the position 209 of the right seat of rear side and schematically showing position 211 place of the left seat of rear side among the display 19a of interface portion 19, does not show (show all different from any emotion and be the also different display of circle 225,227 of blank with inside) whatever.

In addition, to give the alarm sound for emotion shows preparing department 15 if determine, then interface portion 19 to be given the alarm sound by speaker 19b.

In S7, judge whether the ignition switch of vehicle is in " opening ".If " open " and then enter S1, if " closedown " then terminates present treatment.

In addition, be stored in storage part 17 mapping can by by the Speech input of passenger to the mike 19c of interface portion 19, or the operating portion 19d in operation interface portion 19 carries out changing, reseting.If the midway of the process shown in Fig. 2 is changed, reseted mapping, then interrupt this process, be back to S1.In addition, by Speech input to the change mapped, reset can often time carry out, by the operation of operating portion 19d to the change of mapping, reset, can only carry out when vehicle stops.

3. the effect that plays of emotion surveillance 1

(1), when emotion surveillance 1 has a multidigit passenger in vehicle, the emotion display of the emotion representing each passenger can be made.Therefore, passenger can notice the abnormal conditions of other passengers occupying the position that cannot directly see like a cork.Especially, driver can keep the attention needed for driving to grasp the same situation taking advantage of personnel.

(2) emotion surveillance 1 emotion display is shown with this emotion corresponding to the seating position of passenger show with being associated.Therefore, it is possible to understand the corresponding relation of emotion display and passenger like a cork.

(3) emotion surveillance 1 possesses the mapping defining the benchmark relevant with the kind of emotional information, makes emotion display for the emotional information meeting this benchmark.Therefore, it is possible to the display of the emotion of making to be defined as the high emotion display of importance degree.In addition, store above-mentioned mapping for each passenger, therefore, it is possible to the emotion display shown by determining according to the situation of each passenger.

(4) emotion surveillance 1 possesses the mapping of the corresponding relation of the form that the intensity that defines emotion and emotion show, can based on this mapping, according to the metamorphosis that the intensity of emotion makes emotion show.Therefore, it is possible to shown by emotion, understand the intensity of emotion like a cork.In addition, store above-mentioned mapping for each passenger, therefore, it is possible to the form shown according to the situation determination emotion of each passenger.

(5) emotion surveillance 1 identifies passenger, according to its recognition result, uses corresponding mapping.Therefore, even if passenger is when seating position is changed in midway, also the mapping corresponding with this passenger can be used.

4. other embodiment

(1) such as, emotion surveillance 1 only detects driver as passenger, be specific kind with the categories of emotions of the emotional information of this driver and the intensity of emotion for more than defined threshold for condition, by the speaker 19b of interface portion 19 send mechanicalness synthetic video (such as " what's the matter? ") also can.

(2) in addition, emotion surveillance 1 be specific kind with the categories of emotions of the emotional information of passenger and the intensity of emotion for more than defined threshold for condition, part or all of proposing to establish in advance with this passenger the emotion comfortableization portion (such as air-conditioning adjusts, volume adjusts, song adjusts, greeting etc.) associated to this passenger or other passengers also can.

(3) in addition, to obtain the emotional information of the categories of emotions with regulation for condition from the passenger of the regulation ratio exceeded in the passenger that detects, carry out notifying that (such as by the speaker 19b ring buzz of interface portion 19) also can.

(4) when arranging camera 3 for each seating position, the power-off of the camera 3 not detecting passenger can be made.

(5) detection of the passenger of seating position is undertaken also can by the sensor (such as seat sensor) being arranged at each seating position.

(6) emotion surveillance 1 does not possess interface portion 19 and also can.Now, preparing department's 15 pairs of external equipment (such as onboard navigation system, portable terminal device etc.) output signals can be shown from emotion, by the display of this external equipment display emotion, sound of giving notice.The display of external equipment can be set to the display same with the display of above-mentioned display 19a.

(7) form showing emotion display on the display 19a of interface portion 19 is that form as shown in Figure 9 also can.In the left side of emotion display 303,304,305,306, show respectively " driver's seat ", " co-pilot seat ", " the right seat of rear side " that represent and show corresponding seating position with emotion, the left seat of rear side etc.).Namely, emotion display 303,304,305,306 to show with this emotion corresponding to seating position (achieving the seating position of the passenger of the emotional information on the basis shown as this emotion) show with being associated.

In addition, on the right side of emotion display 303,304,305,306, the title showing corresponding passenger with this emotion is shown respectively.Namely, in described S3, if in the identifier being stored in storage part 9, there is or akin identifier identical with the identifier of the passenger occupying seating position, then the title of the passenger corresponding with this identifier is shown in the right side that emotion shows, if in the identifier being stored in storage part 9, there is not identical with the identifier of the passenger occupying seating position or akin identifier, then carry out the display of " guest 1 ", " guest 2 " etc.

(8), in " emotion display making " row in the map, all categories of emotions are "○" and also can.Now, emotion display can be made based on all emotional informations.

(9) form of emotion display can set arbitrarily.Such as, even if categories of emotions is identical, the various forms (such as, the color of emotion display, the light levels of emotion display, the size of emotion display, the action etc. of emotion display) of emotion display also can be changed according to the intensity of emotion.In addition, if the intensity of emotion does not reach defined threshold, then also can not make emotion display.In addition, the intensity of emotion does not reach display during defined threshold, can be set to the display as the circle 225,227 in Fig. 7.

(10) be mapped in storage part 17 to store with being associated with seating position and also can.Now, in described S22, read and establish with seating position α the mapping associated, use this mapping, make emotion display and the notification voice of seating position α.Therefore, the corresponding relation of the form shown for intensity and the emotion of each seating position setting benchmark relevant with the kind of emotional information or emotion.

The application describes in accordance with embodiment, but is construed as the application and is not limited to this embodiment or structure.The application also comprises the distortion in various variation or equivalency range.In addition, various combination and mode and then to comprise among these category and thought range that only key element, its above or its other following combinations and mode are also contained in the application.

Claims

1. an emotion surveillance (1), is characterized in that possessing:

Face-image obtaining section (3), for the plural seating position of vehicle, can obtain the face-image of the passenger (101) occupying this seating position;

Emotional information obtaining section (13), based on the face-image obtained by described face-image obtaining section, obtains the emotional information of the emotion representing described passenger; And

Emotion display preparing department (15), makes the emotion corresponding with the described emotional information obtained by described emotional information obtaining section and shows.

2. emotion surveillance as claimed in claim 1, is characterized in that possessing:

Display part (19a), the described seating position of the described passenger corresponding to described emotion display being shown with this emotion shows with being associated.

3. emotion surveillance as claimed in claim 1 or 2, is characterized in that,

This emotion surveillance possesses the 1st storage part (17), and the 1st storage part (17) stores the benchmark relevant with the kind of described emotional information,

Described emotion display preparing department makes described emotion for the described emotional information meeting the benchmark relevant with the kind of described emotional information and shows.

4. emotion surveillance as claimed in claim 3, is characterized in that,

This emotion surveillance possesses passenger identification part (7), and this passenger identification part (7) identifies described passenger,

Described 1st storage part stores the benchmark relevant with the kind of described emotional information by each described passenger,

Described emotion display preparing department, based on the recognition result of described passenger identification part, uses with the kind of the described emotional information relevant benchmark corresponding with this passenger by each described passenger.

5. the emotion surveillance as described in any one of Claims 1-4, is characterized in that,

Described emotional information comprises the intensity of described emotion,

This emotion surveillance possesses the 2nd storage part (17), and the 2nd storage part (17) stores the corresponding relation of the form that the intensity of described emotion and described emotion show,

The form of the described emotion display made by described emotion display preparing department determines based on described corresponding relation.

6. emotion surveillance as claimed in claim 5, is characterized in that,

Described 2nd storage part stores described corresponding relation by each described passenger,

Described emotion display preparing department, based on the recognition result of described passenger identification part, uses the described corresponding relation corresponding with this passenger by each described passenger.