JP2008017227A - Face recognizing device, and automobile user treating system using the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face recognizing device which turns on an illumination light source only when a sharp photographic image can not be obtained because of diffused light, underexposure, etc., and to provide an automobile user treating system using the same. <P>SOLUTION: The face recognizing device has a photographing device 15 which is fitted to an automobile and photographs the face of a person present in a visual field of photography and a face registration part 17 which stores face feature information on a person registered as a user while making it correspond to user specification information, and performs user recognition processing with face feature information of a photographed face image and the face feature information registered in the face registration part 17 to outputs a recognition result. If the user recognition ends in failure, a lighting device 16 which lights up the face of the person in the visual field of photography is turned on to photograph a face image again, and authentication processing is performed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a face recognition device for recognizing a user who uses a car, and at least one of approaching the user, getting on, driving, getting off, and leaving the car. The present invention relates to an automobile user hospitality system for assisting a user in using a car or for entertaining (or serving) a user.

JP 2004-144512 A

  In Patent Document 1, as a face recognition device for recognizing a user who uses an automobile, infrared light is used as auxiliary illumination light in order to prevent a user's face photographed image from being blurred due to visible disturbance light, In addition, an apparatus has been proposed in which a filter that selectively passes infrared rays is disposed in the incident optical system to capture the face image, thereby eliminating the influence of visible disturbance light and making it easy to obtain a clear captured image.

  However, the above-described apparatus is uneconomical because the infrared illumination light source must be constantly turned on during photographing even during daytime when a sufficient amount of photographing light can be secured, and there is a problem that the life of the vehicle battery is likely to be affected.

  An object of the present invention is to provide a face recognition device in which an illumination light source is turned on only when a clear photographed image cannot be obtained due to the presence of disturbance light or insufficient exposure, and an automobile user hospitality system using the face recognition device. Is to provide.

Means and actions / effects for solving the problems

In order to solve the above problems, the face recognition apparatus of the present invention
An imaging device that is attached to the inside or outside of an automobile and that captures the face of a person existing within a predetermined field of view;
A face registration unit that stores face feature information of a person registered as a car user in association with user-specific information;
A lighting device that illuminates the face of a person present in the field of view of the imaging device;
User recognition means for acquiring face feature information from a face image photographed by a photographing device and performing user recognition processing by collating the acquired face feature information with face feature information registered in a face registration unit;
User recognition result output means for outputting the user recognition result;
And illumination control means for turning on the illumination device when the user recognition by the image taken in the non-illuminated state fails.

  According to the face recognition device of the present invention, the lighting device is turned on when the user recognition unit fails in user recognition based on an image captured when the lighting device is not lit. At the time of shooting, it is possible to effectively solve the problem that the lighting device is always lit even during the daytime.

The automobile user hospitality system of the present invention is
The face recognition device of the present invention;
A hospitality action to assist the user in using the car or to entertain the user at least one of approaching the user, riding, driving or staying in the car, getting off and leaving the car. A hospitality action part,
A hospitality operation information storage unit that stores the registered user and the hospitality operation information that defines the operation content of the hospitality operation unit specific to each registered user in association with each other;
A hospitality operation control unit that reads out the hospitality operation information corresponding to the registered user recognized by the face recognition device from the hospitality operation information storage unit and controls the operation of the hospitality operation unit according to the hospitality operation information. And

  According to the above-mentioned hospitality system for automobiles of the present invention, even if a user cannot recognize a clear photographed image due to insufficient exposure or the like, the user will be recognized next time by lighting the illumination light in response to the result. Therefore, it is possible to reliably obtain a face image having a favorable sharpness. As a result, it is possible to reliably identify the user who is the subject of hospitality, and as a result, it is possible to appropriately perform the hospitality process that suits the user, thereby enhancing the satisfaction of the user who uses the car. it can.

  Next, the imaging device can shoot a face with both visual light and infrared light by using an image sensor that has the sensitivity necessary for face photography in both the visible light region and the infrared region. Can be used. Thereby, in the nighttime when the amount of photographing light is insufficient, since infrared light which is invisible light is used as an illumination light source, there is an advantage that the user is not easily dazzled by illumination. On the other hand, in the daytime or the like, it is possible to shoot without problems using the visible light sensitivity characteristics of the image sensor. The use of a near-infrared light source as the infrared light source makes it possible to construct the illumination light source at a low cost and also facilitates obtaining illumination intensity. Furthermore, if the near-infrared region is adjacent to the low wavelength side of the visible light sensitivity region, there is an advantage that it is relatively easy to ensure the sensitivity of the imaging device.

  As the illumination light source, a visible light source or an ultraviolet light source can be used instead of the infrared light source.

  Next, it is possible to provide a plurality of illumination devices that illuminate faces existing in the field of view of the photographing device from different directions. As a result, the face of the person who is the subject can be uniformly illuminated by the plurality of illumination devices, and the recognition accuracy of the user recognition process performed after the illumination is turned on can be improved. From the viewpoint of illuminating the face with uniform illuminance, it can be said that it is desirable to turn on a plurality of lighting devices in a lump (by the lighting control means). However, in the case where the face image is crushed by strong disturbance light that is incident obliquely with respect to the shooting optical axis, the user recognition failure may be caused by disturbance rather than applying illumination light uniformly to the face. In some cases, it is more effective to correct the bias of the illuminance distribution on the subject due to light.

  Further, when the face presence determination unit determines whether or not a person's face is present in the field of view based on the captured image of the shooting device, and the face presence determination unit determines that the face is present, the shooting device sets the illumination device. The primary face is photographed in a non-lighting state, and the user recognition means performs a primary user recognition process based on the face image in the non-lighting state, and if the user identification by the primary user recognition process fails, the lighting control means The lighting device is turned on, and in this state, the photographing device performs secondary face photographing with the lighting device turned on, and the user recognition means can perform secondary user recognition processing based on the face image in the lighting state. . According to this configuration, since the user recognition process is executed only when it is determined that a face is present in the photographed image, it is not necessary to perform a useless user recognition process. Even if the user identification fails in the primary user recognition process, the secondary user recognition process is performed after the lighting device is turned on, so that the user can be identified reliably.

  Further, in this case, the lighting device is provided with a plurality of lighting devices that illuminate the face existing in the field of view of the photographing device from different directions, secondary face photographing by the photographing device, and secondary user recognition processing by the user recognition means Can be repeatedly executed while sequentially changing the types of lighting devices to be lit so that the illumination directions are different from each other. According to this configuration, since the direction of illumination sequentially changes, a secondary face image can be taken in an appropriate illumination state, and face detection can be reliably performed.

  Next, feature information synthesizing means for synthesizing feature information from each face image obtained by multiple times of face photographing is provided, and the user recognition means can perform user recognition based on the synthesized feature information. . By using the synthesized feature information, it is possible to efficiently perform user recognition using a face image.

  Further, the user recognition processing repetitive control means for periodically performing the face photographing by the photographing device and the user recognition processing by the user recognizing means based on the photographed face image, and the user recognizing means once succeeded in user recognition. Thereafter, even if the user recognition fails in the subsequent iterative process of the user recognition process, the user recognition result output unit does not output the failure recognition result until the number of times of failure recognition reaches a predetermined number. And a user recognition result output control means. Since the lighting state changes from time to time, it is not appropriate to make a recognition NG determination due to it. In particular, in the case of user recognition in the passenger compartment, it is unlikely that the user frequently changes the seating position. According to this configuration, if the user recognition is successful even once, the recognition NG determination is not made due to the failure recognition within a predetermined number of times or within a predetermined period. Various applications of the vehicle to be executed can be appropriately operated.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a conceptual block diagram of an automobile control system in which the face recognition device of the present invention is incorporated. The vehicle control system 1 in FIG. 1 includes a face recognition device 10 according to the present invention that performs user recognition based on a photographed human face image, and an occupant adaptation system master device that executes various controls corresponding to the recognized user. (Referred to as a master controller).

First, the face recognition apparatus 10 in this embodiment will be described.
A face recognition device 10 shown in FIG. 1 and FIG. 2 is attached inside or outside a car, and is a photographing device 15 for photographing a face of a person existing in a predetermined field of view, and a person existing in the field of view of the photographing device. The illuminating device 16 that illuminates the face of the camera, the photographic object detection unit 31 that detects the photographic object (person) in the field of view of the photographic device 15, and the control unit (image processing device) 11 to which these are connected.

  The control unit 11 includes a face registration unit 17 that stores face feature information of a person registered as a car user in association with user-specific information, and acquires face feature information from a face image photographed by the photographing device 15. The user recognition processing is performed by comparing the acquired face feature information with the face feature information registered in the face registration unit 17, and the user recognition result is output to the master control device 20. Specifically, as shown in FIG. 2, the control unit 11 includes a CPU 12, a RAM 13 including a work memory, and a ROM 14 that stores various programs. The ROM 14 is a flash ROM that is a nonvolatile memory. There is a partial storage area as the face registration unit 17. The control unit 11 outputs a control signal to the photographing device 15 and the illumination device 16. The control unit 11 functions as user recognition means, user recognition result output means, illumination control means, face presence determination means, feature information synthesis means, and user recognition result output control means by executing a program stored in the ROM 14. To do.

  The photographing target detection unit 31 is a proximity sensor that is provided corresponding to each photographing device 15 and detects a photographing target that exists in the photographing visual field of the photographing device 15 or a photographing target that approaches the photographing visual field. In the present embodiment, a well-known device that outputs infrared rays, high-frequency waves, ultrasonic waves, or the like and senses entry into the measurement area can be adopted as the imaging target detection unit 31 corresponding to the outside-vehicle imaging device 15. Further, the imaging target detection unit 31 corresponding to the in-vehicle imaging device 15 can adopt the same proximity sensor as described above, and an operation related to a seating sensor provided on the seat or a door opening / closing operation (for example, gripping of a door knob). May be detected.

  As shown in FIG. 6, the imaging device 15 uses an imaging device having sensitivity necessary for face photography in both the visible light region and the infrared region, and captures a face by both visible light and infrared light. Is possible. As shown in FIGS. 3A and 3B, the photographing device 15 includes in-vehicle photographing devices 15a, 15b, and 15c for photographing a user's face seated on a car seat, and a user's face outside the vehicle boarding from the door. And vehicle exterior photographing devices 15d, 15e, and 15f for photographing.

  The in-vehicle photographing device 15 is provided corresponding to a seat in the vehicle, and each photographing field of view is determined so that the face of the user seated in the corresponding seat can be photographed. Here, as shown in FIG. 3A, the in-vehicle image capturing device 15a is provided in the meter cluster or above the meter cluster to image the face of the driver (driver) in the driver seat, and the in-vehicle image capturing device 15b is provided in the passenger seat. An in-vehicle image capturing device 15c is provided on the upper part of the front seat or on the headrest thereof in order to take an image of the passenger's face in the rear seat. Note that the rear-seat imaging device 15c has a wide angle of view so that photographing can be performed even if the seat moves.

  On the other hand, the outside photographing device 15 is provided corresponding to each door of the vehicle. Here, a plurality of vehicle exterior photographing devices 15 correspond to each door. As shown in FIG. 3B, the vehicle exterior photographing device 15d is the front door window frame lower end, the vehicle exterior photographing device 15f is the rear door window frame lower end, and the vehicle exterior photographing device 15e. Is provided in a central pillar between them, and is configured to photograph a passenger approaching the door by a plurality of photographing devices. 3B corresponds to the passenger side doors (passenger door 1101 and slide door 1102), but is also provided in the driver's seat in the same manner. . Moreover, in order to image a passenger approaching the back door, an outside-vehicle imaging device 15g is provided at the lower end of the back door window frame. In addition, when there are other doors, it is possible to provide an outside-vehicle photographing device in the same manner.

  The illuminating device 16 uses an infrared light source as an illuminating light source, specifically, an LED having a near-infrared light source as shown in FIG. 5, and is provided corresponding to the photographing field of view of each photographing device 15. For example, with respect to the field of view of the in-vehicle imaging device 15a corresponding to the driver's seat, the in-vehicle imaging device 16a is at the lower end of the door window frame, the in-vehicle imaging device 16b is at the front pillar, the in-vehicle imaging device 16c is at the meter cowl, and the in-vehicle imaging device. Reference numeral 16d is provided in the upper part of the center console, in-car photographing device 16e is provided in the lower part of the center console, and in-car photographing device 16f is provided in the map lamp part. As described above, in the present invention, a plurality of illumination devices 16 corresponding to the in-vehicle photographing device 15 can be provided so that faces existing in the field of view of the photographing device 15 can be illuminated from different directions. Similarly, a plurality of passenger seats and rear seats can be provided. Also, one or a plurality of illumination devices (not shown) are provided for the outside-vehicle imaging devices 15d, 15e, and 15f. If a common space is included in the field of view of the photographing device 15, a lighting device that is shared with the photographing device 15 may be provided.

  In addition, the control unit 11 acquires face feature information from the face image captured by the image capturing device 15 and recognizes the user by comparing the acquired face feature information with the face feature information registered in the face registration unit 17. A user recognition program is stored in the ROM 14. Hereinafter, the flow of the user recognition program will be described with reference to the flowchart shown in FIG.

  First, in S <b> 101, it is determined whether or not the shooting target detection unit 31 has detected a shooting target (person) within the shooting field of view of the shooting device 15. If not detected, this program is terminated. If it has been detected, the process proceeds to S102, where the imaging device 15 corresponding to the detected imaging object detection unit 31 sequentially performs with the arrival of a predetermined imaging timing. The photographing at this time is primary face photographing performed with the illumination device 16 in a non-lighting state. The captured images are sequentially input to the control unit 11.

  Note that shooting by the shooting device 15 is repeatedly performed at a predetermined shooting cycle while the corresponding shooting target detection unit 31 continues to detect the shooting target. For example, as shown in FIG. 7A, there is a case where one shooting timing arrives for one imaging cycle, or a plurality of shooting timings arrives for one imaging cycle as shown in FIG. 7B. In the present embodiment, it is assumed that the case of FIG. 7A is adopted.

  In S103 following S102, the control unit 11 determines whether or not a person's face is present in the field of view based on the input photographed image, and this is repeated until a face is found. The presence of a face is assumed to be present if a portion recognized as a face is recognized even in a known program for detecting face feature information. If it is determined that a face exists, the process proceeds to S104, and user recognition processing is performed based on the photographed face image. Initially, since a picture is taken in a non-lighted state, a primary user recognition process is performed. In S104, feature information from each face image obtained by multiple times of face photographing is synthesized, and user recognition is performed based on the synthesized feature information. Specifically, as shown in FIG. 9, first, in S10, a predetermined number of recently captured images are acquired, and in S20, these captured images are combined to be suitable for face recognition. Create an image. Then, feature points (organs such as eyes and mouth) are extracted from the synthesized face image in S30, and the feature points and face registration unit are normalized with respect to the feature points extracted in S40. Face recognition is performed by matching with the face feature information stored in 17. This face recognition process ends within a predetermined time (for example, 1 sec). Note that a captured image whose face could not be recognized by this processing may be deleted so that it is not used in the subsequent processing.

  Returning to FIG. In S105, it is determined whether or not the photographed face image matches the registered user's face as a result of the face recognition in S104. If they match, the process proceeds to S106, where the user specifying information relating to the specified user is transmitted to the master control device 20, and this program is terminated. If not, the process proceeds to S107.

  In S107, it is determined whether or not a sufficient time (for example, 2 seconds) for completing the face recognition process in S104 has elapsed. If this predetermined time has not elapsed, the process returns to S104. If it has elapsed, the process proceeds to S108, and the lighting device 16 is turned on. In the present embodiment, a plurality of illumination devices 16 illuminate faces existing in the field of view of the corresponding photographing device 15 from different directions, and the plurality of illumination devices 16 are as shown in FIG. 7A. Lights all at once. As a result, the face of the person who is the subject can be uniformly illuminated by the plurality of illumination devices, and the recognition accuracy of the user recognition process performed after the illumination is turned on can be improved. In addition, in a state where a plurality of lighting devices 16 are provided as described above, as shown in FIG. 7B, the lighting devices 16 may be turned on / off while sequentially changing the lighting directions to be different from each other. In this case, the secondary face photographing is performed in accordance with the lighting state of each lighting device 16. This produces an effect of reducing the burden on the eyes of the person being photographed.

  When the lighting device 16 is turned on, secondary photographing is performed periodically by the photographing device 15 according to the lighting state, and the photographed image is input to the control unit 11 to perform face recognition processing again. The face recognition process performed by the face image obtained by the secondary face photographing is a secondary user recognition process. The primary user recognition process and the secondary user recognition process described above are performed by the same process. However, since the latter uses a photographed image taken with the lighting device 16 turned on, it is easier to recognize.

  When the lighting device 16 is turned on in S108, the lighting device 16 continues to be turned on until a predetermined time (for example, 5 seconds) elapses, and the face recognition is performed in S104 after the predetermined time has passed. If the user identification by the process is not completed, the process proceeds to S110, where the fact that face recognition could not be performed is transmitted to the master control apparatus 20, and the lighting apparatus 16 is turned off in S111, and then this program is terminated. This program is repeatedly executed periodically.

  According to this program, even if the user recognition fails due to insufficient exposure, etc., even if the user recognition fails, the illumination light is turned on in response to the result, and the next time the visibility is convenient for user recognition. It is possible to reliably obtain a face image having As a result, the user can be reliably identified. Further, since the illumination light is turned on only when it fails, there is an advantage that useless lighting control and lighting driving are not performed.

  Note that the user recognition program may have the flow shown in FIG. In this program, the face photographing by the photographing device 15 and the user recognition process based on the photographed face image are periodically executed repeatedly, and after the user recognition is successful once, the user recognition process is performed in the subsequent iteration process. Even when recognition fails, the failure recognition result is not output until the number of failure recognition reaches a predetermined number. Hereinafter, the flow of the user recognition program shown in FIG. 10 will be described.

  In S201 to S205, the same processing as (S101 to S105) of the program of FIG. 8 is performed.

  In S205, it is determined whether the captured face image matches the registered user's face as a result of the face recognition in S204, and if they match, the process proceeds to S206. In S206, the failure recognition count N stored in a predetermined storage area of the predetermined control unit 11 is reset to N = 0. In S207, the face recognition flag T, which is set to “1” when face recognition has been performed in the past, is set to T = 1, and the specific user information of the user specified in S208 is set as the master control device. 20 to finish this program.

  If it is determined in S205 that the photographed face image does not match the registered user's face, the process proceeds to S209. In S209, it is determined whether or not the face recognized flag T is “1”. If it is not “1”, that is, if it is “0”, the process proceeds to S213, and it is mastered that face recognition could not be performed. It transmits to the apparatus 20 and complete | finishes this program.

  If the face recognition flag T is “1” in S209, the process proceeds to S210, and the failure recognition count N is set to N = N + 1. In S211, it is determined whether or not N = 3 as a result of S210. If N = 3, the face recognized flag T is set to T = 0, the process proceeds to S213, the fact that face recognition could not be performed is transmitted to the master controller 20, and this program is terminated. If N = 3, the program is terminated as it is. This program is repeatedly executed periodically.

  According to this program, once user recognition is made, even if failure recognition within a predetermined number of times is met, a recognition NG determination is immediately made by this, and this is not output. Since the lighting state of the object to be photographed changes depending on the target posture / motion and the state of the ambient light, when the recognition NG and the recognition OK are repeatedly output due to this, the control side based on that is performed. In the present embodiment, the control load on the master control device 20 side becomes large. However, according to the above program, the recognition NG determination is not output to the outside in the case of a predetermined number of failure recognitions, so the master control device 20 side The control load is reduced.

  The user recognition program executes a primary user recognition process and a secondary user recognition process as in the program of FIG. 8, and the number of times of failure recognition is predetermined as in the program of FIG. The program may be configured not to output the failure recognition result until the number of times is reached. As a result, the effects of both of the above programs can be realized.

Next, the vehicle control system 1 using the face recognition device 10 will be described.
As shown in FIG. 1, the operation of the devices 513, 541, 511, and 515 to be controlled is controlled by executing an operation control application stored in the ROM of the master controller 20 itself. The master control device 20 changes the control content according to the user recognition result (user identification information) input from the face recognition device 10. That is, the master control device 20 executes control according to the recognized user.

  For example, the master control device 20 can control the door lock device 513 according to the recognized user. Specifically, when the face recognition device 10 recognizes that the user is a pre-registered user from the face image photographed by the outside photographing device 15, it is possible to perform control for releasing the door lock.

  Further, the master control device 20 can control the sliding door device 541 according to the recognized user. Specifically, when the face recognition device 10 recognizes that the user is a pre-registered user based on the face image captured by the outside-vehicle image capturing device 15 and the recognized user is a child or an elderly person. The sliding door can be automatically opened.

  Further, the master control device 20 can control the indoor illumination device 511 according to the recognized user. Specifically, in accordance with the user recognized by the face image captured by the image capturing device (particularly the in-vehicle image capturing device) 15, it is possible to perform control for switching to the indoor illumination setting registered in advance in association with the user. it can.

  Further, the master control device 20 can control the music playback device 515 in accordance with the recognized user. Specifically, according to the user recognized by the face image photographed by the photographing device (particularly the in-vehicle photographing device) 15, the reproduction control of the music reproducing device is performed according to the music list registered in advance in association with the user. Can be done.

Next, an example of a specific embodiment of the vehicle control system 1 will be described.
FIG. 11 shows a concept of an automobile user hospitality system (hereinafter also simply referred to as “hospitality system”) 1 incorporating functions of an automobile hospitality information providing system, which is an embodiment of the automobile control system 1 of the present invention. It is a block diagram. The hospitality system 1 includes a hospitality execution control unit 3 including a first computer to which various hospitality operation units 502 to 517, 534, 541, 548, 549, 550, 551, 552, and 1001B are connected, and various sensors. The vehicle-side mounting unit 100 </ b> A including the hospitality decision-making unit 2 including the second computer to which the camera groups 518 to 528 are connected is configured as the main part. Each of the first computer and the second computer includes a CPU, a ROM, and a RAM, and various functions described later are realized by executing control software stored in the ROM using the RAM as a work memory.

In addition, the face recognition apparatus 10 in the above embodiment shown in FIG. 1 is incorporated as a face recognition unit 10 in a part of the hospitality decision making unit 2, and the master control unit 20 in the above embodiment also has the face recognition. This corresponds to the hospitality decision making unit 2 and the hospitality execution control unit 3 excluding the unit 10. The hospitality determination unit 2 including the face recognition unit 10 is connected to the following sensor / camera group for user recognition.
Vehicle exterior camera 518 (corresponding to the vehicle exterior photographing device 15 in the above embodiment): Photographs a user approaching the vehicle. Acquire user gestures and facial expressions as still images or videos. In order to photograph a long-distance user or to magnify and photograph only the user's face part, an optical zoom method using a telephoto lens and a digital zoom method for digitally enlarging a photographed image can be used in combination.
Face camera 521 (corresponding to the in-car image taking device 15 in the above embodiment): Photographs the facial expression of the seated user. In this embodiment, it is attached to a rearview mirror or the like, and the upper body including the face of the user (driver) who is seated on the seat is photographed from obliquely above from the front glass side. The passenger seat side is similarly attached to the rearview mirror or the like. The rear seat side is attached in the same manner as in the above embodiment. From the attached face camera, the upper body of the passenger in the corresponding seat can be taken. A face image is cut out from the captured image, and various facial expressions of the user are captured and compared with a master image prepared in advance, so that not only the user's face authentication but also a load source other than a person is placed. The case where the person is seated and the case where the person is seated can be distinguished from each other, and various facial expressions of the seated user can be specified.

In addition, the hospitality determination unit 2 including the face recognition unit 10 includes the following sensor / camera group for detecting a shooting target existing in the shooting region of the shooting device 15 or a shooting target approaching the shooting region. Is connected. The detection of the imaging target of the sensor / camera group triggers the execution of the primary face imaging process.
Seating sensor 520 (corresponding to the photographing target detection unit 31 in the above embodiment): detects whether the user is seated on the seat. It can be composed of a proximity switch or the like embedded in the automobile seat. A pressure sensitive sensor can also be used. In this case, the user's operation on the seat can be detected by attaching the pressure sensor to the seat.
Door courtesy switch 537 (corresponding to the photographing target detection unit 31 in the above embodiment): detects opening and closing of the door. It is used as a scene estimation information acquisition means for detecting the transition to the boarding scene and the getting-off scene.

  Note that a well-known GPS for obtaining position information is provided in the user-side terminal device (mobile phone) 100, and this also functions as a photographing target detection unit together with the GPS 533 on the vehicle side.

  In addition, sensors / cameras 519 and 522 to 528 are provided, and these function as a scene estimation information acquisition unit and a biological state detection unit.

  The hospitality decision making unit 2 also receives a branch input of an output of an ignition switch 538 for detecting engine start. An illuminance sensor 539 for detecting the brightness level in the vehicle and a sound pressure sensor 540 for measuring the sound level in the vehicle are also connected to the hospitality decision making unit 2 in the same manner.

  Further, the hospitality decision making unit 2 may be a touch panel (a touch panel superimposed on the monitor of the car navigation device 534 shown in FIG. 20): In this case, the input information is received from the hospitality execution control unit 3 Are connected to a storage device 535 including a hard disk drive that functions as a hospitality operation information storage unit.

  The method of user recognition processing is the same as that in the above embodiment shown in FIG. In a state where a sensor corresponding to the photographing target detection unit 31 detects a photographing target (person), primary face photographing or secondary face photographing is performed, and facial feature information obtained from the photographed image and a user registration unit (face The user is identified by comparing the facial feature information (master information) stored in the registration unit) 17 (partial storage area of the ROM of the hospitality determination unit 2; see FIG. 11).

  Here, user registration in the hospitality information providing system 1 for automobiles will be described. The user of the car has a user registration unit 17 (for example, a ROM of the hospitality decision-making unit 2 (preferably configured with a flash ROM so that rewriting is possible): the face registration unit in the above-described embodiment. ) Or the storage device 535. As shown in FIG. 12, each user name (or user ID (and password) and its face image (master face image for user recognition) are associated with each other in the user registration unit 17. It is registered with.

  Specifically, on the monitor 536 in FIG. 11 (the monitor of the car navigation device 534 may be substituted), the precautions regarding the shooting angle are displayed so that a front image capable of specifying a facial expression is obtained, and then displayed on the seat. The seated user is photographed by the photographing device 15 (here, the in-vehicle photographing device 15 is used). The captured master face image for user recognition is stored in the user registration unit 17 in association with the user name (user specifying information). It should be noted that a step for confirming whether the face recognition process can be correctly performed with the photographed face image may be provided, and if it is not recognized correctly, the image may be photographed again.

  Further, in the present embodiment, as shown in FIG. 12, the age, sex, and personality type (see FIG. 16) of each user are registered in association with each user name. As described above, the personality type is acquired and accumulated as operation history information in a specific operation unit while the user continues to use the vehicle, and is estimated based on the accumulated operation history information. However, if the operation history information is insufficiently accumulated, such as immediately after the start of use of the car, or if it is desired to estimate the personality type without collecting the operational history information, the personality type information or the personality Information necessary for specifying the type information may be input by the user himself and the personality type may be determined based on the input result.

  First, as shown in FIG. 22, the personality type is displayed on the monitor 536 of FIG. 11 (the monitor of the car navigation device 534 may be substituted), and the user selects the personality type that suits the user, and the input unit 529 There is a way to input this. Here, the input unit is a touch panel overlaid on the monitor 536, and a selection input is made by touching the selection button 529B formed on the display. On the other hand, as shown in FIG. 23, there is a method of inputting a questionnaire for determining the personality type instead of directly inputting the personality type. In this case, the questionnaire questions are displayed on the monitor 536, and the user answers by selecting an answer from the answer options (in this case, the option is configured with a selection button on the screen, and the corresponding position of the touch panel superimposed on this is selected. Touch to make selection input). By answering all the questions, one is uniquely determined from a personality type group determined in advance according to the combination of the answers.

  The user registration input including the user name (user identification information) is made from the input unit 529 and stored in the user registration unit 17 together with the photographed face image and the determined personality type. These series of inputs can also be performed from the mobile phone 1, and in this case, the input information is transferred to the automobile side by radio. In addition, when the user purchases a car, there is a method in which the dealer side uses the input unit 529 or a dedicated input tool to perform user registration input in advance. In this case, the user name, age, and gender are also input and stored in association with each other as user registration information. As described above, when the personality type of the user can be specified, the personality code is also registered. As a result, authentication of registered users can be performed by taking face images (primary face photography and secondary face photography), and a hospitality operation corresponding to the registered user can be executed. It is also possible to reflect the personality type of the user.

  In the hospitality system 1 described above, a series of operations related to the use of the user's car until the user approaches the car, gets into the car, drives the car or stays in the car, and then gets off the car. Are divided into a plurality of predetermined scenes. Then, for each of a plurality of divided scenes, the hospitality operation units 502 to 517, 534, 541, 548, 549, 550, 551, 552, and 1001B assist the user in using the car or enjoy the user. The hospitality operation for the purpose is performed. In the present embodiment, a horn 502 and a buzzer 503 are connected as a sound wave generator to the outside of the vehicle. Further, as a lighting device (lamps), a head lamp 504 (beam can be switched between high and low), a fog lamp 505, a hazard lamp 506, a tail lamp 507, a cornering lamp 508, a backup lamp 509, a stop lamp 510, indoor lighting 511 and an underfloor lamp 512 are connected. In addition, as other hospitality operation units, an air conditioner 514, a car audio system (car stereo) 515, an electric seat 516, an angle adjusting drive unit 517 such as a side mirror and a rearview mirror, a car navigation device 534, an assist for opening and closing the door. Mechanism (hereinafter referred to as a door assist mechanism) 541, an aroma generating unit 548 that releases a fragrance into the vehicle, an ammonia generating unit 549 for awareness and awakening for severe physical condition (including a state of severe sleepiness) (for driving) Is attached to the center of the steering wheel so as to eject ammonia in the form of aiming near the driver's face), and a seat vibrator 550 (the seat bottom or backrest) for alerting the driver and awakening from drowsiness Embedded in the handle vibrator 551 (attached to the handle shaft) Are being), noise canceller 1001B for reducing cabin noise is connected. The hospitality execution control unit 3 is also connected with a GPS 533 (also used in the car navigation device 534), a brake sensor 530, a vehicle speed sensor 531 and an acceleration sensor 532 for acquiring vehicle position information.

  In this embodiment, in this embodiment, the approach scene SCN1, the boarding scene SCN2, the preparation scene SCN3, the driving / staying scene SCN4, the getting-off scene SCN5, and the leaving scene SCN6 are set in this order in time series.

  As will be described later, the approach scene SCN1 is specified by specifying the relative distance between the vehicle and the user located outside the vehicle and its change by the GPS of the user terminal device 100 and the GPS 533 of the vehicle. This is done by detecting that the vehicle has been approached within a predetermined distance. The boarding scene SCN2 and the getting-off scene SCN5 are specified based on the door opening detection output of the door courtesy switch 537. However, since it is not possible to specify whether it is a boarding scene or a getting-off scene simply by door opening information, a scene flag is provided as current scene specifying information storage means for storing and holding current scene specifying information in the RAM of the hospitality decision making unit 2 I try to cope with it. Each scene flag has an individual scene flag corresponding to each scene, and each time a scene whose arrival order is determined in time series arrives, the flag corresponding to that scene is set to “arrival (flag value 1)”. Go. By specifying the latest flag of which the value is “1” (the last one in the “1” flag string) in the scene flag, it is possible to specify which scene is currently advanced.

  The preparation scene SCN3 and the driving / staying scene SCN4 are both identified by whether or not the above-described seating sensor detects the user, but until the ignition switch 538 is turned on after entering the automobile, or Until the ignition switch 538 is not turned on and the continuation of sitting beyond a certain level is confirmed, it is recognized as a preparation scene. The transition to the leaving scene SCN6 is identified by the door courtesy switch 537 detecting the door closing after the getting-off scene SCN5.

  The hospitality operation in each scene is controlled by the operation control application of the corresponding hospitality operation unit. The theme determined on the hospitality decision making unit 2 side is notified to the hospitality execution control unit 3, and the operation control application of the corresponding scene is read out and executed there. When executing a motion control application, the priority of adoption is determined in advance for each of the hospitality operations managed under each hospitality theme according to the degree to which the user is oriented to the hospitality. The plurality of prepared hospitality operations are selected from those having a higher priority.

  In the hospitality system having the above-described configuration, the face detection unit 10 of the hospitality determination unit 2 outputs user identification information, and determines what hospitality operation is to be performed by which hospitality operation unit according to the contents thereof, This is commanded to the hospitality execution control unit 3. In response to this, the hospitality execution control unit 3 causes the corresponding hospitality operation units 502 to 517, 534, 541, 548, 549, 550, 551, 552, and 1001B to execute the hospitality operation. That is, the hospitality decision making unit 2 and the hospitality execution control unit 3 cooperate with each other, and the hospitality operation units 502 to 517, 534, 541, 548, 549, 550, The function of changing the operation contents of 551, 552, and 1001B is realized.

  Hereinafter, among the hospitality operation units 502 to 517, 534, 541, 548, 549, 550, 551, 552, and 1001B controlled by the hospitality execution control unit 3, user-specific information output from the face recognition unit 10 What is controlled based on this will be described.

  First, indoor illumination as a hospitality operation will be described. In indoor illumination, lighting conditions (lighting color, lighting pattern, etc.) for indoor illumination are set for each registered user registered on the vehicle side, and are determined in advance when the user is recognized by the face recognition unit 10. As the scene arrives, the room lighting 511 is turned on under lighting conditions corresponding to the recognized registered user.

  FIG. 13 shows a configuration example of the indoor lighting 511, and a plurality of lighting sections each having a unique lighting color (in this embodiment, a red lighting 511r, an amber lighting 511u, a yellow lighting 511y, a white lighting) 511w and blue illumination 511b). These lights are selected by receiving a control command signal input from the hospitality decision making unit 2 through the hospitality execution control unit 3, and specified lighting is selected, and lighting control is performed in various lighting patterns according to the control command signal. . The lighting color and lighting pattern are set for each registered user. FIG. 14 shows a configuration example of lighting setting conditions (lighting control data) set for each registered user, and is stored in the ROM of the hospitality decision making unit 2. For example, for user 1 (see FIG. 12), the red illumination 511r is selected and flashed (only the first, and then continuously illuminated), and for user 2, the amber illumination 511u is selected and faded. This is just an example. When the user is recognized by the face recognition unit 10, the lighting of the room light 511 is output from the face recognition unit 10, and user identification information is output from the face recognition unit 10. When the scene to be received arrives, the hospitality decision-making unit 2 reads the lighting setting condition corresponding to the input user-specific information from the ROM, outputs a lighting command signal and a lighting condition signal to the hospitality execution control unit 3, and performs hospitality execution control The unit 3 outputs a control command signal to the corresponding illumination unit to cause the illumination unit to emit light. The lighting condition setting stored in the ROM can be customized by an input operation from the input unit 529, and the user can arbitrarily set the lighting color / lighting pattern.

  In addition to the incandescent light bulb and the fluorescent lamp, the lighting device using a light emitting diode can be adopted as the lighting device. In particular, various illumination lights can be easily obtained by combining light emitting diodes of the three primary colors of red (R), green (G), and blue (B). The input waveform of the command signal to each color light emitting diode can be changed independently, and an illumination color of an arbitrary color tone can be obtained according to the mixing ratio of the three emission colors, and the color tone and illumination intensity pattern can be changed. It is also possible to change over time according to the input waveform of the command signal. In addition to the PWM control method, the light emission intensity of each color light emitting diode can be adjusted at the drive current level on the premise of continuous lighting, or a combination of this and PWM control is also possible. .

  By executing this indoor illumination in the approach scene SCN1, the lighting mounted on the automobile can be used as illumination for the user's welcome effect, which contributes to raising the mood. Also, at night or in a dark place, there is an advantage that the position of the parked automobile can be easily grasped. It may be executed in the leaving scene SCN6 to perform a user see-off effect.

  Also, in this case, as the lighting device, other lighting devices that illuminate the space outside the vehicle mounted on the vehicle (head lamps, tail lamps, hazard lamps, etc .: indoor lights can also illuminate the space outside the vehicle due to leakage through the window glass. Can be added). This further increases the above effect.

  In addition, the hospitality control unit switches at least one of the operation of the hospitality operation unit that is already in operation and the start of the operation of the non-operational hospitality operation unit (of course, a combination of both operations between a plurality of hospitality operation units). The content of the operation of the hospitality operation unit can be changed stepwise depending on the distance between the user and the automobile. According to this configuration, an appropriate threshold value is set for the distance between the user and the vehicle, and a simple determination as to whether or not the vehicle approaches the vehicle by exceeding the threshold value, and switching control of operation / non-operation for each hospitality operation unit is performed. The above-described effect can be achieved by a relatively lightweight control process.

  In the case of the hospitality operation by the lighting device, the hospitality control unit can control lighting of the lighting device so that the total amount of illumination light decreases as the distance between the user terminal device and the automobile decreases. When the distance to the car is long, increasing the total illumination light quantity makes it easier to understand whether there are any obstacles etc. between the car and it is necessary if the distance to the car decreases. In order to illuminate only the field of view, if the amount of light is reduced, waste can be eliminated, and glare when approaching the illumination light source can be reduced.

  Next, a car audio system that functions as a hospitality operation will be described. In the car audio system, the music performance control information for hospitality related to the music that is reproduced in a manner that reflects the performance of the music selection performance storage unit 403 (formed in the storage device 535 in FIG. 11) stored on the vehicle side. (Including song identification information and volume control information) is created, and the reproduction of the song is executed based on the song list as the predetermined scene arrives.

  FIG. 15 shows an example of the configuration of the car audio system 515. The hospitality music performance control information such as music identification information and volume control information is inputted from the hospitality decision making section 2 through the hospitality execution control section 3. It has an interface unit 515a. The car audio system 515 is also responsible for the function output portion of the automobile hospitality information providing apparatus of the present invention. The interface unit 515a has a digital audio control unit 515e and a music source database 515b storing a large number of music source data. , 515c (the former is an MPEG3 database and the latter is a MIDI database). The music source data selected based on the music identification information is sent to the audio control unit 515e via the interface unit 515a, where it is decoded into digital music waveform data, converted to analog by the analog conversion unit 515f, and then the preamplifier 515g and power The signal is output from the speaker 515j through the amplifier 515h at the volume specified by the hospitality music performance control information.

  FIG. 16 shows an example of the database structure of the music source data, in which music source data (MPEG3 or MIDI) is stored in association with the song ID, song name, and genre code. In addition, each music source data includes the personality type estimated for the user who selected the music (“active”, “adult”, “optimistic”, “pessimistic”, “degenerate”, “athletic” ”,“ Intellect ”,“ romanticist ”, etc., as well as the age code (“ infant ”,“ children ”,“ junior ”,“ young ”,“ middle age ”,“ middle age ”,“ mature age ” “Respect for the elderly”, “age unrelated”, etc.) and gender codes (“male”, “female” and “gender unrelated”) are stored in association with each other. The personality code is one of the user personality specifying information, and the age code and the sex code are sub-classes unrelated to the personality. Even if the personality of the user can be identified from the user-specific information, selecting a music source that does not match the age group or gender halves the effect of “hospitality” to entertain the user. Therefore, in order to further narrow down the suitability of the music source provided to the user, the sub-classification as described above is effective.

  On the other hand, each music source data also stores a song mode code in association with each other. The song mode code is data indicating the mental state and physical condition of the user who selected the song, and the relationship between the song and the song. In the present embodiment, the song mode code is “smooth”, “exhilarating”, “warm / healing”. And “Healing / α-wave system”. Since the personality type code, age code, gender code, genre code, and song mode code are data to be referred to when selecting the hospitality content specific to each user, these are collectively referred to as hospitality reference data. .

  The car audio system 515 is provided with an operation unit 515d (FIG. 15) that is manually operated by the user, and by inputting music selection data from the operation unit 515d, desired music source data can be read and played. The volume / tone control signal from the operation unit 515d is input to the preamplifier 515g. This music selection data is transferred from the interface unit 515a to the hospitality decision making unit 2 via the hospitality execution control unit 3 of FIG. 11, and stored as music selection result data 403 (FIG. 17) in the storage device 535 connected thereto. . When a predetermined scene arrives, the hospitality decision making unit 2 creates a song list based on the stored contents, and the songs included in the song list are sequentially played. The hospitality decision making unit 2 may sequentially select songs one by one.

  However, if the user does not like the song automatically selected by the hospitality decision making unit 2, the user can switch the performance to a favorite song at any time by an input from the operation unit 515d (FIG. 15). When the user selects the music by himself / herself, as shown in FIG. 17, the specific information (user name or user ID) of the user, the ID of the selected music source data, and the aforementioned hospitality reference data (character type code, (Age code, sex code, genre code, and music mode code) are stored in the music selection record storage unit 403 in association with each other. In this embodiment, the date and time of music selection, the user's sex and age are also stored.

  As shown in FIG. 18, the music selection record storage unit 403 generates statistical information 404 (stored in the storage device 535 in FIG. 11) of the music selection record for each registered user. In this statistical information 404, music selection data is counted for each personality code (SKC), and which personality type of music is selected most frequently as a numerical parameter. As the simplest process, it is possible to specify the personality type having the highest frequency of music selection as the personality of the user. For example, if the number of music selections accumulated in the statistical information 404 reaches a certain level, for example, the personality type initially set by user input is replaced with the personality type derived from the statistical information 404 as described above. That's fine.

  The music playback is executed based on the arrival of the preparation scene SCN3 and the driving / staying scene SCN4, or based on the operation of the user operation unit 515d, so that the car audio device mounted on the car can be more comfortable. Can be used. In addition, by executing it in the approaching scene SCN1 (or leaving scene SCN6), it is possible to perform the user welcome (see off) effect, which contributes to improvement or improvement of the user's mental state.

  Next, the door assist mechanism 541 will be described. The door assist mechanism 541 controls the drive torque of the actuator (door assist motor) when the door is opened and closed based on age information (age code) stored for each registered user on the vehicle side.

  As shown in FIG. 3B, a swing-type door (hereinafter also simply referred to as “door”) 1101 for getting on and off is attached to an automobile via a door turning shaft (not shown) at one edge of the entrance. . The door 1101 can be opened from a closed position where the doorway is closed by manual operation to an arbitrary angle position. The manual door opening operation is power-assisted by a motor (actuator: not shown). In this embodiment, the rotational output of the motor is directly torqued to the turning shaft as a rotational driving force for the door opening / closing assist while torque is increased via the reduction gear mechanism with respect to the door turning shaft (not shown) rotating together with the door 1101. I try to communicate.

  The door assist mechanism 541 includes obstacle detection means (for example, a well-known proximity switch, a reflective optical sensor (including an infrared type), an ultrasonic sensor, etc.) that detects an obstacle outside the vehicle that interferes with the door when the door is opened. If the obstacle is not detected, the actuator control means enters the normal assist mode in which a positive assist force in the door opening direction is generated by the actuator when the door is opened. When the object detecting means detects an obstacle, the door opening operation is performed at least halfway in the door turning section from the closed position where the door closes the entrance to the collision position where the door collides with the obstacle when the door is opened. That controls the door assist by the actuator so that the collision between the obstacle and the door is suppressed. And configured.

  The actuator is a motor that can rotate in both forward and reverse directions, and is configured by a DC motor in this embodiment (of course, other types of motors such as an induction motor, a brushless motor, and a stepping motor may be used). The actuator control means rotates the motor in the forward direction in the positive assist mode and rotates the motor in the reverse direction in the reverse assist mode. In this embodiment, the actuator control means is a bidirectional linear control type using a push-pull transistor circuit. The motor driver constitutes the actuator control means.

  As is well known, the door 1101 can be opened and closed by either the vehicle outer operation knob or the vehicle inner operation knob, and when the lock button is tilted, the door 1101 can be opened by a known door lock mechanism 513. Open / close operation is disabled. When the door lock mechanism 513 is locked, the motor is stopped.

  In the door lock mechanism 513, the main part of the lock part is fixed to the door side engaging part (in the present embodiment, a male spline) provided on the turning shaft of the door 1101, and to the vehicle body side. A vehicle body side engaging portion (female spline in this embodiment) that engages detachably at an arbitrary angle phase according to the turning angle of the turning shaft and locks the turning of the turning shaft at the angle phase of the engaged state. And have. The vehicle body side engagement portion is provided so as to be able to approach and separate from the door side engagement portion in the axial direction of the turning shaft, and is brought into a lock engagement state with the vehicle body side engagement portion when approaching, and unlocked when separated. In this embodiment, this approach / separation mechanism is configured by a known solenoid mechanism (cylinder mechanism or the like). The limit angle position of the lock unit is variably determined according to the door angle position when the obstacle detection unit detects the obstacle (that is, according to the position of the obstacle). Thereby, regardless of the relative distance of the obstacle with respect to the automobile, if the door is approached by the opening operation, the door opening operation is prevented so as not to hit the obstacle.

  The door assist mechanism 541 is provided with an operation force detection unit 1002 such as a torque sensor that detects the operation force of the door 1101, and the door operation force detected by the operation force detection unit 1002 is detected by the actuator control unit in the positive assist mode. The operation of the actuator is controlled so that the positive assist force increases as the value decreases. That is, when the boarding scene SCN2 or the getting-off scene SCN5 arrives, the hospitality decision-making unit 2 recognizes the user who gets in or gets off, and also recognizes the age information (age or age code (“infant”, “ Child), “junior”, “youth”, “middle age”, “middle age”, “mature age”, “respect for the elderly”, “age irrelevant”, etc.)), and based on the acquired age information When a person of an age estimated to be weak tries to open the door 1101, the positive assist force by the motor of the corresponding door acts strongly and the door 1101 can be opened easily. On the other hand, when a person of an age estimated to have a strong force tries to open the door strongly, the positive assist force works relatively weakly. For example, if the control is performed so that the sum of the door opening torque generated by the external operating force and the door opening torque generated by the positive assist force is substantially constant, the door 1101 can be opened at a substantially constant standard torque regardless of who is operated. it can.

  It should be noted that, instead of using the swing type door having the assist mechanism as described above, it is also possible to employ the well-known electric automatic opening / closing mechanism or the electric sliding door 1102 of FIG. 3B provided with an assist mechanism. .

  Next, the hospitality operation by the user terminal device 100 will be described. The mobile phone 100 can set and reproduce sound information (music, sound effects, voice, etc.) that flows when the user approaches the vehicle for each registered user.

  Next, the user terminal device 100 is configured as a mobile phone in the present embodiment (hereinafter also referred to as “mobile phone 1”). The mobile phone 1 includes a monitor constituted by a liquid crystal display or the like, an input unit including a keyboard, a transmitter, a receiver, and the like. A pulse sensor (heart rate sensor) 342 is also provided on the side surface.

  FIG. 19 is a block diagram showing an example of the electrical configuration of the mobile phone 1. The main part of the circuit includes an input / output unit 1311 and a control unit 310 including a CPU 312, a ROM 314, a RAM 313, and the like connected thereto. An input unit 305 and an on-hook / off-hook switch 306 are connected to the input / output unit 1311. The receiver 303 is connected to the input / output unit 1311 via the amplifier 315 and the D / A converter 316, and the transmitter 304 is connected to the input / output unit 1311 via the amplifier 317 and the A / D converter 318. Furthermore, a well-known GPS 554 for acquiring position information of the mobile phone 1 is connected to the input / output unit 1311.

  In this embodiment, in order to grasp the distance and azimuth relationship between the user side terminal device 1 and the vehicle, the user side terminal device 1 is provided with the GPS 554 in addition to the GPS 533 on the vehicle side (FIG. 11). A method is adopted in which the terminal device 1 can independently acquire the position information, and the terminal position information is transmitted to the vehicle side via a wireless communication network. As a result, the vehicle side can acquire both the accurate vehicle position by the GPS 533 connected to itself and the accurate terminal position by the GPS 554 received from the user side terminal device 1, and the user side terminal device 1 and the vehicle The distance and azimuth relationship can be grasped very accurately. Moreover, the distance change and the approach direction change between the user-side terminal device 1 and the automobile can be grasped in real time.

  Next, the mobile phone 1 is provided with the following functions for outputting ringtones and playing music. That is, ringtone data and music data (MPEG3 data or MIDI data: also used as a ringtone) downloaded by wireless reception are stored in the sound data flash ROM 316. In the case of MIDI data, musical tone data in which tone color, pitch, tone length, tempo, etc. are described according to the MIDI code is sent to the musical tone synthesis unit 357. The tone synthesizer 357 reads the waveform data of the specified tone color from the waveform ROM 358 functioning as a sound source while buffering the tone data, converts the frequency so that the pitch specified by the MIDI code is obtained, and specifies the tone data. This is sequentially output as digital waveform data according to the set tempo. The output digital waveform data is output from the speaker 311 via the analog conversion circuit 359 and the amplifier 350. Note that sound data composed of compressed waveform data such as MPEG3 is output from the speaker 311 via the analog conversion circuit 359 and the amplifier 350 after being decoded. In this embodiment, the timing information of the voice output by the tone synthesizer 357 is input to the sequencer 352, and the vibrator unit 354 and the LED unit 355 are driven in synchronization with music via the PWM unit 353, so that the sound output from the mobile phone 1 is output. The device has been devised to further enhance the hospitality effect in combination with vibration and LED light emission.

  When the user approaches the vehicle, in addition to the lighting device as described above, a speaker (voice output unit) 311 provided in the mobile phone 1 (user side terminal device) can be used as a hospitality operation unit. In this case, the communication device 4 on the automobile side detects the approach of the mobile phone 1, that is, the user, and the speaker 311 with different output contents according to the personality type corresponding to the user (that is, the acquired user biological state information). The voice for hospitality is output. In the present embodiment, the voice data for hospitality is music source data, but it may be data of sound effects or human voices (so-called incoming voices or incoming voices). The voice data for hospitality may be stored in the storage device 535 on the automobile side shown in FIG. 11, and necessary data may be distributed to the mobile phone 1 via the communication device 4 or the mobile phone 1 side. The sound data may be stored in the sound data flash ROM 316. Here, the latter case will be described as an example.

  First, when a user is recognized by the face recognizing unit 10, user specifying information is output from the face recognizing unit 10, and the hospitality determination unit 2 receives the input and specifies the user. When a user is specified and a corresponding scene (approaching scene SCN1) arrives, the voice data ID (stored in the sound data flash ROM 316) corresponding to the user is acquired wirelessly from the mobile phone 1. (Here, the case of using MIDI data is taken as an example). Next, music source data corresponding to the ID is selected, and the mobile phone 1 starts playing. Note that voice data for hospitality is set according to the approach distance of the user. In addition, the vibrator unit 354 and the LED unit 355 may be driven in synchronization with music to further increase hospitality by sound output. In addition, the lighting device on the automobile side may be blinked in conjunction with music. Furthermore, when the user finally approaches the point of getting into the car, the power window may be activated to open the window and output in synchronization with the performance of the mobile phone 1 to further enhance the excitement effect. Furthermore, on the automobile side, if the pitch code of the main melody portion of the MIDI data is changed so as to be lower (or higher) by the frequency that forms the Kyowa pitch than the main melody portion of the mobile phone 1, the mobile phone 1 The output of the telephone 1 and the output of the car audio system 515 can be added, and the output timing of the main melody portion of the MIDI data is delayed (or advanced) by a fixed number of beats relative to the main melody portion of the mobile phone 1 side. If the output is changed as described above, a ringing effect can be achieved between the output of the mobile phone 1 and the output of the car audio system 515.

  The car navigation device 534 will be described. In the car navigation device 534, setting information set for each registered user registered on the vehicle side can be applied to, for example, destination setting of the car navigation device.

  FIG. 20 is a block diagram illustrating a configuration example of the car navigation device 534 of FIG. The car navigation device 534 serves as a function output part of the hospitality information providing device of the present invention, and includes a position detector 101, a map data input device 106, an operation switch group 107, a remote control (hereinafter referred to as a remote control) sensor. 111, a speaker 115 for performing voice guidance, a non-volatile memory 109 composed of a flash memory, a monitor 110 composed of an LCD, etc., an information system ECU 51 constituting these connected main control units, a remote control terminal 112, and a main storage device And the like. The nonvolatile memory 109 stores information and data necessary for operation of the navigation device such as automobile history information.

  The position detector 101 includes a well-known geomagnetic sensor 102, a gyroscope 103, a distance sensor 104, and a GPS receiver 105 for GPS that detects the position of an automobile based on radio waves from a satellite. Since these sensors 102, 103, 104, and 105 have errors of different properties, they are configured to be used while being complemented by a plurality of sensors. Depending on the accuracy, some of the sensors described above may be used, and further, a steering rotation sensor, a wheel sensor of each rolling wheel, or the like may be used.

  Although the operation switch group 107 can use a mechanical switch or the like, in this embodiment, the touch panel 122 integrated with the monitor 110 is used together, and a touch panel area corresponding to a button image displayed on the monitor 110 is displayed. The operation state can be recognized by touching with a finger (so-called soft button). Various instructions can be input through the operation switch group 107 and the remote control terminal 112.

  In addition to the operation switch group 107 and the remote control terminal 112, it is also possible to input various instructions using the voice recognition unit 130. In this method, when a voice is input from a microphone 131 connected to the voice recognition unit 130, the voice signal is voice-recognized by a known voice recognition technique and converted into an operation command corresponding to the result. .

  The information system ECU 51 is configured as a normal computer, and includes a well-known CPU 181, ROM 182, RAM 183, the above-described nonvolatile memory 109, and an input / output unit 184 connected by a bus. The aforementioned HDD 121 is bus-connected via the interface 129f. In addition, the drawing LSI 187 responsible for outputting an image to the monitor 110 based on the drawing information for displaying the map and the navigation operation screen and the graphic memory 187M for drawing processing are similarly connected by bus, and the monitor 110 described above is connected thereto. It is connected to the. The CPU 181 controls the navigation program 21p and data stored in the HDD 121. The CPU 181 controls the reading / writing of data to / from the HDD 121.

  The HDD 121 stores map data 21m including road data and a destination database 21d. In addition, the user can independently write the data 21 u such as point registration. The contents of these data can be rewritten by operating the operation switch group 107 and the remote control terminal 112 or by voice input. It is also possible to update the contents of the HDD 121 by reading data from the storage medium 120 using the external information input / output device (map data input device) 106. In this embodiment, the information system ECU 51 is connected to the serial communication bus 127 forming the in-vehicle network via the communication interface 126 and the buffer memory 126f for temporarily storing communication data, and the body system ECU 142 and the engine control ECU ( Data is exchanged with other control devices in the vehicle such as (not shown).

  The hospitality execution control unit 3 in FIG. 11 has been described by taking as an example a case where it is configured by a single ECU so as to facilitate understanding of the concept so far. It is more realistic to configure the execution control unit 3 in the ECU. Further, various data in the storage unit 535 used for the hospitality decision making by the hospitality decision making unit 2 and data (for example, a destination database 21d (FIG. 20) and music to be described later) that are output targets of the hospitality information providing device for automobiles. At least a part of the data stored in the source databases 515b and 515c (FIG. 15) may be stored in a data server outside the automobile and downloaded and used by the automobile by wireless communication.

  The monitor 110 is constituted by a color liquid crystal display, and the current position mark of the automobile input from the position detector 101, the map data 21m input from the HDD 121, and a guide route for displaying on the map are displayed on the screen. A menu button for setting route guidance, guidance during route guidance, and switching the screen is displayed on the screen.

  The FM tuner 141 mounted on the car stereo 515 receives broadcast radio waves that carry road traffic information for traffic jam display from, for example, a VICS (Vehicle Information and Communication System) center 114, and passes through an in-vehicle network. The information is sent to the information system ECU 51.

  In the car navigation device 534, the navigation program 21p is activated by the CPU 181 of the information system ECU 51. The driver selects a desired destination from the destination database 21d by operating the operation switch group 107 or the remote control terminal 112 or by inputting voice from the microphone 131. For example, when the route guidance process for displaying the destination route on the monitor 110 is selected from the menu displayed on the monitor 110, the following process is performed. That is, when the driver inputs a destination based on the map on the monitor 110 or the destination selection screen, the current position of the vehicle is obtained based on the satellite data obtained from the GPS receiver 105, and the destination is determined from the current position. The process for obtaining the optimum route to is performed. Then, the guidance route is superimposed on the road map on the monitor 110 to display an appropriate route for the driver. As a method for automatically setting an optimal route, a method such as the Dijkstra method is known. In addition, at least one of the monitor 110 and the speaker 115 performs a guidance according to an operation or a message according to an operation state.

  FIG. 21 shows the contents of the destination database 21d used in the car navigation device 534. Similar to the music source database, the location information of each destination, together with an ID for specifying the individual destination, Classification information composed of similar hospitality reference data is given. Of these, the personality code, age code and gender code are almost the same as those used in the music source database. In other words, the personality type estimated for the user who chooses the destination (“active”, “soft”, “optimistic”, “pessimistic”, “degraded”, “athletic”, “intellect”, Personality code indicating “romanticist”, etc., as well as age code (“infant”, “child”, “junior”, “young”, “middle age”, “middle age”, “middle age”, “respect for the elderly”, “age unrelated” Etc.) and gender codes (“male”, “female” and “gender unrelated”) are stored in association with each other. The personality code is one of the user personality specifying information, and the age code and the sex code are sub-classes unrelated to the personality. Even if the personality of the user can be specified, if the destination that does not match the age group or gender is selected, the effect of “hospitality” to entertain the user is halved. Therefore, in order to narrow down the suitability of the destination proposed to the user, the above-described sub-classification is effective.

  In addition, the genre code classifies the facilities that make up the destination according to the type, and “restaurant”, “entertainment facility”, “park”, accommodation facility, “road related service facility”, “convenience store”, “ It is divided into “supermarkets”. Among these, “restaurants”, “road-related service facilities”, “convenience stores”, “supermarkets” and the like are positioned as eating and drinking facilities.

  Each genre code is assigned a sub-classification code that matches the genre code. In the case of “restaurant”, the type of sub-classification code is considered so that the destination selection associated with the physical condition and mental state of the user can be made in consideration of the “hospitality” effect. In other words, restaurants that should be selected when they are in good health and have an appetite (especially adolescents / matures) and are hungry are given a sub-classification code that gives priority to satiety (“Gutsuri-Kokori”). Sub-classification codes that prioritize light meals for restaurants that should be selected in situations where the physical condition and appetite are not good (especially women), or at least hungry. Karuku and light-colored "). In addition, subcategory codes ("chic and fashionable") that prioritize atmosphere-oriented meals are recommended for restaurants that should be selected for situations where fatigue and progress are desirable, or when you want to enhance the mood of couples, etc. Has been granted.

  In addition to the priority of the “Hospitality” effect, sub-classification codes based on general cuisine types (“Japanese food / sushi”, “Chinese / ramen”, “Western food / curry”) will also be added separately so that they can be selected as appropriate. It has become.

  On the other hand, for recreational / entertainment service providing facilities such as recreational facilities (or tourist spots) and parks, the type of sub-classification code is set so that the destination can be selected in relation to the physical condition and mental state of the user. Has been taken into account. In other words, if you are in good health, want a cheerful and active service (especially adolescents / matures, etc.), you should choose a sub-classification code that gives priority to physical or mental divergence. "Spot") is given, and when the physical condition is not so great or tired (especially women), a sub-classification code ("Relaxing / Healing") giving priority to restraining physical exhaustion is given . In addition, an atmosphere-oriented sub-classification code (“love love spot”) is assigned to facilities that should be selected when it is desired to enliven the mood with a couple.

  On the other hand, the “road-related service facility” is assigned sub-classification codes of “service area”, “parking area”, “road station”, and “drive-in”.

  In this embodiment, since the user is recognized by the face recognition unit 10, when searching for a destination or a music source by collating with the registered information (age, gender, personality) of the recognized user, Refinement by code, gender code and personality code is possible immediately. In other words, a search suitable for the user is performed.

  In the car navigation device 534, for each registered user, user-specific navigation setting information (memory point information, destination selection results, priority condition setting related to route setting, etc.) is also stored in the user data 21u. Accordingly, when the user is recognized by the face recognition unit 10, user identification information is output from the face recognition unit 10, and the hospitality determination unit 2 receives the input and identifies the user, and further, a corresponding scene (for example, preparation) When the scene SCN3) arrives, the hospitality decision making unit 2 reads the input user identification information from the ROM of the hospitality decision making unit 2 and outputs it to the hospitality execution control unit 3, and the hospitality execution control unit 3 The specific information is output to the car navigation device 534. And the car navigation apparatus 534 can reflect a setting based on the user specific information.

  In addition, the car navigation device 534 can receive schedule information corresponding to the destination from the external scheduler 2001. When the schedule information is determined corresponding to the user, the schedule information can be received and the schedule can be notified (by voice or screen display) when the user is recognized.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the above-mentioned embodiment, The following embodiment is also contained in the technical scope of this invention, and also the summary other than the following is also included. Various modifications can be made without departing from the scope.

The block diagram which shows an example of the electrical constitution of the control system for motor vehicles in which the face recognition apparatus of this invention was integrated. The block diagram which shows an example of an electrical structure of the face recognition apparatus of this invention. The figure explaining arrangement | positioning of a vehicle interior imaging device. The figure explaining arrangement | positioning of an imaging device outside a vehicle. The figure explaining arrangement | positioning of an illuminating device. The graph explaining the characteristic of an illuminating device. The graph explaining the spectral sensitivity of an imaging device. The figure explaining the 1st Example of the imaging timing of an imaging device and the lighting timing of an illuminating device. The figure explaining the 2nd Example of the imaging timing of an imaging device and the lighting timing of an illuminating device. The flowchart explaining the 1st Example of a user recognition process. The flowchart explaining a face recognition process. The flowchart explaining the 2nd Example of a user recognition process. 1 is a block diagram showing an example of an electrical configuration of an automobile user hospitality system incorporating a face recognition device of the present invention. The conceptual diagram which shows the content of user registration information. The block diagram which shows an example of the electrical structure of vehicle interior lighting. The conceptual diagram which shows the structural example of the lighting control data of an illuminating device. The block diagram which shows an example of the electrical constitution of a car audio system. The conceptual diagram which shows the content of a music source database. The conceptual diagram which shows the content of the music selection performance memory | storage part. The conceptual diagram which shows the content of the statistical information of a music selection track record. The block diagram which shows an example of the electrical constitution of a user side terminal device. The block diagram which shows the electrical structural example of the car navigation apparatus integrated in the user hospitality system for motor vehicles. The conceptual diagram of a destination database. The schematic diagram which shows the 1st example of the input method of character classification determination. The schematic diagram which shows the 2nd example of the input method of character classification determination.

Explanation of symbols

1 Automotive control system (Automobile user hospitality system)
10 Face recognition device (face recognition unit)
11 Control unit (user recognition means, user recognition result output means, illumination control means, face presence determination means, feature information synthesis means, user recognition result output control means)
15 (518, 521) Shooting device (in-vehicle camera (face camera), out-of-vehicle camera (external camera))
16 Lighting device 20 System master device (master control device)
31 photographing object detection unit 17 face registration unit 2 hospitality decision making unit (scene identification means, hospitality control unit, mental / physical state estimation means)
3 Hospitality execution control unit (hospitality control unit)
100 Mobile phone (user terminal device)

Claims (10)

An imaging device that is attached to the inside or outside of an automobile and that captures the face of a person existing within a predetermined field of view;
A face registration unit that stores face feature information of a person registered as a user of the car in association with user-specific information;
An illumination device that illuminates the face of a person present in the field of view of the imaging device;
User recognition means for acquiring face feature information from a face image photographed by the photographing device, and performing user recognition processing by collating the acquired face feature information with the face feature information registered in the face registration unit;
User recognition result output means for outputting the user recognition result;
Lighting control means for turning on the lighting device when user recognition by an image taken in a non-lighting state of the lighting device fails;
A face recognition device comprising: The imaging device is used as an image sensor having sensitivity necessary for face photography in both the visible light region and the infrared region, and enables face photographing with both visible light and infrared light,
The face recognition device according to claim 1, wherein the illumination device has an infrared light source as an illumination light source.   The face recognition apparatus according to claim 2, wherein the infrared light source is a near infrared light source.   4. The face recognition device according to claim 1, wherein a plurality of illumination devices that illuminate faces existing in the field of view of the photographing device from different directions are provided. 5.   The face recognition apparatus according to claim 4, wherein the illumination control unit lights a plurality of the illumination apparatuses at once. Face presence determination means for determining whether or not a person's face is present in the field of view based on a photographed image of the photographing device;
When the face presence determining unit determines that the face is present, the photographing device performs primary face photographing with the lighting device being in a non-lighted state, and the user recognizing unit is a primary user based on the face image in the non-lighted state. While performing recognition processing,
When the user identification by the primary user recognition process has failed, the lighting control unit turns on the lighting device, and in that state, the photographing device performs secondary face photographing with the lighting device turned on, and the user recognition 4. The face recognition apparatus according to claim 1, wherein the means performs secondary user recognition processing based on the face image in the lighting state.   5. The apparatus according to claim 4, wherein the lighting device is turned on in different types of lighting devices for lighting the secondary face photographing by the photographing device and the secondary user recognition processing by the user recognition means. The face recognition apparatus according to claim 6, wherein the face recognition apparatus is repeatedly executed while being sequentially changed.   The feature information synthesizing means for synthesizing the feature information from each face image obtained by multiple times of face photographing is provided, and the user recognition means performs the user recognition based on the synthesized feature information. Item 8. The face recognition device according to any one of items 7 to 9. User recognition processing repetitive control means for periodically performing face photographing by the photographing device and user recognition processing by the user recognizing means based on the photographed face image;
Even after the user recognition means has succeeded in user recognition once and the user recognition has failed in the subsequent iteration of the user recognition processing, the user recognition means until the number of times of failure recognition reaches a predetermined number of times. 9. The face recognition apparatus according to claim 1, further comprising: a user recognition result output control unit that prevents the recognition result output unit from outputting the failure recognition result. The face recognition device according to any one of claims 1 to 9,
Hospitality to assist the user in using the car or to entertain the user at least one of approaching the user, riding, driving or staying in the car, getting off and leaving the car A hospitality operating part that performs the operation;
A hospitality operation information storage unit that stores the registered user and the hospitality operation information that defines the operation content of the hospitality operation unit unique to each registered user in association with each other;
Hospitality operation information corresponding to the registered user recognized by the face recognition device is read from the hospitality operation information storage unit, and the hospitality execution control unit performs operation control of the hospitality operation unit according to the hospitality operation information;
An automobile user hospitality system characterized by comprising:

Images