KR101892009B1 - User recognition apparatus and method thereof - Google Patents

Abstract

The present invention relates to a user recognition apparatus and method for accurately recognizing a user's face even when a user's face angle changes, and a user recognition apparatus according to embodiments of the present invention includes a camera; A storage unit; Monitoring a face angle of a user in the vehicle through the camera when the vehicle is running and photographing a first image of a face located within the reference tolerance when the monitored face angle is within a reference tolerance, And a controller for automatically registering the photographed first image in the storage unit.

Description

[0001] USER RECOGNITION APPARATUS AND METHOD THEREOF [0002]

The present invention relates to a user recognition apparatus and a method thereof.

In general, biometrics technologies used for personal information protection and user authentication include fingerprint recognition, iris recognition, vein recognition, and the like. Of these biometrics technologies, face recognition technology is less intrusive to users than other biometrics information, and it receives a lot of attention because it has less sense of rejection because of non-contact type.

Wide - area face recognition process can be divided into face detection and face recognition process. The face detection process is a step of extracting a face region of a person from an input image. In order to extract a face of a specific person to be recognized in an image in which various faces acquired in a general situation and a plurality of faces can be displayed together It is a necessary process.

The face area detection method that has been studied so far includes a knowledge based method using low level features of a face, a template based face detection method based on a specific template type, a feature using unchanging structural features of a face feature-based methods, and appearance-based methods using statistical features of faces.

In addition, conventionally, only the front reference detection is performed at the time of face authentication so that the driver restricts the operation of the driver after boarding, and it is impossible to cope with the face authentication due to the rotation of the driver's face, or the face recognition rate .

An object of the present invention is to provide a user recognition apparatus and method which can accurately recognize a user's face even if the user's face angle changes.

A user recognition apparatus according to embodiments of the present invention includes: a camera; A storage unit; Monitoring a face angle of a user in the vehicle through the camera when the vehicle is running and photographing a first image of a face located within the reference tolerance when the monitored face angle is within a reference tolerance, And a controller for automatically registering the photographed first image in the storage unit.

In an embodiment of the present invention, the control unit may include a controller for photographing face images corresponding to a plurality of different face angles through the camera, and automatically registering the photographed face images in the storage unit .

As an example related to the present invention, the control unit may be configured such that when each of the different facial angles of the photographed face images is within a respective reference permissible range corresponding to each of different facial angles, Images can be captured, and the photographed images can be automatically registered in the storage unit.

As an example related to the present invention, the reference permissible range may be a plurality of different reference permissible ranges corresponding to different face angles.

As an example related to the present invention, the control unit may temporarily register the photographed first image in the storage unit.

In one embodiment of the present invention, the control unit temporarily registers the photographed first image in the storage unit, and then matches the first angle when the monitored face angle is matched with the first angle within the reference tolerance range And register the captured second image in the storage unit instead of the first image.

As an example related to the present invention, the first angle may be a fixed angle belonging to the reference tolerance range.

As an example related to the present invention, when the first image photographed through the camera does not match an image already registered in a specific storage area of the memory, the control unit regards the image already registered in the specific storage area as an error image And register the first image photographed by the camera in the specific storage area instead of the error image.

As an example related to the present invention, when the vehicle stops for a preset time after a point in time when a first image photographed through the camera does not match an image already registered in a specific storage area of the memory, A message inquiring whether to register a first image photographed through a camera is displayed on the display unit and the first image can be registered in the storage unit when an input requesting to register the first image is received have.

A user recognition method according to embodiments of the present invention is a user recognition method of a user recognition device mounted on a vehicle, the method comprising: monitoring a face angle of a user in the vehicle through the camera when the vehicle is traveling; Capturing a first image of a face located within the reference tolerance when the monitored face angle is within a reference tolerance; And automatically registering the photographed first image in a storage unit.

The user recognition apparatus and method according to embodiments of the present invention recognize a user's face based on user's face images photographed in different directions so that the user's face can be recognized accurately even if the user's face angle changes.

1 is a block diagram showing a vehicle for explaining embodiments of the present invention.
2 is a configuration diagram showing the configuration of a vehicle 100 for explaining embodiments of the present invention.
3 is an exemplary view showing installation positions of the first and second cameras according to the embodiments of the present invention.
4A-4B are exemplary diagrams illustrating first and second user face images taken according to embodiments of the present invention.
Figures 4C-4D are further illustrations of first and second user face images taken in accordance with embodiments of the present invention.
5 is an exemplary view showing an installation position of a camera according to embodiments of the present invention.
6 is a flowchart illustrating a method of recognizing a user according to an embodiment of the present invention.
7 is a diagram illustrating an example of a method of registering an operator's face image according to an exemplary embodiment of the present invention.
8 is a diagram illustrating another method of registering an operator's face image according to an embodiment of the present invention.
9 is a diagram illustrating another method of registering an operator's face image according to an exemplary embodiment of the present invention.
FIG. 10 is an exemplary view showing another method of registering an operator's face image according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar components are denoted by the same reference numerals, and redundant explanations thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

1 is a block diagram showing a vehicle for explaining embodiments of the present invention.

1, the vehicle 100 receives traffic information and various data (for example, a program, an executable file, and the like) from the information providing center 30, and communicates with the remote wireless communication network 20 and / Receives the traffic information through the short-range wireless communication network, and provides the guidance service based on the GPS signal received through the satellite 10 and the traffic information. Here, the communication network may further include a wired / wireless communication network such as a LAN (Local Area Network) and a WAN (Wide Area Network).

Traffic information such as traffic light information (eg, road traffic information, interest area information) is collected through the communication network and the collected information is transmitted from the information providing center 30 (for example, a server) Group) standard and transmitted to the broadcasting station. Accordingly, the broadcasting station inserts the traffic information including the traffic light information into the broadcast signal and broadcasts the traffic information to the vehicle 100.

The server is connected to various networks connected to a communication network, for example, an operator input, a wired / wireless Internet, a Transparent Data Channel (TDC), a digital broadcasting service such as a Multimedia Object Transport (MOC) Various traffic information, for example, TPEG (Transport Protocol

Expert Group) format and transmits it to the broadcasting station.

The server generates a traffic information format of the TPEG standard including the traffic light information

To a broadcasting station.

A broadcasting station transmits traffic information including traffic signal information received from a server to a traffic information receiving terminal mounted on the vehicle 100, for example, a navigation device, on a broadcasting signal and wirelessly transmits the traffic information. The traffic information includes traffic light information and information on various traffic conditions necessary for roads, marine, and air navigation such as accidents, road conditions, traffic congestion, road construction, road shutdown, . ≪ / RTI >

The broadcasting station receives traffic information including traffic signal information processed from a server, and transmits the traffic information to the vehicle 100 through a digital signal according to various digital broadcasting standards. In this case, the broadcasting standard may be a digital audio broadcasting (DAB) standard based on Eureka-147 [ETSI EN 300 401], terrestrial or satellite digital multimedia broadcasting (DMB), terrestrial digital video broadcasting A Digital Video Broadcasting (DVB-T) standard, a Digital Video Broadcasting-Handheld (DVB-H) standard, and an MFLO (Media Forward Link Only) standard.

Also, the broadcasting station can transmit traffic information including traffic light information through a wired / wireless network such as wired / wireless Internet.

The vehicle 100 refers to all possible carriers that are implemented using mechanical or electronic devices for the purpose of moving people or objects, such as a general passenger car, a bus, a train, a ship, or an aircraft.

The vehicle 100 mounts a traffic information reception terminal, receives traffic light information from a broadcasting station using the installed traffic information reception terminal, processes the traffic light information, and outputs the processed traffic light information to graphics, text and / or audio To the user.

2 is a configuration diagram showing the configuration of a vehicle 100 for explaining embodiments of the present invention.

2, the vehicle 100 includes a communication unit 120, an input unit 130, a sensing unit 150, an output unit 170, a vehicle driving unit 180, a memory 190, an interface unit 196 A control unit 110, a power supply unit 160, a steering device 140, a DSM (Driver Status Monitoring) system, and a vehicle display device 195.

The communication unit 120 is connected to the vehicle 100 and the other vehicle 300 to allow wireless communication between the vehicle 100 and the mobile terminal 200, between the vehicle 100 and the external server 300, Modules. In addition, the communication unit 120 may include one or more modules that connect the vehicle 100 to one or more networks.

The communication unit 120 may include a broadcast receiving module 121, a wireless Internet module 122, a short distance communication module 123, a location information module 124, an optical communication module 125, and a V2X communication module 126 have.

The communication unit 120 can receive weather information. The communication unit 120 can receive weather information from the outside through the broadcast receiving module 121, the wireless Internet module 122, or the V2X communication module 126. [

The communication unit 120 can receive the running road information. The communication unit 120 recognizes the position of the vehicle 100 via the position information module 124 and transmits the position of the vehicle 100 corresponding to the position of the vehicle 100 via the wireless Internet module 122 or the V2X communication module 126 Road information can be received.

The communication unit 120 can receive the traffic light change information from the external server 300 via the V2X communication module 126. [ Here, the external server 300 may be a server located in a traffic control station that controls traffic.

The broadcast receiving module 121 receives broadcast signals or broadcast-related information from an external broadcast management server through a broadcast channel. Here, the broadcast includes a radio broadcast or a TV broadcast.

The wireless Internet module 122 is a module for wireless Internet access, and may be embedded in the vehicle 100 or externally. The wireless Internet module 122 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA, WiBro World Wide Interoperability for Microwave Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), and Long Term Evolution-Advanced (LTE-A) (122) transmits and receives data according to at least one wireless Internet technology in a range including internet technologies not listed above. For example, the wireless Internet module 122 can exchange data with the external server 300 wirelessly. The wireless Internet module 122 can receive weather information and road traffic situation information (for example, TPEG (Transport Protocol Expert Group)) information from the external server 300. [

The short-range communication module 123 is for short-range communication and may be a Bluetooth ™, a Radio Frequency Identification (RFID), an Infrared Data Association (IrDA), a UWB (Ultra Wideband) It is possible to support near-field communication using at least one of Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct and Wireless USB (Universal Serial Bus)

The short distance communication module 123 may form short range wireless communication networks to perform short distance communication between the vehicle 100 and at least one external device. For example, the short-range communication module 123 can exchange data with the mobile terminal 200 wirelessly. The short distance communication module 123 may receive weather information and traffic situation information (e.g., TPEG (Transport Protocol Expert Group)) from the mobile terminal 200. For example, when the user has boarded the vehicle 100, the user's mobile terminal 200 and the vehicle 100 can perform pairing with each other automatically or by execution of the user's application.

The position information module 124 is a module for obtaining the position of the vehicle 100, and a representative example thereof is a Global Positioning System (GPS) module. For example, when the vehicle utilizes a GPS module, it can acquire the position of the vehicle using a signal sent from the GPS satellite.

The optical communication module 125 may include a light emitting portion and a light receiving portion.

The light receiving section can convert the light signal into an electric signal and receive the information. The light receiving unit may include a photodiode (PD) for receiving light. Photodiodes can convert light into electrical signals. For example, the light receiving section can receive information of the front vehicle through light emitted from the light source included in the front vehicle.

The light emitting unit may include at least one light emitting element for converting an electric signal into an optical signal. Here, the light emitting element is preferably an LED (Light Emitting Diode). The optical transmitter converts the electrical signal into an optical signal and transmits it to the outside. For example, the optical transmitter can emit the optical signal to the outside through the blinking of the light emitting element corresponding to the predetermined frequency. According to an embodiment, the light emitting portion may include a plurality of light emitting element arrays. According to the embodiment, the light emitting portion can be integrated with the lamp provided in the vehicle 100. [

For example, the light emitting portion may be at least one of a headlight, a tail light, a brake light, a turn signal lamp, and a car light. For example, the optical communication module 125 can exchange data with another vehicle 400 via optical communication.

The V2X communication module 126 is a module for performing wireless communication with the server 300 or the other vehicle 400. V2X module 126 includes modules capable of implementing inter-vehicle communication (V2V) or vehicle-to-infrastructure communication (V2I) protocols. The vehicle 100 can perform wireless communication with the external server 300 and the other vehicle 400 via the V2X communication module 126. [

The input unit 130 may include a camera, a driver's operation unit 131, a microphone 132, and a user input unit 133.

The microphone 132 can process an external sound signal as electrical data. The processed data can be utilized variously according to functions performed in the vehicle 100. The microphone 132 converts the voice command of the user into electrical data, and the converted electrical data can be transmitted to the control unit 110.

According to the embodiment, the camera or the microphone 132 may be a component included in the sensing unit 150, not a component included in the input unit 130. [

The user input unit 133 is for receiving information from a user. When information is inputted through the user input unit 133, the control unit 110 can control the operation of the vehicle 100 to correspond to the input information. The user input unit 133 may include a touch input means or a mechanical input means. According to an embodiment, the user input 133 may be located in one area of the steering wheel. In this case, the driver can operate the user input unit 133 with his / her finger while holding the steering wheel.

The user input unit 133 may receive a turn signal input.

The sensing unit 150 senses a signal relating to the running of the vehicle 100 or the like. To this end, the sensing unit 150 may include a sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight sensor, a heading sensor, a yaw sensor, an acceleration sensor, a gyro sensor a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, A sensor, a tire air pressure sensor, an ultrasonic sensor, a radar, a light detection and ranging (LiADAR), and the like.

Thus, the sensing unit 150 can detect the vehicle collision information, the vehicle direction information, the vehicle position information (GPS information), the vehicle angle information, the running speed information of the vehicle, the vehicle acceleration information, It is possible to acquire a sensing signal for battery information, fuel information, tire information, vehicle lamp information, vehicle interior temperature information, vehicle interior humidity information, information on rain, steering wheel rotation angle, ambient illuminance information, tire air pressure information, have.

The sensing unit 150 may include an acceleration pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor AFS, an intake air temperature sensor ATS, a water temperature sensor WTS, A position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like.

On the other hand, an ultrasonic sensor, a radar or a rider can detect and track an object. The ultrasonic sensor, the radar or the rider can calculate the distance to the detected object and the relative speed.

Ultrasonic sensors, radar or riders can detect dangerous situations. A processor included in an ultrasonic sensor, radar or rider can detect a dangerous situation based on the distance to the object.

The sensing unit 150 may include an orientation sensor. The attitude sensor can detect the attitude of the vehicle. The attitude detection sensor can generate attitude information of the vehicle.

The attitude detection sensor may include the yaw sensor, the acceleration sensor, the gyro sensor, and the vehicle body inclination sensor described above.

The sensing unit 150 may include a wind sensor. The wind sensor can sense wind direction or wind speed. The wind sensor can generate wind direction information or wind velocity information. The wind sensor may include an ultrasonic type wind sensor. The wind sensor can measure the speed and direction of the wind using the property that the transmission speed of the ultrasonic wave transmitted through the air medium increases or decreases by the wind.

The sensing unit 150 may include a biometric information sensing unit. The biometric information sensing unit senses and acquires the biometric information of the passenger. The biometric information may include fingerprint information, iris-scan information, retina-scan information, hand geo-metry information, facial recognition information, Voice recognition information. The biometric information sensing unit may include a sensor for sensing passenger ' s biometric information. Here, the internal camera and the microphone 132 may be operated as sensors. The biometric information sensing unit can acquire hand shape information and facial recognition information through an internal camera.

The output unit 170 is for outputting information processed by the control unit 110 and may include a display unit 171, an acoustic output unit 172, and a haptic output unit 173.

The display unit 171 can display information processed by the control unit 110. [ For example, the display unit 171 can display the vehicle-related information. Here, the vehicle-related information may include vehicle control information for direct control of the vehicle, or vehicle driving assistance information for a driving guide to the vehicle driver. Further, the vehicle-related information may include vehicle state information indicating the current state of the vehicle or vehicle driving information related to the driving of the vehicle.

The display unit 171 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) display, a 3D display, and an electronic ink display.

The display unit 171 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. Such a touch screen may function as a user input 133 that provides an input interface between the vehicle 100 and the user and may provide an output interface between the vehicle 100 and the user. In this case, the display unit 171 may include a touch sensor that senses a touch with respect to the display unit 171 so that a control command can be received by a touch method. When a touch is made to the display unit 171, the touch sensor senses the touch, and the control unit 110 generates a control command corresponding to the touch based on the touch. The content input by the touch method may be a letter or a number, an instruction in various modes, a menu item which can be designated, and the like.

On the other hand, the display unit 171 may include a cluster so that the driver can check the vehicle state information or the vehicle driving information while driving. Clusters can be located on the dashboard. In this case, the driver can confirm the information displayed in the cluster while keeping the line of sight ahead of the vehicle.

Meanwhile, according to the embodiment, the display unit 171 may be implemented as a Head Up Display (HUD). When the display unit 171 is implemented as a HUD, information can be output through a transparent display provided in the windshield. Alternatively, the display unit 171 may include a projection module to output information through an image projected on the windshield.

The sound output unit 172 converts an electric signal from the control unit 110 into an audio signal and outputs the audio signal. For this purpose, the sound output unit 172 may include a speaker or the like. It is also possible for the sound output section 172 to output a sound corresponding to the operation of the user input section 133. [

The haptic module (haptic output unit) 173 generates a tactile output. For example, the haptic output section 173 may operate to vibrate the steering wheel, the seat belt, and the seat so that the user can recognize the output.

The vehicle drive unit 180 can control the operation of various devices of the vehicle. The vehicle drive unit 180 may be provided with a control signal from the steering apparatus 140. The vehicle drive unit 180 can control each device based on the control signal.

The vehicle driving unit 180 includes a power source driving unit 181, a steering driving unit 182, a brake driving unit 183, a lamp driving unit 184, an air conditioning driving unit 185, a window driving unit 186, an airbag driving unit 187, A driving unit 188, a wiper driving unit 189, and a suspension driving unit.

The power source driving unit 181 may perform electronic control of the power source in the vehicle 100. [

For example, when the engine of the internal combustion engine is a power source, the power source driving section 181 can perform electronic control on the engine. Thus, the output torque of the engine and the like can be controlled. When the power source driving unit 181 is an engine, the speed of the vehicle can be limited by limiting the engine output torque under the control of the control unit 110. [

As another example, when the electric-based motor is a power source, the power source driving section 181 can perform control on the motor. Thus, the rotation speed, torque, etc. of the motor can be controlled.

The power source drive section 181 can receive the acceleration control signal from the steering apparatus 140. [ The power source driving unit 181 can control the power source in accordance with the received acceleration control signal.

The steering driver 182 may perform electronic control of the steering apparatus in the vehicle 100. Thus, the traveling direction of the vehicle can be changed.

The steering driver 182 may receive the steering control signal from the steering device 140. [

The steering driver 182 may control the steering device to be steered according to the received steering control signal.

The brake driver 183 may perform electronic control of the brake apparatus in the vehicle 100. [ For example, it is possible to control the operation of the brake (or brake device) disposed on the wheel, thereby reducing the speed of the vehicle 100 or preventing the vehicle from moving. As another example, it is possible to adjust the traveling direction of the vehicle 100 to the left or right by differently operating the brakes (brake devices) respectively disposed on the left wheel and the right wheel. The brake driver 183 may receive the deceleration control signal from the steering device 140. [ The brake driver 183 may control the brake device in accordance with the received deceleration control signal.

The lamp driver 184 can control the turn-on / turn-off of the lamp disposed in the vehicle. Also, the intensity, direction, etc. of the light of the lamp can be controlled. For example, it is possible to perform control on a direction indicating lamp, a brake lamp, and the like.

The air conditioning driving unit 185 may perform electronic control on the air conditioner in the vehicle 100. For example, when the temperature inside the vehicle is high, the air conditioner can be operated to control the cool air to be supplied to the inside of the vehicle.

The window driving unit 186 may perform electronic control of a window apparatus in the vehicle 100. [ For example, it is possible to control the opening or closing of the side of the vehicle with respect to the left and right windows.

The airbag drive 187 may perform electronic control of the airbag apparatus in the vehicle 100. [ For example, in case of danger, the airbag can be controlled to fire.

The sunroof driving unit 188 may perform electronic control of the sunroof apparatus in the vehicle 100. [ For example, the opening or closing of the sunroof can be controlled.

The suspension driving unit may perform electronic control of suspension apparatus in the vehicle 100. [ For example, when there is a curvature on the road surface, it is possible to control the suspension device so as to reduce the vibration of the vehicle 100. [ The suspension driving section can receive the suspension control signal from the steering apparatus 140. [ The suspension driving unit may control the suspension device according to the received suspension control signal.

The memory 190 is electrically connected to the control unit 110. The memory 190 may store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory 190 may store various data for operation of the vehicle 700, such as a program for processing or controlling the control unit 110. [

The memory 190 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The vehicle 100 may be operated in association with a web storage that performs the storage function of the memory 190 on the Internet.

The memory 190 may be formed integrally with the controller 110.

The interface unit 196 may serve as a pathway to various kinds of external devices connected to the vehicle 100. For example, the interface unit 196 may include a port that can be connected to the mobile terminal 200, and may be connected to the mobile terminal 200 through the port. In this case, the interface unit 196 can exchange data with the mobile terminal 200.

Meanwhile, the interface unit 196 may serve as a path for supplying electrical energy to the connected mobile terminal 200. The interface unit 196 provides the electric energy supplied from the power supply unit 160 to the mobile terminal 200 under the control of the control unit 110 when the mobile terminal 200 is electrically connected to the interface unit 196 do.

The interface unit 196 serves as a pathway to various kinds of external devices connected to the vehicle 100. The interface unit 196 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the vehicle 100, corresponding to the connection of the external device to the interface 196, it is possible to perform appropriate control relating to the connected external device.

The control unit 110 can control the overall operation of each unit, device, or each feed element in the vehicle 100. [

The control unit 110 may be referred to as an ECU (Electronic Control Unit). The controller 110 may be implemented in hardware as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs) ), Controllers, microcontrollers, microprocessors, and other electronic units for performing other functions.

The power supply unit 160 can supply power necessary for the operation of each component under the control of the control unit 110. [ In particular, the power supply unit 160 can receive power from a battery (not shown) in the vehicle.

The steering apparatus 140 can exchange data with the control unit 110. [ The various information, data, or control signals generated by the steering apparatus 140 may be output to the controller 110.

The vehicle 100 may further include a Driver Status Monitoring (DSM) system (not shown).

The DSM (Driver Status Monitoring) system is a system that detects the driver's state and controls the vehicle 100 according to the driver's state. The DSM system may include an input device such as an internal camera, a microphone, and the like.

The DSM system can sense driver conditions such as whether the driver looks ahead, is sleeping, consumes food, manipulates the device, and the like. In addition, the DSM system can detect the driving concentration of the driver during driving.

The DSM system may include a PPG (Photoplethysmogram) sensor. The PPG sensor may be disposed in a region of the steering wheel that is in contact with the user (e.g., driver) body. The PPG sensor may be disposed in one region of the steering wheel rim. The DSM system can acquire and analyze the driver's bio-signal through the PPG sensor.

For example, the DSM system can acquire the biological signal and generate the driver's body condition information using the driver's condition information.

For example, the DSM system can acquire biometric information and generate driver's excitement status information using driver status information.

For example, the DSM system can analyze the driver image acquired from the internal camera and generate driver's drowsiness status information using driver status information.

For example, the DSM system can analyze the driver image acquired from the internal camera and generate driver device operation status information.

The DSM system can provide driver status information to the steering device 140. [

The vehicle display device 195 can exchange data with the control unit 110. [ The control unit 110 may receive navigation information from the vehicle display device 195 or a separate navigation device (not shown). Here, the navigation information may include set destination information, route information according to the destination, map information about the vehicle driving, or vehicle location information.

Meanwhile, the vehicle 100 related to the present invention may include various lamps. The various lamps include, for example, a head lamp formed to irradiate visible light (light) to the front of the vehicle, a rear lamp configured to irradiate visible light (light) to the rear of the vehicle, And the like.

The rear lamp may be formed of one or more of a head lamp, a brake lamp that emits light when the brake device is operated, and a direction indicating lamp. The rear lamp may be referred to as a rear combination lamp (sub-module) in view of the combination of lamps that perform various functions.

Hereinafter, the feature points (feature areas) of face images of users (e.g., driver) photographed in different directions in the vehicle 100 are detected, the user is recognized based on the detected feature points, A user recognition apparatus and a method thereof capable of providing various services and / The user recognition apparatus and method according to embodiments of the present invention can be variously applied to the mobile communication terminal 200, the telematics terminal, the navigation (vehicle navigation) device, the television, and the like.

3 is an exemplary view showing installation positions of the first and second cameras according to the embodiments of the present invention.

3, the first camera 301 may be fixedly mounted on a ceiling of a vehicle or a side surface of a room mirror, and may face the user (for example, a driver) (Capturing) in a direction (direction). The second camera 302 may be fixedly installed on a cluster of a vehicle, a dashboard or a lower end of a windshield, and may face the user (for example, a driver) (Two directions).

The control unit 110 controls the first camera (not shown) based on a driver detection signal generated from a driver's surveillance sensor mounted on the driver's seat when a driver's unlock signal is received through the communication unit 120, 301) and the second camera 302 (for example, turn on).

The control unit 110 recognizes the user (e.g., a driver) based on the first and second user face images, and based on the setting information (vehicle information) preset by the recognized user, The position of the driver's seat in the driver's seat, the height of the steering wheel, the side mirror, and the room mirror. For example, the control unit 110 reads out the seat posture information previously set by the recognized user from the memory 190, and based on the read seat posture information, As shown in FIG. The control unit 110 reads the side mirror posture information previously set by the recognized user from the memory 190 and determines the posture of the side mirror based on the read side mirror posture information And controls a driving unit for controlling the driving unit.

The control unit 110 recognizes the user (e.g., a driver) based on the first and / or second user face images, and displays contents (e.g., photographs, moving images, Information, etc.) may be provided via an in-vehicle infotainment system or via a communication network.

The first camera 301 transmits a first user's face image obtained by capturing (capturing) a face of a user (for example, a driver) in a first direction (for example, . For example, the first camera 301 is fixedly installed on a ceiling of a vehicle or on a side of a room mirror, and photographs the face of the user (for example, a driver) in an upward direction (first direction) )do.

The second camera 302 transmits a second user face image obtained by capturing (capturing) the face of a user (e.g., a driver) in a second direction (e.g., downward direction of the user's face) . For example, the second camera 302 is fixedly installed on a vehicle cluster, a dashboard, a windshield lower end, etc., and detects the face of the user (for example, a driver) (Capture).

The controller 110 detects common feature point (s) and non-common feature point (s) of the first and second user face images. For example, the controller 110 detects positions (two-dimensional coordinates) of eye, nose, mouth, etc. as basic feature points of a face in each of the first and second user face images. The control unit 110 can distinguish whether the images photographed by the first and second cameras 301 and 302 are face images based on the basic feature points of the first and second user face images.

The control unit 110 detects intrinsic feature points (s) of the face from the first and second user face images, if the image photographed by the first and second cameras 301 and 302 is a face image. The characteristic feature point (s) of the face may be a specific part of the face such as the starting position of the eye, the large position of the eye, the eyebrow position, the eyebrow shape, the lip, the jaw line, the eye, the whisker, , Which can be detected using a known feature detector or the like.

The control unit 110 detects intrinsic feature points (matching intrinsic feature points) common to the intrinsic feature points of the first user face image and the second user face image.

The controller 110 detects a position of the detected common intrinsic feature points (common intrinsic feature regions). For example, the control unit 110 obtains three-dimensional coordinates (actual spatial coordinate system) of the detected common inherent characteristic points using a stereo matching technique or the like.

The control unit 110 generates unique feature points (non-common intrinsic feature points) (non-common intrinsic feature regions) that are uncommon among the intrinsic feature points of the first user face image and the second user face image, As shown in FIG. For example, the controller 110 obtains only the two-dimensional coordinates (image coordinate system) of the detected non-common unique feature points.

Figures 4A-4B illustrate first and second user face images taken according to embodiments of the present invention, illustrating first and second user face images taken when the user is in a straight posture.

As shown in FIGS. 4A and 4B, the controller 110 determines whether common feature points 4-1 of the first user face image 4-1 and unique feature points 4-2 of the second user face image are common to each other (Inherent feature points that can be matched).

FIGS. 4C-4D are diagrams showing another example of the first and second user face images taken according to the embodiments of the present invention. In this case, And a second user face image.

As shown in FIGS. 4C-4D, the controller 110 determines whether or not the intrinsic feature points 4-3 of the first user face image and the intrinsic feature points 4-4 of the second user face image are common to each other (Inherent feature points that can be matched).

The controller 110 may calculate distances between common intrinsic feature points of the first user face image and unique feature points of the user face previously stored in the memory 190 The distance values between the common intrinsic feature points of the second user face image and the intrinsic feature points of the user face previously stored in the memory 110 The feature points of the user's face previously photographed in two directions), and if the average value of the first difference value and the second difference value is less than a preset reference value, The user who is photographed by the cameras 301 and 302 is recognized as an authenticated user.

The controller 110 may calculate distance values between the uncommon intrinsic feature points of the first user face image and unique feature points of the user face previously stored in the memory 190 Of the second user facial image and the distance values between the distance values between the uncommon intrinsic feature points of the second user facial image and the unique feature points The feature points of the user's face photographed in the second direction), and if the average value of the third difference value and the fourth difference value is less than a preset reference value, The user photographed by the second camera 301 or 302 may be recognized as an authenticated user.

If the average value of the first to fourth difference values is less than a preset reference value, the control unit 110 may recognize that the user photographed by the first and second cameras 301 and 302 is an authenticated user have. That is, the controller 110 compares the detected common and non-common feature points with a plurality of user feature points stored in advance, and recognizes the user as an authenticated user based on the comparison result.

The controller 110 may calculate the similarity K between the unique feature points of the first and second face images and the previously stored unique feature points using Equation (1).

Here, d _i represents a distance value between unique feature points of the first or second user face image, and e _i D _i represents a distance value between pre-stored unique feature points of the first or second user face image, E _i Quot; represents a distance value between pre-stored unique feature points of the second or first user face image. a _i and b _j are constant values corresponding to three-dimensional and two-dimensional, respectively, and can be changed according to the designer's intention.

The controller 110 obtains the difference between the distances d _i and e _i between the intrinsic feature points and the distance D _i and E _i between the intrinsic feature points previously stored, The similarity degree K may be obtained and the user photographed by the first and second cameras 301 and 302 may be recognized as an authenticated user only when the similarity degree K is higher than a predefined threshold value .

Therefore, the user recognition apparatus and method according to embodiments of the present invention can acquire one face of a face at least irrespective of the direction in which the user faces, without taking a straight posture, User authentication can be performed.

Hereinafter, another user recognizing apparatus and method for accurately recognizing a user's face even if the face angle of the user changes using one camera (or a plurality of cameras) installed in a cluster of vehicles or a dashboard will be described. For example, when a driver is operating a vehicle, free face pose information of a driver is monitored to automatically register and delete face images at various angles in a user face authentication database (DB) Describes how to update the database.

5 is an exemplary view showing an installation position of a camera according to embodiments of the present invention.

5, the camera 501 may be fixedly installed on a ceiling of a vehicle or a side surface of a room mirror, or may be fixedly installed on a lower portion of a cluster, a dashboard, or a wind shield of a vehicle .

6 is a flowchart illustrating a method of recognizing a user according to an embodiment of the present invention.

First, the controller 110 controls the camera (not shown) based on a driver detection signal generated from a driver's surveillance sensor mounted on the driver's seat when a driver's unlock signal is received through the communication unit 120, 501) can be controlled (for example, turned on).

The control unit 110 photographs the face of the driver through the camera 501, and compares the photographed face image with the face image registered in advance (S11). For example, the control unit 110 photographs the driver's face through the camera 501, compares the photographed face image with a previously registered face image, and compares the photographed face image with the registered face image in advance And determines whether the facial images match each other.

The controller 110 detects whether the user who is seated in the driver's seat of the vehicle is a valid driver based on the face verification result (S12). For example, when the photographed face image and the previously registered face image are matched with each other, the control unit 110 determines that the user who is seated in the driver's seat of the vehicle is a valid driver, If the images do not match each other, it is determined that the user who is seated in the driver's seat of the vehicle is an invalid driver.

The control unit 110 may register the photographed face image as a new face image if the photographed face image and the previously registered face image do not match with each other, and will be described with reference to FIGS. 7 and 8. FIG.

7 is a diagram illustrating an example of a method of registering an operator's face image according to an exemplary embodiment of the present invention.

7, the control unit 110 controls the camera 501 to receive the center image f, the central bottom face image b, the center lower face image h, the upper left face image a, , A left center face image (d), a left lower face image (g), a right upper face image (c), a right center face image (f), and a lower right face image (i) Can be registered in the memory 190 in advance.

The control unit 110 registers the face of the driver in the memory 190 when the driver boarded the vehicle when no image registered in the memory 190 exists at all. For example, when the face image registered in the specific storage area 7-1 of the memory 190 is not present at all, the controller 110 registers the face of the driver in the memory 190 The face registration guide message (for example, see front) is displayed on the display units 171 and 195, the face is photographed in accordance with the face registration guide message, and the photographed face image 7-2 (Stored) in the memory 190 together with identification information (e.g., unique name, number, etc. corresponding to the photographed face).

On the other hand, if the photographed face image (for example, the center face image) and the registered face image (for example, the registered central face image in advance) do not match each other, May be registered as a new face image.

8 is a diagram illustrating another method of registering an operator's face image according to an embodiment of the present invention.

8, the control unit 110 controls the camera 501 to receive the center image f, the central bottom face image b, the center lower face image h, the upper left face image a, , A left center face image (d), a left lower face image (g), a right upper face image (c), a right center face image (f), and a lower right face image (i) Can be registered in the memory 190 in advance.

If the face image of the specific direction (for example, the center) photographed through the camera 501 matches the image already registered in the specific storage area 8-1 of the memory 190, And retains the image already registered in the storage area 8-1.

If the face image of the specific direction (for example, the center) photographed through the camera 501 does not match the image already registered in the specific storage area 8-1 of the memory 190, It is assumed that the image already registered in the specific storage area 8-1 of the camera 190 is an error image and the face of the specific direction (e.g., center) photographed through the camera 501 You can also register images.

The control unit 110 determines whether or not the face image of the specific direction (for example, the center) photographed through the camera 501 matches the image already registered in the specific storage area 8-1 of the memory 190 A display unit 171 or 195 displays an inquiry message for inquiring whether to register a face image of a specific direction (for example, center) photographed through the camera 501 in place of the error image when the vehicle is stopped. (For example, in the middle) photographed through the camera 501 instead of the error image when a user input requesting registration of the face image in place of the error image is received May be registered.

The control unit 110 determines whether or not the face image of the specific direction (for example, the center) photographed through the camera 501 matches the image already registered in the specific storage area 8-1 of the memory 190 It is determined whether or not to register the face image of a specific direction (for example, center) photographed by the camera 501 in place of the error image when the vehicle stops for a predetermined time (for example, 10 seconds) When a user input requesting registration of the face image in place of the error image is received, the inquiry message inquiring is displayed on the display units 171 and 195, It is possible to register a face image in a specific direction (e.g., center).

The control unit 110 controls the posture of the driver's seat of the vehicle, the height of the steering wheel, the side mirror, the room mirror, Mirror) can be automatically controlled. For example, the control unit 110 reads out predetermined seat posture information from the memory 190 by a recognized user (valid driver), and based on the read seat posture information, And controls a driving unit for controlling the posture of the seat. The control unit 110 reads the side mirror posture information previously set by the recognized user from the memory 190 and determines the posture of the side mirror based on the read side mirror posture information And controls a driving unit for controlling the driving unit.

The control unit 110 recognizes the user (e.g., a driver) based on the valid face image, and stores contents (e.g., photographs, moving pictures, navigation information, preference channels, etc.) And may be provided through an in-vehicle infotainment system or received via a communication network.

The controller 110 detects positions (two-dimensional coordinates) of eyes, nose, mouth, etc. as basic feature points of the face from the valid face image. The control unit 110 can discriminate whether or not the image photographed by the camera 501 is a face image based on the basic feature points of the face image.

If the image photographed by the camera 501 is a valid face image, the control unit 110 detects intrinsic feature points (s) of the face in the face image. The characteristic feature point (s) of the face may be a specific part of the face such as the starting position of the eye, the large position of the eye, the eyebrow position, the eyebrow shape, the lip, the jaw line, the eye, the whisker, , Which can be detected using a known feature detector or the like.

If the photographed face image and the previously registered face image are matched with each other, the controller 110 determines that the user who is seated in the driver's seat of the vehicle is a valid driver, and then determines whether the vehicle is running (S13).

If the vehicle is running, the control unit 110 shoots a face image corresponding to different face angles through the camera 501 (S14), and stores the captured face images in the memory (storage unit) 190 (Stored) (S15). For example, the control unit 110 may monitor the face angle of the user in the vehicle in real time through the camera 501 when the vehicle is running, and when the monitored face angle is within the reference allowable range, Captures a first image of a face located within an allowable range, and automatically registers the captured first image in the memory (storage unit) 190.

The control unit 110 photographs a face image corresponding to different face angles through the camera 501 when the vehicle is stationary, and automatically registers the photographed face images in the memory (storage unit) 190 (Stored). For example, when the vehicle is stationary, the control unit 110 monitors the face angle of the user in the vehicle through the camera 501 in real time, and when the monitored face angle is within the reference allowable range, The first image of the face located within the allowable range may be photographed, and the photographed first image may be automatically registered in the memory (storage unit) 190. [

The reference tolerance range may be a plurality of different reference tolerances corresponding to different face angles.

The control unit 110 may temporarily register the photographed first image in the memory 190. FIG.

The control unit 110 temporarily registers the photographed first image in the memory 190, and then, when the monitored face angle is matched with the first angle within the reference allowable range, The second image is captured, and the captured second image is registered (stored) in the memory 190 instead of the first image.

9 is a diagram illustrating another method of registering an operator's face image according to an exemplary embodiment of the present invention.

9, when the user who is seated in the driver's seat of the vehicle is an effective driver and the vehicle is running, the control unit 110 photographs face images corresponding to different face angles through the camera 501 , And automatically registers the photographed face images in the memory 190.

For example, when it is assumed that only the central image (e) of the driver's face is registered in the memory 190, the control unit 110 determines whether the user sitting on the driver's seat of the vehicle is a valid driver, b), a center-down face image (h), a left-side face image (a), a left-center face image (d), a left- , The right lower face image (i), and the like, and registers the captured face images in the memory 190 automatically.

The controller 110 monitors the face angle of the user in the vehicle in real time through the camera 501 when the vehicle is running, and if the monitored face angle is in the preset upper left face image (a) 9-1 (9-2) when the reference angular range (e.g., the face angle of the upper left face image (head angle) is within 30 degrees to 60 degrees) And automatically registers the photographed first image (9-5) in the memory (storage unit) 190. The first image (9-5)

The control unit 110 monitors the face angle of the user in the vehicle in real time through the camera 501 when the vehicle is running, and if the monitored face angle is within the reference allowable range of the preset lower right face image (i) (For example, the face angle of the lower right face image (head angle) is 30 degrees when the face angle of the lower right face image (head angle) is within 30 degrees to 60 degrees) To 60 degrees), and automatically registers the captured second image in the memory (storage unit)

The control unit 110 photographs a face image corresponding to different face angles through the camera 501 when the vehicle is stationary, and automatically registers the photographed face images in the memory (storage unit) 190 (Stored). For example, when the vehicle is stationary, the control unit 110 monitors the face angle of the user in the vehicle through the camera 501 in real time, and when the monitored face angle is within the reference allowable range, The first image of the face located within the allowable range may be photographed, and the photographed first image may be automatically registered in the memory (storage unit) 190. [

The reference tolerance range may be a plurality of different reference tolerances corresponding to different face angles.

The control unit 110 may temporarily register the photographed first image in the memory 190. FIG.

The control unit 110 temporarily registers the photographed first image 9-5 in the memory 190 and then the monitored face angle is within the reference tolerance range (for example, 30 degrees to 60 degrees) A second image of a face matched to the first angle when matched with a first angle (first fixed angle) (e.g., 45 degrees), and to capture the second captured image instead of the first image (Stored) in the memory 190.

FIG. 10 is an exemplary view showing another method of registering an operator's face image according to an embodiment of the present invention.

As shown in FIG. 10, the controller 110 monitors the face angle of the driver in real time even after temporarily registering the photographed first image 9-5 in the memory 190, When the angle 9-3 is matched with the first angle (the first fixed angle) (for example, 45 degrees) 9-4 within the reference tolerance range (for example, 30 degrees to 60 degrees) The second image 10-1 of the face matched with the first angle 9-4 is photographed and the photographed second image 10-1 is photographed in the memory 10-1 instead of the first image 9-5. (Stored) in the memory 190 as a final image. At this time, the final image is registered, and then the memory 190 is controlled so that no further updating of the face image is possible.

As described above, the user recognition apparatus and method according to embodiments of the present invention recognize a user face based on user face images photographed in different directions, so that even if the user's face angle changes, Can be accurately recognized.

The user recognition apparatus and method according to embodiments of the present invention can accurately recognize a user even if the user does not take a straight posture based on common and / or non-common unique feature points in a face image, (Position) of the driver's seat of the vehicle, the height of the handle, the side mirror, the room mirror, etc., based on the vehicle information previously set by the driver have.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (10)

A user recognition apparatus mounted on a vehicle, the user recognition apparatus comprising:
A camera;
A memory;
A display unit;
Monitoring a face angle of a user in the vehicle through the camera when the vehicle is running and photographing an image of a face located within the reference tolerance when the monitored face angle is within a reference tolerance range, And a controller for automatically registering the image in the memory,
Wherein,
When only some of the face images corresponding to different face angles to be registered in the memory are registered in the memory, face images corresponding to different face angles are photographed when the vehicle is running , Automatically registers the photographed face images in the memory,
When the first image out of the face images photographed by the camera does not match the image already registered in the specific storage area of the memory and the vehicle stops for a preset time after the non-matching point, A message inquiring whether to register an image is displayed on the display unit,
And registers the first image in the memory instead of an image already registered in a specific storage area of the memory when an input requesting to register the first image is received. delete The apparatus of claim 1,
When each of the different facial angles of the photographed face images is located within each of the reference permissible ranges corresponding to different facial angles, images of the respective faces located within the respective reference permissible ranges are photographed, And automatically registers it in the memory. The apparatus according to claim 1,
Wherein the predetermined range is a plurality of different reference allowable ranges corresponding to different face angles. The apparatus of claim 1,
And temporarily registers the photographed image in the memory. 6. The apparatus of claim 5,
Temporarily registering the photographed image in the memory and then photographing a second image of the face matched to the first angle when the monitored face angle matches a first angle within the reference tolerance, And registers the photographed second image in the memory instead of the second image. 7. The method of claim 6,
Is a fixed angle belonging to the reference allowable range. delete delete A user recognition method of a user recognition device mounted on a vehicle,
Monitoring a face angle of a user in the vehicle through the camera when the vehicle is running;
Capturing an image of a face located within the reference tolerance when the monitored face angle is within a reference tolerance;
Automatically registering the photographed image in a memory;
When only some of the face images corresponding to different face angles to be registered in the memory are registered in the memory, face images corresponding to different face angles are photographed when the vehicle is running Automatically registering the photographed face images in the memory;
When the first image out of the face images photographed by the camera does not match the image already registered in the specific storage area of the memory and the vehicle stops for a preset time after the non-matching point, Displaying a message on a display unit to inquire whether to register an image;
Registering the first image in the memory instead of an image already registered in a specific storage area of the memory when an input requesting to register the first image is received.

Images