WO2011030958A1

WO2011030958A1 - A system and method for automatic adjustment of vehicle mirrors

Info

Publication number: WO2011030958A1
Application number: PCT/KR2009/006165
Authority: WO
Inventors: Yoon Seok Cho; Sang Uk Hong
Original assignee: Yoon Seok Cho; Sang Uk Hong
Priority date: 2009-09-14
Filing date: 2009-10-23
Publication date: 2011-03-17
Also published as: KR101154018B1; KR20110028760A

Abstract

A system and method for automatic adjustment of vehicle mirror orientations. An image-sensing means processes two-dimensional images of a seated driver. By reconstructing these two-dimensional images as three-dimensional images, the system predicts the most probable coordinates of the driver's eyes relevant to each mirror. The system then adjusts each mirror for optimal orientation corresponding to the applicable visual direction extracted from the position of the driver's eyes. Applying a monocular visual direction, in consideration of various neurological features, enables the system and method applicable universally to most drivers. A mode selection apparatus is also furnished for a driver to select a preferred exterior rearview mirror setting.

Description

A SYSTEM AND METHOD FOR AUTOMATIC ADJUSTMENT OF VEHICLE MIRRORS

The present invention relates to a system and method for automatic adjustment of vehicle mirror orientations. More particularly, in such a system and method, the motions of a driver’s body parts are predicted to extract the position of the driver’s irises and/or pupils (referred as eyes hereinafter) relevant to each mirror. And, in consideration of both anatomical and neurological features, a monocular visual direction which is almost free from individual variables is determined for each mirror so that the disclosed system and method applying the said monocular visual directions are universally applicable with few restrictions to most drivers.

It is important for a safe driving to adjust and maintain properly the vehicle mirror orientations. However, many drivers are driving with improperly adjusted mirrors which often put them in dangerous situations. Such improper mirror adjustment would be mainly due to the driver’s carelessness or negligence of safety but, in fact, the inconvenience of separate and manual adjustment generally offered by the conventional methods would also considerably be responsible for such behaviors of the drivers. As a solution to this problem, various automatic mirror adjustment methods have been proposed. The methods can be divided in two major categories, a semi-automatic scheme and a full-automatic scheme. The semi-automatic scheme is a method that a driver adjusts one of the mirrors manually, and then at least one of the remaining mirrors is automatically adjusted. The full-automatic scheme is a method that all three mirrors are automatically adjusted at the same time. Most of those semi-automatic and full-automatic methods adopt various position sensors, image obtaining devices and image processors to locate the driver’s eyes and to compute the corresponding mirror orientations. Owing to the recent development of computer science and the enhancement of accuracy and affordability of optical devices, sensors and the like, various automatic mirror adjustment methods have become realized.

There are two decisive factors that define the accuracy of the automatic mirror adjustment system, locating the diver’s eyes sighting each mirror and determining the applicable visual direction for each mirror that relevantly decides the orientation of each mirror. Hereinafter, the solutions generally employed in the prior arts to cope with the above mentioned two factors are discussed.

First, the method for determining the positions of a driver’s eyes sighting each mirror is discussed. The vehicle mirror orientation is determined in correspondence with a specific visual direction lying on a specific sightline which starts the driver’s eyes at a specific position and fixates on a target point somewhere behind the driver via a specific point of the mirror surface. Nevertheless the position of the driver’s eyes differ by each mirror, most of the prior arts apply one common eye position for all the mirrors in different locations. For example, Japanese Patent Registration No. 4026471 discloses a method locating the driver’s eyes using a camera and an infrared illumination and one common eye position is applied to the adjustment of all three mirrors, however, no evidence supporting such method is found. Korean Patent Registration No. 10-0792934 and the Korean Patent Registration No. 10-0512855 suggest the same questionable concept as one common eye position is applied for all three mirrors. Japanese Patent Laid-Open Publication No. 2007-045217 discloses a method in which each mirror’s positioning data relevant to the one common eye position extracted from an image photographed by a camera is adopted, however, neither of the data nor the method building such data is disclosed. U.S. Patent No. 5,694,259 proposes an apparatus for automating the adjustment of rearview mirrors in which the angle of a driver’s head when the driver gazes at each mirror is calculated by using information of one common position of the driver’s eyes primarily extracted and an analysis program for the pivot movement of the driver’s head stored in a microprocessor. However, the above U.S. Patent entails a potential error as it is not sufficient to predict the positions of the driver’s eyes reasonably only by using the pivot movement of the driver’s head but without taking the movement of other body parts such as the rotational movement of the eyeballs into consideration.

Secondly, the method generally adopted in the prior arts for determining the applicable visual direction corresponding to each mirror’s orientation is discussed hereinafter.

When the positions of a driver’s eyes viewing each mirror are located, determining the applicable visual direction for each mirror will be the next critical step as a mirror orientation is actually corresponding to the applicable visual direction. The visual direction generally applied in the prior arts is based on a binocular vision. More to explain, it is an imaginary central visual direction which is located geometrically at the midway between left and right visual directions. For example, the above mentioned U.S. Patent No. 5,694,259 adopts the cyclopean eye as an imaginary central eye in which both eyes are integrated into a single eye and premises that the visual direction relevant to the determination of the mirror orientation is formed begining from the cyclopean eye so as to demonstrate that the said imaginary visual direction is a geometrically derived central direction. Japanese Patent Laid-Open Publication No. 2007-045217, Korean Patent Registration Nos. 10-0512855 and 10-0792934, and U.S. Patent No. 6,840,637 also suggest basically same concept of the imaginary central eye and the central visual direction as adopted in the above U.S. Patent No. 5,694,259.

As described above, in most of the prior arts, potential errors are involved in both of the critical parameters, locating the driver’s eyes and determining the applicable visual directions.

As the position of each mirror is stationary while the position of the driver’s eyes gazing at each mirror varies by his position seated and his posture, locating the positions of the driver’s eyes at a specific position and posture will be the first task to be accomplished by taking anatomic features of body parts like the head, neck and eyeballs into account.

The second task will be determining the applicable visual direction relevant to each mirror. Since a human’s two eyes are separated by about 55-70mm, the images received by the two eyes differ from each other and the visual directions as well, which is a binocular vision involving intricate neurological features. It explains that there can be more than one potential visual direction that can be applied to the adjustment of a mirror orientation. Actually, as will be discussed later, the location of a binocular visual direction varies by person and by situation.

This tells that the method of applying a binocular visual direction as generally adopted in the prior arts would be impractical because of lack of consideration of the anatomic and neurological features.

The present invention discloses a system and method for an automatic adjustment of vehicle mirror orientations. To eliminate the potential problems might be involved in the prior arts, a process predicting the motion of the driver’s body parts during gaze shifts is employed by taking the anatomic features into account. To locate the positions of the driver’s eyes relevant to each mirror and a monocular visual direction which is simple and has few individual variables is applied in consideration of the neurological features. Consequently, enhanced accuracy in mirror orientation adjustment can be achieved.

To accomplish the above subjects, as an exemplary embodiment of the present invention, a system and method for automatic adjustment of mirror orientations are disclosed. The present invention includes: acquiring an image or images of a driver; extracting the positions of the driver’s eyes relevant to each mirror using the acquired image(s) of the driver; determining the applicable visual direction(s) corresponding to the orientation of each mirror by extracting at least one of left or right monocular visual direction relevant to each mirror from the extracted position of the driver’s eyes; and adjusting the orientation of each mirror in accordance with the calculated setting angle.

Acquiring the image(s) of the driver is triggered by at least one of the predefined trigger signals detected by a trigger signal detecting means. The predefined trigger signals include such events that electric power is engaged in primary part of an ignition circuit, a change is detected in the driver’s seat position, a change is detected in a mirror orientation and the driver’s command to start the procedure is received. Upon receiving a trigger signal, the image acquiring means takes two-dimensional (2D) image(s) of the driver.

Extracting the position of the driver’s eyes relevant to each mirror comprises of: digitalizing the acquired 2D image(s) of the driver using at least one processing technology such as Gaussian process, Bayesian method, Particle filter, Kalman filtering and Adaboost; reconstructing the digitalized 2D image of the driver into three-dimensional (3D) image using at least one processing technology such as SCAPE (Shape Completion and Animation of People). Segmentation of the skeletal structure such as vertebrae, scapulae, cervical vertebrae and cranium and the driver’s eyes may be accommodated so as to predict the driver’s motion during gaze shifts between the mirrors and thereby to predict the position of the driver’s eyes relevant to each mirror.

Determining the applicable visual direction(s) corresponding to the orientation of the interior rearview mirror, rectangular shaped, includes: extracting the right monocular visual direction, at a head position binocularly gazing rearward via the midpoint of left vertical edge of the mirror, that fixates on the left reference point, an imaginary point which is on the left side of the driver and on or out of the rear windshield, and/or extracting the left monocular visual direction, at another head position binocularly gazing rearward via the midpoint of the right vertical edge of the mirror, that fixates on the right reference point, an imaginary point which is on the right side of the driver and on or out of the rear windshield. Both of the above left and right monocular visual directions simultaneously or either of them solely can be used as the applicable visual direction determining orientation of the interior rearview mirror. An imaginary common visual direction that begins at an imaginary common eye formed at the crossing point of the above two monocular visual directions can also be an applicable visual direction. In this application, the horizontal orientation of the interior rearview mirror when an angle formed by the imaginary common visual direction connecting the imaginary common eye and the center point of the rear windshield via the center point of the mirror is bisected by a line perpendicular to the mirror surface at the center point of the mirror can be the desired horizontal setting orientation of the mirror.

Based on the same concept, when the two vertical spaces in the projected image of the rear windshield and its vicinities, a space between the image of the upper reference point and the upper horizontal edge of the mirror and another space between the image of the lower reference point and the lower horizontal edge of the mirror are seen equal in the driver’s view, it can be regarded as the vertical orientation of the mirror is well adjusted at the desirable setting orientation.

Determining the applicable visual directions for exterior rearview mirrors mounted on the driver side and passenger side of a vehicle includes: extracting the right monocular visual direction, as the applicable visual direction for the passenger side exterior rearview mirror, when the driver binocularly gazes at the right reference point via the midpoint of the left edge of the passenger side exterior rearview mirror; extracting the left monocular visual direction, as the applicable visual direction for the driver side exterior rearview mirror, when the driver binocularly gazes at the left reference point via the midpoint of the right edge of the driver side exterior rearview mirror.

Determining the orientation of each exterior rearview mirror includes: determining the horizontal orientation of the mirror by determining the desirable x- and y- coordinates of the relevant reference point on which each monocular visual direction fixates via the relevant vertical edge of the relevant exterior rearview mirror; determining the vertical orientation of the mirror by determining the desirable z- coordinates of the relevant reference point on which each monocular visual direction fixates via the midpoint of the relevant horizontal edge of the relevant exterior rearview mirror.

To accomplish the above mentioned tasks, the present invention may be comprised of an image acquiring unit acquiring a photo image or images of a driver, a system main controller for controlling the entire system and a mirror orientation adjusting unit. The invention includes a process for the extraction of the monocular visual directions relevant to the determination of each mirror’s orientation; a process for the calculation of each mirror orientations; and a mirror orientation adjusting unit for adjusting the orientation of the mirror in accordance with the calculated orientation of each mirror.

The system main controller includes: an image receiving unit that receives the image(s) taken by the image acquiring devices, an image processing unit for extracting the position of the driver’s body parts from the received image(s) of the driver, predicting the motion of the driver’s body parts to extract the positioned driver’s eyes relevant to each mirror and determining the visual directions corresponding to each mirror orientation; a mirror orientation calculating unit calculating the corresponding orientation of each mirror to the relevantly and respectively determined monocular visual direction; and a database storing data to support the above mentioned processes and the calculation of the mirror orientations.

The image processing unit first analyzes the acquired 2D image(s) to extract the photographed position of the driver’s body parts such as torso, head, face, irises and pupils, and from there, extracts the coordinates of the finally positioned driver’s eyes relevant to each mirror by reconstructing the 2D image(s) into 3D image(s) to predict the motion of the body parts related to the driver’s gaze shifts from the photographed position to the position gazing a specific point of each mirror. In the motion prediction, various data related to the mechanisms of the motion combination of the driver’s body parts during a gaze shift can be utilized and an artificial intelligent processor such as a neural network may be applied for the analysis and processing of the 2D and 3D images.

The mirror orientation calculating unit determines each mirror orientation by calculating an angle formed by the applicable monocular visual direction line at the midpoint of the relevant edge of each relevant mirror.

The present invention further includes: information means for performing the communication between the driver and the system main controller; and a trigger signal detecting unit for detecting predefined trigger signals such as an event of electric power application to primary part of an ignition circuit, an event of change and generating a trigger signal when electric power is applied, when the position of a driver seat is changed, or when the orientation of the mirror is changed, and driving the system main controller.

As described above, the present invention has the following advantageous effects.

A three-dimensional (3D) reconstruction technology is utilized in extracting the coordinates of the finally positioned driver’s eyes so as to increase the accuracy of the position extraction, and the driver’s monocular visual direction is applied to the adjustment of the mirror orientation so that it is possible to suggest a mirror orientation calculating method which can be commonly applied to the unspecific majority and allows the present invention to minimize a possibility of an error in the adjustment of the mirror orientation. Together with the above mentioned solutions, application of a exterior rearview mirror setting mode selection enhances the robustness of the commodity to correspond with the users’ satisfaction.

In addition, the present invention is of benefit not only to the main driver of a vehicle but to anyone dives the same car by furnishing a fully automatic mirror adjustment system. Moreover, the present invention can be commonly applied to all types of semi-automatic and fully-automatic mirror adjustment schemes

Further detailed understanding about the present invention can be acquired from the following description in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a systematic structure of the automatic mirror adjustment system, according to a preferable embodiment of the present invention;

FIG. 2 is a block diagram illustrating a schematic structure of the system main controller, according to a preferable embodiment of the present invention;

FIG. 3 is a conceptual plan view illustrating a state that a driver’s visual direction lines fixate on the reference points on a rear windshield and its vicinity, according to a preferable embodiment of the present invention;

FIG. 4 is a conceptual front view illustrating a state, in conjunction with FIG. 3, that the rear windshield is projected onto the interior rearview mirror and the reflected image is seen by the driver, according to a preferable embodiment of the present invention;

FIG. 5 is a conceptual plan view illustrating a state that a driver’s right monocular visual direction line fixates on a rear part of a vehicle via a passenger side exterior rearview mirror, according to a preferable embodiment of the present invention;

FIG. 6 is a conceptual front view illustrating a state, in conjunction with FIG. 5, that the rear part of the vehicle is projected onto the passenger side exterior rearview mirror and the reflected image is seen by the driver, according to a preferable embodiment of the present invention;

FIG. 7 is a conceptual view illustrating exterior rearview mirror setting modes, as an example, according to a preferable embodiment of the present invention;

FIG. 8 is a flowchart illustrating a schematic flow of the automatic mirror orientation adjustment process, according to a preferable embodiment of the present invention; and

FIG. 9 is a view illustrating a user manipulation apparatus, for example, according to a preferable embodiment of the present invention.

10: interior rearview mirror 20: exterior rearview mirror

100: image acquiring unit 110: image acquiring means

120: illumination means 130: distance sensing means

200: system main controller 210: image receiving unit

220: image processing unit

230: mirror orientation calculating unit

240: database 300: mirror orientation adjusting unit

310: mirror position detecting means 320: drive motors

400: information means

500: trigger signal detecting unit

600: user manipulation unit

The preferred embodiments of the present invention are described hereinafter. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the drawings, like reference numerals refer to like elements throughout.

FIG. 1 is a block diagram illustrating a systematic structure of the automatic mirror adjustment system according to a preferable embodiment of the present invention, and FIG. 2 is a block diagram illustrating a schematic structure of the system main controller according to a preferable embodiment of the present invention.

Referring to FIGs. 1 and 2, a system for automatic adjustment of vehicle mirror orientations according to the preferable embodiments of the present invention includes: an image acquiring unit 100 installed in the inside of a vehicle for photographing an image or images of a driver and acquires the photographed image; a system main controller 200 for controlling the entire system, extracting the positions of the driver’s irises or pupils relevant to each mirror and applicable monocular visual directions corresponding to the desired orientation of each mirror from the acquired image(s) of the driver, calculating the orientation of each mirror corresponding to the applicable monocular visual direction, and outputting an angle control signal for controlling the mirror orientation using the calculated orientation of each mirror; and a mirror orientation adjusting unit 300 for adjusting the orientation of each mirror in response to the control signal outputted from the system main controller 200.

The image acquiring unit 100 employs an image acquiring means 110 (image sensing devices such as a camera) so as to acquire an image or images of the driver’s body parts such as the torso, head, face, irises and pupils.

The image acquiring unit 100 can adopt various illumination means 120 such as, for example, an infrared illumination device in case it is difficult to obtain a desired image quality due to insufficient ambient light or depending on the requirement of an applied technology. In addition, the image acquiring unit 100 may adopt a separate distance measuring means 130 for extracting the position of the driver’s head or eyes. In this case, the image acquiring unit 100 may include a light intensity measuring means (not shown) for measuring the intensity of ambient light along with the illumination means 120. By doing so, the illumination means 120 can be driven only in the case where the surroundings are dark.

The image acquiring means 110, the illumination means 120 and the distance measuring means 130, all of which belong to the image acquiring unit 100, can be installed at various locations in the interior space of the vehicle as required by the applied technologies.

First, preferably, the image acquiring means 110 of the image acquiring unit 100 is installed at a place where it is efficient to photograph the torso, head, face, irises or pupils of the driver, or to simultaneously photograph two or more of the body parts. The image acquiring means 110 may employ any type of image pick-up devices, having an built-in image sensor such as a CCD, CMOS or the like, including a video camera, digital camera, camcorder, web camera, and so forth. Also, the image acquiring means 110 may employ an image pick-up device for simultaneously photographing multiple images such as a stereo camera and the like, or a camera equipped with a separate zoom lens. The image acquiring means 110 may also employ an ultra-low-light camera or an infrared camera which enables an object to be photographed during day and night without a separate illumination source. Further, the image acquiring means 110 may adopt various filters such as an infrared filter, a complementary color filter and the like. The image acquiring means 110 may start operation upon power application to the primary ignition circuit of the vehicle to take at least one frame of the image of the driver’s body part.

Also, preferably, the illumination means 120 of the image acquiring unit 100 is installed at a place where it is efficient to illuminate at least two of the driver’s body parts like the torso, head, face, irises and pupils. The type, location and number of the illumination means 120 are determined by the circumstantial condition such as lack of ambient light, technical requirement of the technology applied to the image acquisition and processing, or the like. The illumination means 120 may be interlocked with the image acquiring means 110, i.e., a camera and operated continuously or intermittently depending on the image acquiring and processing technology. The illumination means 120 may include all kinds of illumination devices such as an infrared illumination device and the like, and may be built in or separated from the camera.

The distance measuring means 130 of the image acquiring unit 100 can be installed at a place where it is efficient to measure the distance to a specific part of a driver’s body such as the head and a specific part of the face such as eyes. The distance measuring means 130 may employ a distance measuring method using a phase difference of the acquired images, or a distance measuring method using a camera zoom lens. The distance measuring means 130 can employ all kinds of distance and position measuring methods including a method using a time difference between a transmitted wave and a reflected wave, such as an ultrasonic sensor, an infrared sensor, a laser sensor, an LED sensor and the like, a method using a position vector such as a magnetic sensor, etc., and may employ various devices or sensors capable of performing these tasks. The distance measuring means 130 may be built in or separated from the image acquiring means 110. Also, in the method using a time difference between an transmitted wave and a reflected wave, a transmitting unit and a receiving unit used in the measurement of the distance may be constructed in an integral or separate type, and all the applicable schemes may be used in which the transmitting unit and the receiving unit are arranged in a single number or in plural numbers, respectively.

Of course, the distance measuring means 130 is not limited thereto, but said means, i.e., the image acquiring means 110, the illumination means 120 and the distance measuring means 130 may be arranged diversely in kinds, quantity and installation locations, which may be determined in various manners depending on the technology applied to the acquisition and processing of the image.

The image acquiring unit 100 is operated in response to a control signal such as, for example, an image capturing signal or a photographing command signal outputted from the main controller 200. In this case, the image acquiring means 110 and other means are operated according to the predefined driving and photographing procedure so as to capture one or more frames of a driver’s images. At this time, necessary body parts of the driver are photographed continuously or intermittently (i.e., every given period, time or interval) until the desired images are obtained.

The system main controller 200 may employ or the system itself may be built in a microcomputer mounted at a part of a vehicle. Also, a plurality of computer systems may serve as/for a single main controller 200. The system main controller 200 determines a start and an end of the automatic mirror adjustment process according to various types of predefined procedures, and controls the overall process.

As shown in FIG. 2, the system main controller 200, includes an image receiving unit 210, an image processing unit 220, a mirror orientation calculating unit 230 and a database 240.

The image receiving unit 210 receives the image(s) photographed and captured by the image acquiring unit 100 from the image acquiring unit 100. Also, the image receiving unit 210 transmits an image photographing command signal to the image acquiring unit 100 in response to a trigger signal applied to the system main controller 200. Then, the image acquiring unit 100 is operated to capture an image or images of a driver in response to the image photographing command signal transmitted from the image receiving unit 210. Also, the image receiving unit 210 continuously compares the image(s) of the driver photographed and captured by the image acquiring unit 100 with image samples stored in the database 240. As a result of the comparison, when a useful image or images are acquired, the receiving unit 210 transmits a termination signal to the image acquiring unit 100 to terminate the photographing operation so as to stop the operation of the image acquiring unit 100. That is, for example, if an image or images within an error range (preferably ±5%) is acquired through the comparison of the photographed body part(s), the photographed angle, the image quality and the image definition between the sample images and the acquired image, the photographing operation can be terminated.

In addition, the system main controller 200 includes the database 240 which contains a variety of data necessary for a process of predicting a visual direction determining the mirror orientation. The database 200 includes: anatomical data like a human skeleton and muscles, etc.; motion prediction data about movement of a driver’s body parts related to gaze shifts such as the neck, head and eyeballs; and the posture and motion modeling data of a human body according to a race, gender, physical figure, age, etc., constructed through experiments and learnings, and the like. In addition, the database 240 may contain the design and manufacturing data of the vehicle features, including the location of each mirror, associated with the calculation of mirror orientations so that the data can be utilized in the calculation of the mirror orientations.

The image processing unit 220 analyzes and processes the image of the driver acquired by the image acquiring unit 100 by using the information data stored in the database 240 so as to locate the positions of the driver’s irises or pupils for the mirror adjustment so as to extract the visual direction relevant to each mirror. That is, the image processing unit 220 reconstructs a two-dimensional (2D) image received from the image acquiring unit 100 into a three-dimensional (3D) image so as to predict the motion of the driver’s body parts, such as a head, face, irises, pupils and the like, when the driver looks at each mirror. By doing so, the coordinates of the finally positioned driver’s irises or pupils relevant to each mirror orientation can be extracted.

The image processing unit 220 includes an image analysis module 221, and an eye (i.e., irises and/or pupils) position extracting module 222.

The image analysis module 221 converts the acquired image of the driver into a digitalized image. In this process, a digital image processing method can employ all types of image processing technologies including Gaussian process, Bayesian method, Particle filter, Kalman filtering, Adaboost and the like, and the modification and combination of such technologies. The image analysis module 221 may extract the position of a driver’s body parts such as the torso, head, face, irises, pupils and the like from the acquired image(s) of the driver.

The eye position extracting module 222 may predict the driver’s motion during a gaze shift which is shifting an eye gaze from one point to another in order to extract the positions of the driver’s eyes which differs by each mirror.

Generally, the gaze shift is performed by a combination of the movements of at least two body parts of a driver such as the eyeballs and the head. Also, the movement of other body parts such as neck, shoulder, spine, etc., can be related with the gaze shift depending on the relative positions between the driver’s eyes and each mirror. Thus, the eye position extracting module 222 of this embodiment employs at least one each technology of eye gaze recognition and 3D head motion prediction, and researches on the combined motions of the driver’s body parts during the gaze shifts are also taken into account in the analysis so as to predict the movement of the body parts to determine the optimally reasonable positions of the driver’s eyes.

Here, the eye gaze recognition is a method in which the position of a driver’s head, irises, pupils and the like is extracted from the image of the driver photographed by the image acquiring unit 100 and the visual direction relevant to each mirror can be extracted based on the extracted position of the driver’s specific body parts. The eye gaze recognition includes a variety of adaptation and modification methods.

The 3D head motion prediction which is a field of 3D human pose prediction technology enables to reconstruct and predict the motion of the facial parts like eyes and of the body parts related to the head motion such as the head, neck and the shoulder by reconstructing the 2D image(s) of the upper part of the driver’s body, acquired by the image acquiring unit 100, into 3D image(s). Various methods combined with the 3D reconstruction technology such as segmenting the skeletal structure of vertebrae, scapulae, cervical vertebrae, cranium, etc., can be accommodated in the 3D reconstruction process.

Various options for the 3D reconstruction process are available as the technologies have already reached the level precisely embodying facial and body expressions by adapting advanced technologies like modeling the features of human bodies including anatomical traits like skeletal and muscular movements sorted by races, ages, genders and body proportions. The 3D reconstruction technologies are widely developed, especially in the computer graphics and animation industries, to be practically exploited by many industries like computer games, movies, virtual realities, medical imaging and sports science. Conclusively, today’s technologies have already been advanced enough to predict the driver’s simple motion viewing a mirror without major obstacles.

The 3D reconstruction process may include at least one of various motion reconstruction and prediction technologies, such as SCAPE (Shape Completion and Animation of People), which are based on the processing technologies like motion capture, motion mapping and motion mimic. Moreover, the findings of modern neurology and ophthalmology regarding the neurological mechanisms related to the combined motions of the body parts during gaze shifts may be utilized in the analysis to enhance the accuracy even higher. In the image processing, an artificial intelligent processors like a neural network may be exploited.

As, in actual fact, the driver’s motion to view rear side via mirrors are simple and typical, the computational processes can partially be replaced by coefficients developed through experiments to reduce the computation loads.

As described above, in these preferable embodiments, the positions of a driver’s irises or pupils relevant to each mirror are extracted by predicting the motion of the body parts like the face or the eyes, of which the reason is that the positions of the driver’s eyes vary depending on the position and posture of the driver. This method ensures a higher accuracy in determining the applicable visual direction relevant to each mirror than applying a single common visual direction to all the mirrors as adopted in the prior arts.

The eye position extracting module 222 extracts the coordinates of the finally positioned driver’s irises or pupils relevant to each mirror upon receiving the image information from the image acquiring unit 100

The mirror orientation calculating unit 230 calculates respectively each mirror’s orientation, by utilizing the relative position information of the driver’s irises or pupils and each relevant mirror, corresponding to the applicable monocular visual direction by computing the view angle formed by the applicable monocular visual direction line flies from the finally positioned driver’s eye to rearward via a specific point of each mirror to fixate on the reference point.

The mirror orientation calculating unit 230 calculates the visual directions through a reasonable method in consideration of both an anatomical aspect and a neurological aspect.

Following is a general review of the anatomical aspect.

As mentioned above, a gaze shift involves various combined 3D motions of different body parts. For example, a pupil rotates on an obit about the center of an eyeball, the head pivots in every direction at the first cervical vertebra and the second cervical vertebra and the neck makes 3D movements, including bending and twisting, at seventh cervical vertebra. The motions of the driver’s gazing at the mirrors can be performed by various combinations of the different 3D motions and the contribution ratio of each 3D motion in a combination is different by each mirror. Thus, there is little chance that, as proposed by the prior arts, all the binocularly centered visual directions for mirrors are concentrated at the single central eye, i.e., the same x-, y- and z-coordinates.

The neurological aspect is reviewed as follows.

In a binocular vision, different visual directions of two eyes is recognized as a single integrated visual direction by various neurological mechanism to avoid a visual confusion and efforts are made to obtain the most accurate visual information by the visual center of a brain. The binocular single vision was known almost two thousand years ago and a theoretical concept of an imaginary central eye positioned at the geometrical midpoint between the two eyes, so called the cyclopean eye, was established at the end of the 19th century and based on which studies have been made to explain the mechanisms of the binocular single vision. It is supposed that the concept of the geometrical central eye and central visual direction proposed by the prior arts is an early concept of the cyclopean eye theory. The cyclopean eye concept has so far played an important role in the research of the binocular single vision, but the today’s cyclopean eye theory largely differs from that of 100 years ago. Particularly, since various intricate phenomena of a binocular vision has been known, such as the fact that the location of a binocular single visual direction can be same as the location of either of the left or right monocular visual direction or can be any position that leans towards either side in between two monocular visual directions so that the concept of the geometrical central visual direction which does not comply with the known fact described above can be no longer supported. On the other hand, in 1930, W. R. Miles proposed a theory that the location of a binocular single visual direction is determined by a dominant eye, and in 1986, Clare Porac and Stanley Coren proposed a theory that the binocular single visual direction tends to lean towards left or right due to the influence of a sighting dominant eye while demonstrating that there exists a sighting dominant eye playing a key role in the judgment of the direction in a binocular vision. Since then, remarkable scientific progress has been achieved based on the sighting dominant eye theory and the mechanisms of the binocular single vision is being revealed one by one.

The researchers reports that the location of a binocular single visual direction can be same as that of the dominant eye’s visual direction or can be formed somewhere in between a dominant eye’s monocular visual direction and the midway between the two monocular visual directions due to the influence of the dominant eye. If the intensity of the sighting dominant eye is high enough, the binocular single visual direction lies on the dominant eye’s visual direction but if it is low, the binocular single visual direction tends to lean towards the midway between the two monocular visual directions. It has been known that the intensity of the dominant eye differs by person. It also has been known that in the case where the contrast or the size of the image incident to a non-dominant eye is recognized as being stronger or larger depending on the eye gaze angle than that to the dominant eye, the non-dominant eye serves as the dominant eye temporarily so that the binocular single visual direction is also changed from the dominant eye’s to the non-dominant eye’s. Yet many phenomena related with the binocular vision are being found but more efforts need to be strived until all the mechanisms are revealed. Nevertheless, what we can learn from the series of technologies and researches is that the location of the binocular single visual direction is influenced by the sighting dominant eye to vary by person and by the gazing conditions such as gaze direction, focusing distance and the like.

The experiments carried out by the applicants also demonstrate that the location of the binocular single visual direction differs by mirror and by person and the results generally match with the sighting dominant eye theory. The researches and the results of the applicants’ experiments show that the general and common binocular single visual direction which can be generally applied to all the mirrors and the majority of the drivers, like the method proposed by the prior art, is unambiguous in its scientific basis. Also, as the geometrically extracted central visual direction is different from the physiological binocular single visual direction which a driver physically recognizes, the mirror orientation adjusted using the geometrically extracted central visual direction can be recognized as inaccurate in the driver’s view. In addition, since the mechanisms of the binocular single visual direction have not yet been revealed in spite of its various unknown variables, it may be concluded that it is nearly impossible to predict the binocular single visual direction which can be applied commonly to all three mirrors and to the majority of the drivers.

Therefore, in this embodiment, the anatomical and neurological variables described above are taken into consideration. That is, the present invention does not apply the conventional type of the geometrically extracted central visual direction that has lost its theoretical basis and yet of which the mechanisms have not been fully revealed, but applies a monocular visual direction that is simple and hardly has an individual difference to adopt in the calculation of the each mirror orientation, and thereby enables to increase an accuracy of the finally set mirror orientation. The method proposed in the present invention is a method in which a driver binocularly gazes at a relevant reference point in rear side via left or right vertical edge of a mirror, however, while one monocular visual direction contacts the mirror, the other one flies by the mirror without meeting with it because there is no enough space at an edge of the mirror to accommodate both the left and right visual directions.

Therefore, two completely different images are imported into the driver’s two eyes respectively. And, to avoid a visual confusion, the visual center in the brain selects only the image of the eye of which the visual direction contacts the mirror and the image from the other eye is suppressed. In other words, although the driver binocularly sights an object, it is like that he uses a monocular vision which is the same effect that he gazes at the object only with one eye while the other is blocked. This is similar to a phenomenon in which an archery player aims the arrow at a target with both eyes opened but he actually uses only one eye to align a sighting device with the target.

As described, when a driver gazes at the rear side of a vehicle via left or right edge of a mirror, he uses a monocular vision inevitably. It is a natural phenomenon occurs to anyone without an exception as the human’s two eyes are separated from each other. And, unlike the binocular single visual direction, it is free from the variables caused by the neurological mechanisms or the individual differences. Thus, a method of calculating the orientation of each mirror using the monocular visual direction proposed in this embodiment can be commonly applied to all the persons. Moreover, the proposed method of applying monocular visual directions actually conforms to a typical behavior of drivers adjusting the mirror orientations manually so that the mirror orientations adjusted by the disclosed method can be well recognized as properly adjusted.

The mirror orientation calculating unit 230 includes an interior rearview mirror orientation calculating module 231 and a exterior rearview mirror orientation calculating module 232.

The method of calculating the orientation of the interior rearview mirror is as follows.

FIG. 3 is a conceptual plan view illustrating a state that a driver’s visual direction lines fixate on the reference points on a rear windshield and its vicinity, according to a preferable embodiment of the present invention and FIG. 4 is a conceptual front view illustrating a state, in conjunction with FIG. 3, that the rear windshield is projected onto the interior rearview mirror and the reflected image is seen by the driver, according to a preferable embodiment of the present invention.

In FIG. 3, H1 and H2 denote a driver’s heads viewing different points on the interior rearview mirror 10. The first and second head positions H1 and H2 are to show the maximum rotation angle to view the whole horizontal mirror length. The first head position H1 represents the position of the driver’s head when the driver’s right visual direction line falls on the left edge of the interior rearview mirror 10. Likewise, the second head position H2 represents the position of the driver’s head when the driver’s left visual direction line falls on the right edge of the interior rearview mirror 10.

BP1 and BP2 denote longitudinal plane bisecting the driver’s heads H1 and H2 respectively. P is the pivot point of the head.

The lines EL1 and ER1 are the left and right eye’s visual direction lines respectively at the head position H1, and the lines EL2 and ER2 are the left and right eye’s visual direction lines respectively at the head position H2.

The line RMC1 is perpendicular to the mirror surface at a horizontal and vertical center point 10-C of the interior rearview mirror 10.

The line RS1 which is indicated by a

line interconnecting

11 and 12 is a rear windshield. 11 and 12 are midpoints of left and right frames of the rear windshield. RS1-C is the horizontal and vertical center point of the rear windshield.

14 and 15 are left and right reference points in the vicinity of the rear windshield and are on the same horizontal line as 11 and 12. 14 and 15 actually represents the maximum horizontal range that can be seen to the driver via the interior rearview mirror and may be positioned either within or beyond the rear windshield.

11 and 12, and, 14 and 15 are respectively symmetrical each other about the center point RS1-C. So, the spaces 11-14 and 12-15 are same in length.

11', 12', 14' and 15' indicated in FIG. 4 correspond to 11, 12, 14 and 15 indicated in FIG. 3, respectively. In FIG. 4, the two midpoints 11' and 12' of the rear windshield and images of the two reference points 14’ and 15’ are on the same horizontal line HL1. Also, the images of the two upper and lower reference points, 16 and 17, and the images of the two midpoints, 18 and 19, of the upper and the lower frames of the rear windshield image RS1’ are all on the same vertical line VL1.

The left and

right reference points

14 and 15 in FIG 3 are the fixation points of the right monocular visual direction ER1 and the left monocular visual direction EL2 are fixated respectively. Thus, the two

reference points

14 and 15 can vary in their positions depending on the positions of the driver’s eyes and the horizontal length of the interior rearview mirror 10.

In FIG. 4, the vertical line VL1 and the horizontal line HL1 lie on the vertical and horizontal center line of the interior rearview mirror 10 respectively as an example, but arbitrary positions on the mirror surface may be selected.

Normally when a driver manually adjusts the horizontal orientation of an interior rearview mirror, the driver adjusts the orientation of the interior rearview mirror to position the image of a rear windshield roughly at the mid of the mirror, and then, makes a further precise adjustment while comparing the images on left and right end of the mirror until the left and right images on the mirror look symmetrical in the driver’s view.

Following the above logic, in this embodiment, it can be the desired horizontal orientation of an interior rearview mirror 10 when the images of the two

reference points

14 and 15 of FIG. 3 are projected onto the left and right edges respectively of the mirror, which are as 14' and 15' of FIG. 4. Then, for the driver, it looks that the mirror orientation is properly adjusted as 14’ and 15’ are symmetrical images.

Thus, as shown in FIG. 3, it is possible to determine the orientation of the interior rearview mirror 10 by determining the angle formed by the visual direction lines ER1 and/or EL2 that fixate on the

reference points

14 and 15 via the left and right edges of the mirror 10 in the head positions H1 and H2 respectively.

Like the above, the driver uses the monocular vision using the visual information of the driver’s one eye, but not the binocular vision using visual information of the driver’s two eyes. That is, the visual direction directed towards the left reference point 14 through the left short edge of the rearview mirror 10 is the first right eye’s visual direction ER1 in the first head position H1, and the visual direction directed towards the right reference point 15 through the right short edge of the rearview mirror 10 is the second left eye’s visual direction EL2 in the second head position H2.

The visual information inputted from the visual directions fly by the interior rearview mirror 10 without contacting, i.e., the left eye’s visual direction EL1 in head position H1 and the right eye’s visual direction ER2 in each head position H2, is suppressed by the visual center in the brain so as to avoid the visual confusion.

A method utilizing an imaginary common eye can also be an option to determine the orientation of the interior rearview mirror 10. The crossing point ICE of the two visual directions ER1 and EL2 can be the imaginary common eye to apply and the line CE1 connecting the imaginary common eye ICE, the center point 10-C of the interior rearview mirror 10 and the center point RS1-C of the rear windshield can be the imaginary common visual direction. Thus, the orientation of the interior rearview mirror 10 when RMC1 bisects the angle formed by CE1 at the center point 10-C can be the desired horizontal orientation of the mirror.

In FIG. 3, it has been illustrated that the head position rotates about the pivot point P to fixate on the

reference points

14 or 15 by changing its position between H1 and H2. However, the rotation angle can differ by person and, in fact, it is very small or even could be zero, any fixed head position in between and including H1 and H2 can be applied. Although it has been described based on the methods using the visual directions ER1, EL2 and CE1, this embodiment is not limited thereto but includes all kinds of methods based on the same concept.

Based on the same logic as above, the vertical orientation of the interior rearview mirror 10 when the vertical length of the upper space which is the length between 16 and 18 in FIG. 4 and that of the lower space which is the length between 17 and 19 in FIG.4 look same in the driver’s view can be the desired vertical orientation of the mirror.

For the calculation of the vertical orientation of the interior rearview mirror 10, either of left or right visual direction can be used as, unlike the case of the horizontal orientation calculation, no significant difference in the images received by the eyes has been found. But it is preferred to use right visual direction as the visual center in the brain strongly tends to prefer the information from the eye closer to the object.

In the above-mentioned embodiment, a noticeable fact is that the two

midpoints

11 and 12 and the two

reference points

14 and 15 in FIG. 3 are symmetrical each other about the center point RS1-C so that the space length between the

points

11 and 14, and that between the

points

12 and 15 are equal. But the space length between 11’ and 14’, the images of 11 and 14, and the space length between 12’ and 15’, the images of 12 and 15, are not equal but the space between 12’ and 15’ is shorter than the space between 11’ and 14’, which means that image of the rear windshield RS1’ is not positioned at the center of the rearview mirror 10, but slightly leans towards the right. This phenomenon is contributed by the visual directions ER1 and EL2 which have different flying distances to the fixation points and different reflection angles at the edges as the driver’s eyes are positioned on the left side of the interior rearview mirror. Although the image of the rear windshield RS1’ is actually not positioned at the center of the interior rearview mirror 10, in the driver’s view, the image of the rear windshield RS1’ is well positioned at the center of the mirror as the images 14’ and 15’ are symmetrical each other. On the other hand, similarly to the mirror orientation calculating method adopted in the prior art, if the image RS1' is positioned exactly at the center of the interior rearview mirror 10, the space length between 11’ and 14’ and that between 12’ and 15’ are equal to each other, but the projected space between 12 and 15 gets longer than that between 11 and 14 to make the images 14’ and 15’ unsymmetrical each other so that it is recognized as leaning towards the left in the driver’s view.

As described, while the prior arts adopt only the geometrical aspect, the present invention accommodates the driver’s actual visual recognition in the analysis, which makes the present invention distinguishable from the conventional methods.

Now, the method for calculating the orientation of a exterior rearview mirror is discussed hereinafter with reference to FIGs 5 and 6.

FIG. 5 is a conceptual plan view illustrating a state that a driver’s right monocular visual direction line fixates on a rear part of a vehicle via a passenger side exterior rearview mirror, according to a preferable embodiment of the present invention and FIG. 6 is a conceptual front view illustrating a state, in conjunction with FIG. 5, that the rear part of the vehicle is projected onto the passenger side exterior rearview mirror and the reflected image is seen by the driver, according to a preferable embodiment of the present invention

The following description is for the case that a driver sights the rearward via a passenger side exterior rearview mirror.

In FIG. 5, the driver’s head HD is rotated towards the right and simultaneously the driver’s eyes are further rotated towards the right to fixate the gaze on a reference point in the rearward, in order to adjust the orientation of the passenger side exterior rearview mirror.

ER10 and EL10 are the right and left visual direction lines respectively in the above-mentioned driver’s posture. Also, BL1 of FIGs. 5 and 6 is a right side boundary surface of a vehicle, i.e., an exterior surface of a vehicle exterior. The further right side from BL1 is the outside of the vehicle. Also, LSC1 is the midpoint of the left vertical edge of the passenger side exterior rearview mirror 20 and LSV1 is a line perpendicular to the mirror surface at the midpoint LSC1. PD1 can be either a reference point at which the right visual direction line ER10 fixates and of which the location can be settled by determining the desired x-, y- and z- coordinates. As shown in FIGs 5 and 6, PD1 may be one point on exterior surface BL1 of the vehicle exterior. The angle (β) is an angle formed by the visual direction line ER10 fixating the reference point PD1 via the midpoint LSC1. BL2 of FIG. 6 is an image of a part of the right exterior surface BL1 of the vehicle exterior, which is projected onto the exterior rearview mirror 20 and is seen by the driver. W is a width of the image seen by the driver.

The horizontal orientation of an exterior rearview mirror can be determined by adjusting the width of the image of the vehicle exterior projected onto the mirror, which is actually what the drivers do when manually adjusting an exterior rearview mirror. Following the same logic as the above and as shown in FIG. 6, in case of a passenger side exterior rearview mirror 20, the horizontal orientation can be determined by determining the width W of the image on the exterior rearview mirror 20. The width W of the image is preferably set to be within 1/4 of the entire horizontal length of the exterior rearview mirror 20 so that the dead zone would not be created excessively.

As shown in FIG. 5, in order to determine the width W, the driver gazes at the reference point PD1 via the vertical midpoint LSC1 of the left edge of the exterior rearview mirror 20 using the right visual direction ER10.

The visual information inputted by the left eye’s visual direction EL10 which flies by the mirror without contacting is suppressed by the visual center in the brain so as to avoid a visual confusion.

Thus, the width W of the image can be determined by determining the desired x- and y- coordinates of the reference point PD1. And the orientation of the exterior rearview mirror 20 when the line LSV1 bisects the angle (β) is the desired horizontal orientation of the mirror.

In FIGs. 5 and 6, the width W of the image is illustrated as being situated on the right side of the left edge of the exterior rearview mirror 20, but may be formed on the left side of the left edge of the exterior rearview mirror 20 depending on the driver’s decision on the mirror orientation. The vertical orientation of the exterior rearview mirror 20 can be determined by determining the z-coordinate of the reference point PD1 using the same principle.

The foregoing is directed to the method of determining the orientation of the passenger side exterior rearview mirror. Same principle can be applied to the method of determining the orientation of the driver side exterior rearview mirror. The horizontal orientation of the driver side exterior rearview mirror can be determined by determining the x- and y- coordinates of the left reference point on which the driver’s left monocular visual direction fixates via the midpoint of the right vertical edge of the driver side exterior rearview mirror and the vertical orientation can be determined by determining the z-coordinate of the left reference point.

FIG. 7 is a conceptual view illustrating exterior rearview mirror setting modes, as an example, according to a preferable embodiment of the present invention.

The exterior rearview mirror orientation calculation module 232 of the mirror orientation calculation unit 230 can be implemented with an exterior rearview mirror setting mode selection function which is, as shown in FIG. 7, that the exterior rearview mirror orientations may be preset in two or more setting modes so that a driver can select a preferred mode.

For example, the width of an image of the vehicle exterior seen by the driver via an exterior rearview mirror, i.e., the width W of the image of the vehicle exterior shown in FIG. 5, can be determined differently depending on the driving habit or preference of individual drivers. Thus, in this embodiment, a setting mode selection function may be provided in the system to enable the driver to select a preferred mode among the two or more setting modes of which the width W of the images are preset differently. This function can contribute to prevent a potential dissatisfaction of users with a uniformed setting mode and enables the driver to easily change the setting mode as preferred even while driving.

The driver can select the setting mode for both driver side and the passenger side exterior rearview mirrors or a different setting mode respectively. Same function for a driver to select a vertical orientation setting mode of exterior rearview mirrors can be provided as well.

In FIG. 7, two conceptual examples of the exterior rearview mirror setting mode are shown. For a convenience of explanation, the setting modes are named as “narrow angle setting” and “wide angle setting”. The narrow angle setting is a setting mode that a driver, while seated to face forward, can see a part of the vehicle exterior via an exterior rearview mirror by slightly rotating his or her head and eyes as can be referred to the illustration of FIG. 6. The wide angle setting is a setting mode that the view range of an exterior rearview mirror is shifted further to outside than that of the narrow angle setting. Namely the BGE setting which is widely used in the North America recently can be an example of the wide angle setting mode. The BGE setting mode is a setting method that a driver should lean towards the mirror to see a part of the exterior of his or her vehicle. The setting modes may be provided in such a manner as to subdivide the setting modes presented in FIG. 7 such as an intermediate setting in between the two setting modes presented and as setting modes of which the view ranges are shifted towards further outside or closer to the vehicle than the presented setting modes. A setting mode selection function having the same concept as the above may be applied to the adjustment of the vertical orientation of the exterior rearview mirrors.

The start and finish of the entire system can be determined automatically by the main controller 200 or manually i.e. in response to a user’s command. The system main controller 200 controls the interfaces and processes of all the events related to the adjustment of the mirror orientations. And the system main controller 200 provides information on the mirror orientation including a variety of warnings to a driver using the information means 400 in real-time. In addition, the system main controller 200 checks the orientation of mirrors, the position of a driver seat and the angle of a backseat in real-time, and stores and manages data. The system main controller 200 transmits and receives signals using a dialogue means, a menu select means, a command input means and the like with respect to the matter needing a driver’s decision so as to communicate with the driver and operate means necessary to perform the driver’s decision.

The information means 400 delivers all the information and warnings related with the orientation of the mirror including the progress situation of a mirror orientation adjustment procedure after/before driving and during driving to the driver using visual and audible means in real-time. That is, the information means 400 may be used as a communication means between the driver and the main controller 200.

At least one information means 400 is installed at a place where the driver identifies and manipulates the information means easily. The information means 400 includes all kinds of visual and audible indicators and signaling means such as light-emitting devices like LEDS, signal sound, voice, character, graphics and the like. In addition, the information means 400 may adopt all sorts of command input means including a keypad, a touch pad, a touch screen, a joystick, a mouse, a speech recognition device and the like, and may be applied in a manner incorporated with other information transfer means such as a dashboard, a navigation panel, a head-up display and the like.

The system main controller 200 may perform its operation using the detection and sensing result of a plurality of external sensing and detecting means. For example, the system main controller 200 starts a procedure defined by a trigger signal. In this case, the trigger signal is a condition or an event granting an initiative to start an automatic mirror orientation adjustment procedure. Thus, when the defined trigger signal is received by the system main controller 200, the automatic mirror orientation adjustment procedure is initiated.

Such a trigger signal is generated by the trigger signal sensing unit 500. The trigger signal sensing unit 500 includes a power application sensing unit 510, a driver seat position detecting means 520, and a mirror orientation change sensing means 530. That is, the trigger signal can be generated in a case where power is applied to the vehicle, in a case where a driver seat is changed, or in a case where the orientation of the mirror is changed. Of course, the trigger signal may be generated by the manipulation of the user.

Here, in the case where the power is applied to the vehicle, a separate power application sensing unit 510 generates a power application signal which may be a trigger signal.

Further, the change of the driver seat means that there occurs a variation in the track position of the driver seat or the angle of a back seat during the driving of a vehicle. A driver seat change signal is generated from a separate driver seat position detecting means 520, and the driver seat change signal may be a trigger signal. The driver seat change information may be used as auxiliary information which the image acquiring unit 100 and the system main controller 200 uses to presume the position of the driver. Also, the driver seat information may be used as auxiliary data for determining whether or not there is an error in the orientation of the mirror calculated by the system main controller 200. In this embodiment, driving refers to an operation performed for a certain period of time from a time point when power is applied to the vehicle to a time point when the power is interrupted. Of course, the driver seat position detecting means 520 generates a driver seat data signal containing information on the track position of the driver seat and the angle of the back seat, and transmits the generated driver seat data signal to the main controller 200. In this case, the driver seat position detecting means include a driver seat track position detecting sensor and a back seat angle detecting sensor, each of which is mounted at a necessary position. Thus, the position information detected by the sensors can be updated to a memory of the system main controller 200 in real-time so as to check a change of the driver seat in real-time. The driver seat data signal may be used as reference data during the adjustment of the orientation of the mirror later.

In addition, the change of the mirror orientation refers to that there occurs a change in the orientation of any one mirror of the respective mirrors during driving. For example, in the case where the orientation of one or more mirrors is manually adjusted or in the case where there occurs a change in the orientation of the mirror irrespective of the intention of the driver due to an external impact, a separate mirror orientation change sensing means 530 generates a mirror orientation change signal, which may be a trigger signal. At this time, a predefined automatic mirror orientation adjustment procedure is performed depending on the cause of the change.

Thus, when the above-mentioned type of the trigger signal is applied to the system main controller 200, the system main controller 200 starts a mirror orientation adjustment operation according to the type of the trigger signal. For example, among the types of the trigger signal, in the case where a driver manually changes the orientation of the mirror in person, the system main controller 200 performs a manual mirror orientation adjustment procedure. Alternatively, in the case where there occurs a change in the orientation of the mirror irrespective of the intention of the driver, the mirror orientation change sensing means senses this orientation change and generates a trigger signal. The system main controller 200 is operated in response to the trigger signal and applies a restoring signal for adjustment of the orientation of the mirror to a mirror orientation adjusting unit 300 of a mirror in which the orientation change occurs so that the orientation of the mirror in which the orientation change occurs can be restored to the original orientation prior to occurrence of the change.

In addition, the system main controller 200 performs a predefined procedure in response to an adjustment command signal. In this case, the adjustment command signal is generated when a driver inputs the adjustment command signal according to the need of the driver during driving. That is, the adjustment command signal is generated in the case where the driver desire to change his or her posture and change the orientation of the mirror to conform to the changed posture during driving, or in the case where the driver desires to change the setting mode of the exterior rearview mirrors during driving. When the adjustment command signal is applied to the main controller 200, a predefined procedure is performed according to the type of the orientation change.

Now, the mirror orientation adjustment process will be briefly described hereinafter.

FIG. 8 is a flowchart illustrating a mirror orientation adjustment process according to a preferable embodiment of the present invention; and

First, the main controller 200 determines whether or not a trigger signal is applied to the main controller 200. If the trigger signal is applied to the main controller 200, the main controller 200 performs the mirror orientation adjustment process.

Subsequently, first, in the mirror orientation adjustment process, the image acquiring unit 100 acquires an image of a driver (S100). The acquired image of the driver is temporarily stored in the database 240 within the main controller 200. Of course, at this time, information on the distance to the driver is stored in the database 240. In addition, the main controller 200 receives current mirror orientation data from the mirror orientation adjusting unit and stores it in the database 240.

Next, the image processing unit 220 predicts the human body’s motion using the photographed image of the driver and extracts the positions of the driver’s eyes by each mirror (S110). That is, the image analysis module 221 of the image processing unit 220 analyzes the image of the driver and converts a 2D analog/digital image into a digitalized image. Also, the eye position extracting module 222 of the image processing unit 220 converts the 2D image into a 3D image, and grasps the motion of the driver’s other body parts such as neck, shoulder, spine, etc., to extract the positions of the driver’s eyes by each mirror depending on the relative position between the positions of the driver’s eyes and each mirror.

Thereafter, the image processing unit 220 extracts visual directions for each mirror using the extracted positions of the driver’s eyes (S120). In this case, the visual directions for each mirror employ monocular visual directions for each mirror. That is, in the case of an exterior rearview mirror, a right monocular visual direction or a left monocular visual direction is used as the applicable visual directions for adjustment of the mirror orientation. Alternatively, in the case of an interior rearview mirror, the mirror orientation calculating unit 230 uses any one of two visual directions or generates a virtual common visual direction of the two visual directions to use as an applicable visual direction for adjustment of each mirror. Here, the interior rearview mirror orientation calculating module 231 generates the applicable visual direction corresponding to the interior rearview mirror, and the exterior rearview mirror orientation calculating module 232 generates an applicable visual direction corresponding to each exterior rearview mirror.

Then, the mirror orientation calculating unit 230 calculates the orientation of each mirror using the extracted visual direction (S130). It is effective to calculate the orientation of each mirror using the monocular visual direction. Also, it is preferable to calculate the orientation of each mirror using the above-mentioned method. Of course, the calculation of the orientation of each mirror is not limited thereto. Alternatively, in order to calculate the orientation of each mirror, the extracted visual direction information and the mirror orientation information data stored in the database 240 are compared with each other, and then the matched orientation may be selected. In this case, the values previously stored in the database 240 through the experiments and researches are effectively used as the orientation information data stored in the database 240. The mirror orientation adjusting unit 300 adjusts the orientation of each mirror depending on the calculated orientation (S140). Before adjustment of the orientation of each mirror, the information means 400 may ask the driver to determine whether or not to automatically adjust the orientation of each mirror.

The mirror orientation adjusting unit 300 operated in response to a control signal from the system main controller 200 to adjust the mirror orientation will be described hereinafter.

The mirror orientation adjusting unit 300 includes at least one mirror position detecting means 310 and at least one drive motor 320 disposed in each mirror, i.e., the interior rearview mirror or the exterior rearview mirrors. The mirror orientation adjusting unit 300 receives a mirror orientation adjustment signal by execution command, manual adjustment or restoring command from the main controller 200. The drive motor is driven in response to the mirror orientation adjustment signal to adjust the orientation of each mirror. Also, the angle data detected by the mirror position detecting means 310 is transmitted to the system main controller 200 and is updated to the memory. In this case, the position of each mirror is detected by the mirror position detecting means 310 in real-time so that it is provided as mirror data to the system main controller 200 in real-time and can be used as basis data upon the determination whether or not there is a change in the mirror orientation or the adjustment of the mirror orientation. The case where there occurs a change in the mirror orientation due to an external impact and the like during driving may be a type of a trigger signal generated in the mirror orientation adjustment procedure. That is, this means that the mirror position detecting means 310 can be used as the mirror orientation change sensing means 530 of the drive sensor 500.

Here, the execution command is a procedure of allowing the driver to determine whether or not to adjust the orientation of each mirror personally. When the driver who receives a request of allowing the driver to determine whether or not to execute the mirror orientation adjustment from the system main controller 200 inputs an execution command, the orientation adjustment procedure is performed. Also, when a predetermined period of time has been elapsed in a state where the driver does not input a cancellation command or an execution command, the orientation adjustment procedure may be cancelled. Like this, as long as the driver inputs a command personally, the adjustment of the orientation of each mirror is performed so that a driver can decide finally whether or not the orientation of the mirror needs to be adjusted, and it is possible to exclude the possibility that the driver will cause confusion in recognition of the surrounding situation due to the change of the mirror orientation while not recognizing it.

In addition, the manual adjustment may be performed in such a manner as to manually adjust the orientation of one or more mirrors according to the need of the driver during driving. That is, the driver can manipulate wired or wireless remote control means such as a remote switch, a joystick or the like personally so as to adjust the orientation of each mirror.

To this end, in this embodiment, the system for automatic adjustment of the mirror orientations includes a user manipulation unit 600. In this case, the user manipulation unit 600 may incorporate a setting mode select unit, a command input unit and a manual adjustment unit into a integrated device.

Such an integrated selection device of the exterior rearview mirror setting mode, i.e., the user manipulation unit 600 is installed at a place where the driver manipulates the user manipulation unit means easily so that the drive can select a setting mode of an exterior rearview mirror depending on the driving habit or preference of the driver.

The user manipulation unit 600 includes all sorts of information transfer methods and means which can transfer a selection of the driver to the controller, including a keypad, a touch pad, a touch screen, a joystick, a mouse, a voice recognition device and the like. This setting mode select means may incorporate or combine a partial or entire function along with an execution command input means, a manual mirror orientation adjustment means and an adjustment command input means.

A command input and cancellation device, i.e., the user manipulation unit 600 is installed at a place where the driver manipulates the device easily. The user manipulation unit 600 may employ all sorts of command input devices including a button, a switch, a touch screen, a voice input device and the like, and may be incorporated with an execution command input means and a adjustment command input means. Moreover, the user manipulation unit 600 may incorporate or combine a partial or entire function along with an exterior rearview mirror setting mode selection means, a manual mirror orientation adjusting means and an adjustment command input means. The command input means may be incorporated with the command cancel means and a separate means may be applied.

FIG. 9 is a view illustrating a user manipulation section according to an embodiment of the present invention.

As shown in FIG.9, the user manipulation unit 600 includes a mirror selection switch 610. When a driver pushes the mirror selection switch 610 in the direction of an arrow L, the driver side exterior rearview mirror is selected. When the driver pushes the mirror selection switch 610 in the direction of an arrow R, a passenger side exterior rearview mirror is selected. In the meantime, when the driver pushes the mirror selection switch 610 in the direction of an arrow C, an interior rearview mirror is selected.

As shown in the drawings, the central position of the three buttons is a neutral position has a default value.

The user manipulation unit 600 includes a direction input button 620. That is, the direction input button can be constructed such that the adjustment direction of the mirror orientation is inputted manually, and the orientation adjustment is performed in four directions, i.e., in the left, right, up, and down directions using four direction input buttons. Of course, the user manipulation unit 600 may adopt a direction input device such as a joystick and the like.

The user manipulation unit 600 includes a setting mode select button 630. The setting mode select button 630 is a button for selecting one of the exterior rearview mirror setting modes provided in the system, i.e., a wide angle setting mode, a medium angle setting mode and a narrow angle setting mode.

The user manipulation unit 600 includes a command input button 640. That is, the command input button 640 is a button for inputting a command so that a determined procedure is performed according to the driver’s intention. In the embodiment of FIG. 9, the command input button 640 is an ‘ENTER’ button.

The selection of the exterior rearview mirror setting mode using the user manipulation unit 600 will be described hereinafter.

First, when a driver selects one of the setting mode selection buttons 630 and presses the ‘ENTER’ button 640 in a state where the mirror select switch 610 is maintained in a neutral position, the angle of a driver side exterior rearview mirror and a passenger side exterior rearview mirror are adjusted by the selected setting mode. In addition, the driver selects the driver side exterior rearview mirror or the passenger side exterior rearview mirror using the mirror selection switch 610 and selects one of the three setting mode selection buttons 630. Then, when the driver presses the ‘ENTER’ button 640, the orientation of the selected mirror is adjusted in accordance with the selected setting mode.

Next, an execution command input function will be described hereinafter.

The system main controller 200 asks the driver to determine whether or not to adjust the orientations of all the mirrors according to a mirror orientation specification provided by the mirror orientation adjusting unit 300 using the information means 400. In this case, when the driver wants, the system main controller 200 instructs the driver to execute the adjustment of the mirror orientation by pressing the ‘ENTER’ button 640. If the drier does not press the ‘ENTER’ button 640 and a predetermined time period is elapsed, the main controller 200 cancels the adjustment of the mirror orientation and terminates the mirror adjustment procedure. When the driver again presses the ‘ENTER’ button 640 during an execution of the command, the command can be cancelled. A separate cancellation button may be applied to the user manipulation unit 600.

Also, the manual mirror orientation adjustment is performed such that the driver selects one mirror using the mirror selection switch 610 and manually adjusts the angle of a mirror selected by using four direction input buttons 620.

Further, in the case where the driver wants to adjust a mirror orientation during driving, when he or she presses the ‘ENTER’ button 640, the mirror orientation adjustment procedure is carried out through the application of the adjustment command. It is possible to selectively adopt a function in which when the mirror orientation of a specific driver is stored and can be repeatedly used, the driver can output and apply the stored mirror orientation data. The mirror orientation adjustment method can be incorporated with a function of setting the position of a driver’s seat or the angle of a seat back by individual.

It has been described in this embodiment that the case exemplified in which a driver’s seat is on the left of the vehicle, one interior rearview mirror is installed in the interior of the vehicle and two exterior rearview mirrors are disposed on each exterior side of the vehicle. The same principle is applied to the case where the driver’s seat is on the right of the vehicle, and the number and arrangement of the mirrors are modified. In addition, what has been described in the present invention, including these items, is merely illustrative embodiments and is not limited thereto. Also, the embodiments of the present invention may include all the methods and means conforming to the main purport and object of the present invention.

Although the preferred embodiments of the present invention have been described in connection with the exemplary embodiments illustrated in the drawings, they are merely illustrative embodiments. It will be appreciated that and various equivalent modifications and variations of the embodiments can be made by a person having an ordinary skill in the art without departing from the spirit and scope of the present invention. Therefore, the true technical scope of the present invention should be defined by the technical spirit of the appended claims.

Claims

A method for automatic adjustment of the mirror orientations for a vehicle, the method comprising the steps of:

acquiring an image or images of a driver;

extracting the coordinates of the driver’s irises/pupils (referred as eyes hereinafter) relevant to each mirror using the acquired image(s) of the driver;

extracting left and right monocular visual directions corresponding to each mirror’s orientation using the extracted positions of the driver’s eyes, and determining at least one of the extracted left and right monocular visual directions as an applicable visual direction;

calculating the orientation of each mirror using the determined applicable visual direction; and

adjusting the orientation of each mirror of the vehicle in accordance with the calculated orientation of each mirror.
The method according to claim 1, wherein the step of acquiring the image(s) of the driver further comprises a step of detecting a trigger signal so that the image acquiring procedure is started when at least one of the predefined trigger signals is detected.
The method according to claim 2, wherein the trigger signal is generated in at least one of the predefined events such as where electric power is applied to the primary part of an ignition circuit, where the driver’s seat is changed in its position and where the driver’s command is inputted.
The method according to claim 1, wherein the step of extracting the coordinates of the driver’s eyes relevant to each mirror further comprises the steps of:

analyzing the acquired two-dimensional (2D) image(s) of the driver using at least one image processing technology such as Gaussian process, Bayesian method, Particle filter, Kalman filtering and Adaboost;

reconstructing the analyzed 2D image(s) of the driver into three-dimensional (3D) image(s) using at least one processing technology such as SCAPE (Shape Completion and Animation of People); and

segmentating the skeletal structure, such as vertebrae, scapulae, cervical vertebrae, cranium, etc., of the driver and at least one of the irises/pupils of the driver’s eyes so as to predict the driver’s motions during gaze shifts thereby to extract the positions of the driver’s eyes relevant to each mirror.
The method according to claim 1, wherein the step of extracting at least one of the left and right monocular visual directions corresponding to each mirror’s orientation and determining at least one of the extracted left and right monocular visual directions as an applicable visual direction to calculate the mirror orientation, comprises the step of:

determining the right monocular visual direction as the applicable visual direction when the driver’s head is in one position to fixate binocularly on the relevant left reference point in the rearward via the left edge of an interior rearview mirror, and/or determining the left monocular visual direction as the applicable visual direction when the driver’s head is in another position to fixate binocularly on the relevant right reference point in the rearward via the right edge of the interior rearview mirror,

wherein the interior rearview mirror is formed in a rectangular shape, disposed on the right side of the driver and has two long upper and lower edges and two short left and right edges.
The method according to claim 5, wherein the crossing point where the right monocular visual direction and the left monocular visual direction cross is determined as an imaginary common eye, and a virtual common visual direction begining from the imaginary common eye is determined as the applicable visual direction.
The method according to claim 6, wherein in the step of calculating the orientation of the interior rearview mirror,

in the case where an angle formed by the virtual common visual direction which connects the imaginary common eye and the horizontal and vertical center point of a rear windshield via the horizontal and vertical center point of the interior rearview mirror can be determined as an applicable visual direction and the horizontal orientation of the interior rearview mirror when a line perpendicular to the mirror surface at the center point of the interior rearview mirror bisects the angle formed by the virtual common visual direction is determined as the desired horizontal orientation of the interior rearview mirror, and

the vertical orientation of the interior rearview mirror when the space between the projected image of the horizontal midpoint of the rear windshield’s upper frame and the horizontal midpoint of the upper edge of the interior rearview mirror and the space between the projected image of the horizontal midpoint of the rear windshield’s lower frame and the horizontal midpoint of the lower edge of the interior rearview mirror are seen equal in the driver’s view is the desired vertical orientation of the interior rearview mirror.
The method according to claim 7, wherein the visual length of the right space between the image of the right reference point and the right edge of the interior rearview mirror is shorter than the visual length of the space between the image of the left reference point and the left edge of the interior rearview mirror.
The method according to claim 1, wherein the step of determining the applicable visual directions for exterior rearview mirrors disposed on the right side and the left side of the driver, comprises of:

using the right monocular visual direction as an applicable visual direction for the exterior rearview mirror disposed on the right side of the driver; and

using the left monocular visual direction as an applicable visual direction for the exterior rearview mirror disposed on the left side of the driver.
The method according to claim 1, wherein the step of calculating the orientation of each exterior rearview mirror, comprises the steps of:

determining the x-, y- and z-coordinates of the relevant reference point on which left monocular visual direction fixate via the vertical midpoint of the right edge of the left exterior rearview mirror so as to determine its desired horizontal and vertical orientations, and

determining the x-, y- and z-coordinates of the relevant reference point on which right monocular visual direction fixate via the vertical midpoint of the left edge of the right exterior rearview mirror so as to determine its desired horizontal and vertical orientations.
A system for automatic adjustment of the vehicle mirror orientations comprises of:

an image acquiring unit for photographing at least one image of a driver and a process to acquire the photographed image(s);

a system main controller for controlling the entire system, extracting at least one applicable monocular visual direction or a virtual common visual direction corresponding to each mirror orientation using the acquired image(s) of the driver, calculating the orientation of each mirror using the extracted monocular visual directions; and

mirror orientation adjustment units for adjusting the orientation of each mirror in accordance with the calculated orientation of each mirror.
The system according to claim 11, wherein the system main controller comprises of:

an image processing unit for extracting the position of the driver’s body parts required by the applied technologies using the acquired image(s) of the driver by predicting the motion of the driver’s body parts to extract the positions of the driver’s eyes relevant to each mirror, and to extract the visual directions fixate on the reference points via the relevant points on each mirror.;

a mirror orientation calculating unit for selecting a left monocular visual direction and/or a right monocular visual direction or a virtual common visual direction as the applicable visual direction respectively to determine each mirror orientation and for calculating the corresponding orientation of each mirror; and

a database for storing data needed to process the image(s) and calculate the orientation of each mirror.
The system according to claim 12, wherein the image processing unit analyzes the acquired two-dimensional (2D) image to extract the position of the driver’s body parts including the upper part of the driver’s body, the driver’s head, irises, pupils, and the like, reconstructs the 2D image(s) of the driver’s body parts into three-dimensional (3D) image(s) to predict the motion of the driver’s body parts, and extracts the positions of the driver’s eyes relevant to each mirror, and

wherein the process to predict the motion of the driver’s body parts utilizes data of mechanisms of a motion combination of the driver’s body parts during gaze shifts, and an artificial intelligence processor such as a neural network is adapted to the analysis and processing of the 2D and 3D images.
The system according to claim 12, wherein the mirror orientation calculating unit calculates the relative angular position between the applicable monocular visual directions and the relevant left and/or right edge of each mirror and determines the desired orientation of each mirror.
The system according to claim 11 further comprises of:

an information means for performing the communication between the driver and the system main controller; and

a trigger signal detecting unit for detecting at least one trigger signal such as an event that electric power is applied to the primary part of the ignition circuit, an event that the position of a driver seat is changed or an event that the orientation of at least one of the mirrors is changed, and for generating a successive signal driving the system main controller.