KR20220013834A - Method and apparatus of image processing - Google Patents

Abstract

According to an embodiment of the present invention, a method and device for image processing are provided to obtain an image frame including the face of a user, detect the face area of the user in an image frame, classify a type of occlusion for the face area according to whether an occlusion exists in at least some of the face area, track the user's eyes based on the type of occlusion, and output information about the face area including the tracked eye position.

Description

Image processing method and apparatus {METHOD AND APPARATUS OF IMAGE PROCESSING}

The following embodiments relate to an image processing method and apparatus.

The camera-based eye tracking technology may be utilized in many fields, for example, a glasses-free 3D super multi-viewpoint display based on viewpoint tracking, and/or a head-up display (HUD). The performance of the camera-based eye tracking technology may depend on the image quality of the camera and/or the eye tracking method. The camera-based eye tracking technology works well for a bare face, but the stability of the operation may be poor when some areas of the face are covered, such as when wearing sunglasses or a hat. As described above, in consideration of the actual driving environment, there is a need for an eye tracking method that can operate stably even when a part of the face is covered.

The above-mentioned background art is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and it cannot necessarily be said to be a known technology disclosed to the general public prior to the present application.

According to an embodiment, an image processing method includes: acquiring an image frame including a user's face; detecting the face region of the user from the image frame; classifying the type of occlusion for the face region according to whether occlusion exists in at least a part of the face region; tracking the user's eyes based on the type of occlusion; and outputting information on the face region including the tracked eye position.

The occlusion type may include a first type corresponding to a bare face in which the occlusion does not exist; a second type in which at least one of a forehead, an eyebrow, and at least a portion of an eye of the user's face area is covered; a third type in which an eye region of the user's face region is covered; and at least one of a fourth type in which at least one of a mouth and a nose among the face regions of the user is covered.

The type of occlusion may include type A corresponding to a bare face in which the occlusion does not exist; and at least one of type B in which an eye region is covered by sunglasses among the face region of the user.

The tracking of the user's eyes may include tracking the user's eyes by aligning different numbers of vertices to at least some shapes of the face region based on the type of occlusion.

The tracking of the user's eyes may include: when the occlusion type is the first type, tracking the user's eyes by aligning a first number of vertices to a partial shape of the face region; and when the type of occlusion is any one of the second to fourth types, tracking the user's eyes by aligning a second number of vertices greater than the first number to the overall shape of the face region. may include

When the type of occlusion is the first type, the tracking of the user's eyes may include: reducing a size of a detection box for detecting the face region to correspond to shapes of eyes and nose among the face regions; cropping the shape of the eyes and nose by the reduced detection box; and tracking the user's eyes by aligning the first number of vertices to the cropped shapes of the eyes and nose.

When the type of occlusion is any one of the second type to the fourth type, the step of tracking the user's eye may include adding the second number to the overall shape of the face area by a detection box for detecting the face area. aligning the vertices of and tracking the user's eye by estimating the position of the user's pupil in the face region by the aligned second number of vertices.

The tracking of the user's eyes may include: re-detecting the face region according to a determination that tracking of the user's eyes by the aligned vertices has failed; and classifying the type of occlusion based on the re-detected face region, and repeatedly performing the steps of tracking the user's eyes.

The classifying the occlusion type classifies the occlusion type for the face region using a pre-trained classifier to distinguish the occlusion region among the face regions by machine learning. may include the step of

According to an embodiment, an image processing method of a Head Up Display (HUD) apparatus includes: acquiring an image frame; determining whether the user wears sunglasses in the image frame; tracking the user's eyes using different types of eye trackers according to the determination of whether to wear the sunglasses; and outputting information on the face region including the tracked eye position.

The step of tracking the user's eyes may include: tracking the user's eyes using a first type of eye tracker in response to a determination that the sunglasses are not worn; and tracking the user's eyes using a second type of eye tracker in accordance with the determination that the sunglasses are worn.

The first type of eye tracker tracks the user's eyes by aligning a first number of vertices to a partial shape of the user's facial region, and the second type of eye tracker is configured with an overall shape of the user's facial region. The eye of the user may be tracked by aligning a second number of vertices greater than the first number.

The step of tracking the user's eyes using the first type of eye tracker may include reducing a size of a detection box for detecting the user's face region to correspond to the shape of eyes and nose among the face regions; cropping the shape of the eyes and nose by the reduced detection box; and tracking the user's eyes by aligning a first number of vertices to the cropped shape of the eyes and nose.

The step of tracking the user's eyes using the second type of eye tracker may include: aligning a second number of vertices to the overall shape of the facial region by a detection box that detects the user's facial region; and tracking the user's eye by estimating the position of the user's pupil in the face region by the aligned second number of vertices.

The determining whether to wear the sunglasses may include determining whether to wear the sunglasses by using a detector previously learned to detect whether the eye region of the user is covered by the sunglasses. According to an embodiment, an image processing apparatus includes: a sensor for acquiring an image frame including a user's face; and detecting the user's face region from the image frame, classifying the type of occlusion with respect to the face region according to whether occlusion exists in at least a portion of the face region, and based on the occlusion type and a processor for tracking the user's eyes and outputting information on the face region including the tracked eye position.

The processor may track the user's eyes by aligning different numbers of vertices to at least some shapes of the face region based on the type of occlusion.

When the type of occlusion is the first type, the processor tracks the user's eyes by aligning a first number of vertices to some shapes of the face region, and the type of occlusion is a second type to a fourth type. In the case of any one of the types, the user's eyes may be tracked by aligning a second number of vertices greater than the first number to the overall shape of the face region.

The image processing apparatus may include at least one of a head up display (HUD) device, a 3D digital information display (DID), a navigation device, a 3D mobile device, a smart phone, a smart TV, and a smart vehicle.

1 is a flowchart illustrating an image processing method according to an exemplary embodiment;
FIG. 2 is a view for explaining a method of classifying a type of occlusion for a face region, according to an exemplary embodiment;
3 is a flowchart illustrating a method of tracking a user's eyes based on a type of occlusion according to an exemplary embodiment;
4 is a diagram for describing a method of aligning different numbers of vertices based on a type of occlusion by an image processing apparatus according to an exemplary embodiment;
5 is a flowchart illustrating a method of tracking a user's eyes according to another embodiment;
6 is a view for explaining an image processing method according to another embodiment;
7 is a flowchart illustrating an image processing method according to another exemplary embodiment;
8 is a block diagram of an image processing apparatus according to an exemplary embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, a description described in one embodiment may be applied to another embodiment, and a detailed description in the overlapping range will be omitted.

1 is a flowchart illustrating an image processing method according to an exemplary embodiment. Referring to FIG. 1 , the image processing apparatus according to an exemplary embodiment may output information on a face region including the position of the user's eyes by performing steps 110 to 150 .

Examples to be described below include tracking the user's eyes using an infrared camera or RGB camera when using an autostereoscopic 3D monitor, an autostereoscopic 3D tablet/smartphone, and a vehicle 3D head-up display (HUD). It can be used to output the coordinates of In addition, the embodiments may be implemented in the form of a software algorithm on a chip in a monitor, in the form of an app in a tablet/smart phone, for example, or may be implemented as a hardware eye tracking device. The embodiments may be applied to, for example, an autonomous vehicle, an intelligent vehicle, a smart phone, and a mobile device.

In operation 110, the image processing apparatus acquires an image frame including the user's face. The image frame may include, for example, a color image frame, and/or an infrared image frame. The image frame may correspond to, for example, an image of a driver captured by a camera, an image sensor, a vision sensor, etc. installed inside the vehicle.

In operation 120 , the image processing apparatus detects the user's face region from the image frame obtained in operation 110 . The image processing apparatus may detect the user's face region from the image frame by, for example, setting the scan region using a detector previously learned to detect the face region by machine learning. In this case, the detector may be learned to detect not only the user's bare face but also the user's face wearing sunglasses. The scan area by the detector may be expressed, for example, in the form of a detection box to be described later.

For example, when the image frame acquired in step 110 is the first image frame acquired by the image processing apparatus, that is, an initial image frame, the image processing apparatus may display the user's face included in the image frame. area can be detected. Contrary to this, when the image frame obtained in step 110 is not the initial image frame, the image processing apparatus may obtain or receive the detected face region based on any one of at least one previous image frame of the corresponding image frame. have. For example, if the corresponding image frame is acquired at time t, at least one previous frame is a previous time (eg, time t-1) or previous viewpoints (eg, time t-1) based on time t at which the corresponding image frame was acquired For example, it may be obtained at time t-1, time t-2, time t-3).

In operation 130 , the image processing apparatus classifies the type of occlusion of the face region according to whether occlusion exists in at least a part of the face region detected in operation 120 . The image processing apparatus may classify the type of occlusion with respect to the face region by using a pre-learned classifier to distinguish the region where the occlusion exists among the face regions by, for example, machine learning. The type of occlusion is, for example, a first type corresponding to a bare face in which occlusion does not exist, and at least one of a forehead, an eyebrow, and at least a part of an eye among the user's face regions by a hat, etc. It may include a second type that is covered, a third type in which an eye area is covered among the user's face areas by sunglasses, etc., and a fourth type in which at least one of a mouth and a nose in the user's face area is covered by a mask or the like.

In operation 130 , for example, when all of the eyebrows, eyes, nose, and mouth regions are detected in the face region detected in operation 120 , the image processing apparatus determines the type of occlusion of the face region as a bare face. It can be classified into the first type corresponding to . For example, when at least a part of an eye region, a nose, and a mouth region of a face region are detected, and any one region of a forehead, an eyebrow, and at least a part of an eye region is not detected, the image processing apparatus may determine the type of occlusion. It can be classified as a second type in which the eyebrows and/or eyes are covered by For example, when the nose and mouth regions are detected among the face regions and the eye region is not detected, the image processing apparatus may classify the occlusion type as a third type in which the eyes are covered by sunglasses. In addition, when, for example, eyebrows and eye regions are detected among the face regions and at least one of the nose and mouth is not detected, the image processing apparatus determines the type of occlusion by the mask. It can be classified as a hidden fourth type.

Alternatively, according to an embodiment, the type of occlusion may be divided into at least one of type A, which corresponds to a bare face in which no occlusion exists, and type B, in which an eye region is covered by sunglasses among the user's face regions. have. A method of tracking a user's eyes when the type of occlusion is divided into two types, a bare face and a type covered by sunglasses, will be described in detail with reference to FIGS. 6 and 7 below.

In operation 140 , the image processing apparatus tracks the user's eyes based on the type of occlusion classified in operation 130 . For example, when there is no occlusion in the face region, image processing finds the center of the pupil in the user's eye using, for example, image features of the region corresponding to the shape of the eyes and nose, and according to the center of the pupil It can track the user's eyes. Contrary to this, when the user's eyes are covered by sunglasses or a hat, it is impossible to track the user's eyes, so a method of tracking the user's eyes may vary depending on the type of occlusion. The image processing apparatus may track the user's eyes by aligning different numbers of vertices to at least some shapes of the face region based on the type of occlusion classified in operation 130 . The image processing apparatus may move and align a different number of vertices with respect to at least a partial shape of the face region, for example, based on image information within the face region. Here, the 'vertices' may be feature points corresponding to key points representing facial features, such as, for example, eyes, nose, mouth, and face outline. Vertices may be indicated, for example, in the form of a dot (·) as shown in FIG. 4 , an asterisk (*) or other various forms.

The image processing device, for example, uses a descent vector learned from the initial shape configuration to align the vertices to the shape of the image using the SDM (Supervised Descent Method) technique, and analyzes the shape and major components of the shape. Based on principal component analysis (PCA), the user's eyebrows and eyes are extracted from the face area of the image frame by the Active Shape Model (ASM) technique, the Active Appearance Model (AAM) technique, or the Constrained Local Models (CLM) technique that aligns the vertices. , the position of a plurality of characteristic parts corresponding to the nose, mouth, and the like may be recognized.

The image processing apparatus may move and align a different number of vertices to positions of a plurality of recognized feature regions based on the type of occlusion classified in operation 130 . For example, when the image frame is an initial image frame, a plurality of vertices before alignment may correspond to an average position of feature portions of a plurality of users. Also, when the image frame is not the initial image frame, the plurality of vertices before alignment may correspond to the plurality of vertices aligned based on the previous image frame. A method for the image processing apparatus to track the user's eyes will be described in detail with reference to FIGS. 2 to 4 below.

According to an embodiment, the image processing apparatus may check whether or not tracking of the eye by the aligned vertices is successful. The image processing apparatus re-detects the face region according to the determination that the tracking of the user's eyes has failed, and classifies the type of occlusion based on the re-detected face region ( 130 ), and By repeatedly performing the tracking 140 , the tracking of the user's eyes may be successful. For example, an operation of the image processing apparatus when the user's eye tracking fails will be described in detail with reference to FIG. 5 below.

In operation 150 , the image processing apparatus outputs information on the face region including the position of the eyes tracked in operation 140 . In operation 150 , the image processing apparatus may output information on a view point based on the position of the pupil and information on the user's facial expression, in addition to the position of the eye or pupil tracked in operation 140 . In this case, the information on the face region may include, for example, positions of pupils and noses included in the scan region, a viewpoint based on the position of the pupils, a user's facial expression expressed in the scan region, and the like. The image processing apparatus may output information about the face region explicitly or implicitly. 'Explicitly outputting information on the face region' means, for example, displaying the position of the pupil included in the face region and/or the expression expressed in the face region on the screen of the display panel, and/or as audio. It may include outputting, etc. Alternatively, 'implicitly outputting information on the face region' means adjusting the image displayed on the HUD (Head Up Display) by, for example, the position of the pupil included in the face region, a viewpoint based on the position of the pupil, or Alternatively, it may include providing a service corresponding to an expression expressed in the face region, and the like.

In operation 150 , the image processing apparatus may reproduce an image corresponding to the eye position tracked in operation 140 by performing light field rendering based on the eye position tracked in operation 140 . According to an embodiment, when the type of occlusion classified in operation 130 corresponds to any one of the second to fourth types, the image processing apparatus includes the eye position tracked in operation 140 . An image to which a glow effect is applied to the face region may be output. For example, when the type of occlusion is the second type or the third type, when the eyes are obscured, the position of the eyes is estimated and thus the precision may be low. Accordingly, when the occlusion type corresponds to the second type or the third type, the image processing apparatus may display 3D content having low crosstalk with respect to the estimated eye position and apply the glow effect to increase precision.

2 is a diagram for explaining a method of classifying a type of occlusion for a face region according to an embodiment, and FIG. 3 is a method of tracking a user's eyes based on the type of occlusion according to an embodiment. It is a flowchart shown.

2 and 3 , the image processing apparatus according to an embodiment may classify the type of occlusion of the face region using a pre-learned classifier. At this time, for example, the classifier aligns a different number of vertices according to the type of occlusion of at least some shapes of each face region stored in the training image DB, and then learns the SIFT features extracted from the aligned vertices. It can be a classifier. The classifier may be, for example, a Support Vector Machine (SVM) classifier.

The image processing apparatus may classify, for example, a type of occlusion with respect to the face region 210 of a bare face in which occlusion does not exist, as the first type. The image processing apparatus may classify a type of occlusion with respect to the face region 220 in which at least a portion of the forehead, eyebrows, and eyes are covered by, for example, a hat as the second type. The image processing apparatus may classify an occlusion type with respect to the face region 230 in which the eye region is covered by sunglasses as a third type. Also, the image processing apparatus may classify an occlusion type with respect to the face region 240 in which the mouth or the mouth and nose are covered by the mask as a fourth type.

When the occlusion type is the first type, the image processing apparatus may track the user's eyes by aligning the first number of vertices to the partial shape of the face region 210 as in operation 310 . When the occlusion type is the first type, the image processing apparatus may reduce the size of the detection box for detecting the face region to correspond to the shape of the eyes and nose among the face regions like the detection box 250 . The image processing apparatus may crop the shape of the eyes and nose by the reduced detection box 250 . The image processing apparatus may track the user's eyes and nose by aligning a first number (eg, 11) of vertices to the cropped shapes of the eyes and nose ( 260 ). A method in which the image processing apparatus aligns the first number of vertices will be described in detail with reference to the drawing 410 of FIG. 4 below.

In operation 310, the image processing apparatus may track the user's eyes by checking the alignment result of the face region corresponding to the user's eyes and nose using 11 vertices. The image processing apparatus may verify the alignment result of the face region corresponding to the eyes and the nose in the scan region, based on the image information in the scan region by the detection box 250 . The image processing apparatus may verify whether the scan area corresponds to the eye and the nose by using, for example, a verifier based on a Scale Invariant Feature Transform (SIFT) feature. Here, the 'SIFT feature' can be obtained through the following two steps. The image processing apparatus extracts, for example, candidate vertices having a locally maximum or minimum image brightness in the scale space by the image pyramid from the image data of the scan region, and filters the vertices with a low contrast ratio to select the vertices to be used for image registration. can be selected The image processing apparatus may generate a descriptor by obtaining a direction component through a gradient of a peripheral area based on the selected vertices, resetting the ROI based on the obtained direction component, and detecting the size of the vertex. Here, the descriptor may correspond to the SIFT feature.

For example, when the classified type of occlusion is the first type, the image processing apparatus aligns the first number of vertices to the shapes of the eyes and nose cropped by the reduced detection box 250 to align the user's eyes and a separate module for tracking the nose, and the module may be referred to as a 'bare face tracker'.

Otherwise, when the occlusion type is any one of the second to fourth types, the image processing apparatus aligns the second number of vertices, which is greater than the first number, in the overall shape of the face region as in step 320 . This allows tracking of the user's eyes.

In operation 320, the image processing apparatus aligns a second number (eg, 98) of vertices to the overall shape of the face region by a detection box for detecting the face region, and the aligned second number of vertices It is possible to estimate the position of the user's pupil in the face region. When the occlusion type is any one of the second to fourth types, the image processing apparatus may track the entire face ( 270 ). A method in which the image processing apparatus aligns the second number of vertices will be described in detail with reference to the drawing 430 of FIG. 4 below.

Also, the image processing apparatus according to an embodiment aligns the second number of vertices to the overall shape of the face region, for example, when the classified type of occlusion is the third type, and the aligned second number of vertices A separate module for estimating the position of the user's pupil in the face region may be included, and the module may be called a 'sunglasses tracker'.

4 is a diagram for describing a method of aligning different numbers of vertices based on a type of occlusion by an image processing apparatus according to an exemplary embodiment. Referring to FIG. 4 , for example, a diagram 410 showing aligned vertices when the type of occlusion is classified as the first type and vertices aligned when the type of occlusion is classified as the third type. An illustrated diagram 430 is shown. Hereinafter, for convenience of explanation, a method of arranging vertices around a third type corresponding to one of the second to fourth types will be described. As with type 3, vertices can be aligned.

For example, when the classified type of occlusion is the first type, the image processing apparatus may include a first number (eg, 11) in some shapes corresponding to the eyes and nose among the face regions as shown in the drawing 410 . By aligning the vertices of , we can track the user's eyes. In this case, the 11 vertices correspond to, for example, three vertices corresponding to the right eye, three vertices corresponding to the left eye, one vertex between the brows, the tip of the nose, and the contour of the nose, as shown in the figure 410 . may correspond to four vertices, but is not necessarily limited thereto.

Alternatively, when the classified type of occlusion is the third type, the image processing apparatus closes the user's eyes by aligning the second number (eg, 98) of vertices to the overall shape of the face region as shown in the figure 430 . can be tracked In this case, the 98 vertices are, for example, 19 vertices corresponding to the shapes of the user's right eyebrow and right eye, 19 vertices corresponding to the shapes of the left eyebrow and left eye, and the nose as shown in the figure 430 . It may correspond to 12 vertices corresponding to the shape, 15 vertices corresponding to the shape of the mouth, and 33 vertices corresponding to the outline shape of the face, but is not limited thereto. The image processing apparatus may track the user's eyes by estimating the position of the user's pupil in the face region based on the aligned 98 vertices.

5 is a flowchart illustrating a method of tracking a user's eyes according to another exemplary embodiment. Referring to FIG. 5 , when the tracking of the user's eyes fails, the image processing apparatus according to an embodiment re-detects the face region through steps 510 to 550 to succeed in tracking the user's eyes. can make it

Hereinafter, for convenience of explanation, a method of tracking eyes when the type of occlusion is classified as the first type will be described, but the same method is applied even when the type of occlusion is classified as the second type to the fourth type. can be applied.

In operation 510 , the image processing apparatus may align the first number of vertices to a partial shape of the face region. The image processing apparatus may recognize positions corresponding to, for example, the shape of the user's eyes and nose from the face region of the image frame, and move and align the first number of vertices to the recognized positions.

In operation 520 , the image processing apparatus may verify a result of aligning the first number of vertices in operation 510 . The image processing apparatus may verify whether the region in which the vertices are aligned corresponds to the actual eye and/or nose by using image information in the eye and/or nose region of the image frame.

In operation 530 , the image processing apparatus may determine whether the user's eyes are tracked according to the verification result of operation 520 .

If the user's eye tracking is successful in operation 530, the image processing apparatus may output the eye position in operation 540 (operation 540). In this case, the position of the eye may correspond to, for example, two-dimensional eye position coordinates or may correspond to a three-dimensional eye position coordinate.

If the user's eye tracking fails in operation 530 , the image processing apparatus may re-detect the face region from the image frame in operation 550 . Thereafter, the image processing apparatus may track the eyes by performing operations 510 to 530 on the face region re-detected in operation 550 .

6 is a diagram for describing an image processing method according to another exemplary embodiment. Referring to FIG. 6 , a process of outputting the coordinates of the eyes according to the type of occlusion by the bare face tracker 620 and the sunglasses tracker 640 by the image processing apparatus according to an exemplary embodiment is illustrated.

In operation 605, the image processing apparatus may acquire an image frame.

In operation 610 , the image processing apparatus may detect the user's face region from the image frame obtained in operation 605 . In this case, the image processing apparatus may detect the entire face area by, for example, a detection box or a scan window that sets a scan area in the image frame. According to an embodiment, the image processing apparatus may detect a region corresponding to eyes and a nose of a face or a region corresponding to sunglasses by using, for example, a detection box or a scan window.

In operation 615 , the image processing apparatus may classify the type of occlusion of the face region detected in operation 610 .

For example, suppose that the face region detected in step 610 corresponds to a bare face, so that the occlusion type is classified as the first type in step 615 . In this case, the image processing apparatus may output the coordinates of the eyes by tracking the eyes in the bare face by performing the processes of steps 621 to 625 by the bare face tracker 620 .

More specifically, in step 621, the image processing apparatus aligns the first number (eg, 11) vertices to the shapes of the eyes and nose cropped in the face region by the reduced detection box to determine the user's Eyes and nose can be tracked.

In operation 623 , the image processing apparatus may verify whether the eye or the eye and nose are properly tracked by the SIFT feature corresponding to the eleven vertices used in operation 621 . In operation 623 , the image processing apparatus may verify whether the eye is properly tracked by, for example, giving a higher weight to the eye than the nose among the SIFT features. The image processing apparatus may perform verification using the aforementioned support vector machine (SVM).

If it is verified that the eye is properly tracked in operation 623 , the image processing apparatus may output eye coordinates corresponding to the tracked eye position in operation 625 . If it is verified in operation 623 that the eyes are not properly tracked, the image processing apparatus may re-detect the face region in operation 610 . The image processing apparatus may track the user's eyes by reclassifying the type of occlusion in the redetected face region.

Alternatively, suppose that the face region detected in step 610 corresponds to a face whose eye region is covered by sunglasses, and thus the type of occlusion is classified as the third type in step 615 . In this case, the image processing apparatus may output the coordinates of the eyes by estimating the positions of the eyes on the face wearing the sunglasses by performing the processes of steps 641 to 645 by the sunglasses tracker 640 .

More specifically, in operation 641, the image processing apparatus aligns a second number (eg, 98) of vertices to the overall shape of the face region, and uses the aligned second number of vertices in the face region. By estimating the location of the user's pupil, the user's eyes can be tracked. In operation 641 , the image processing apparatus may estimate the position of the user's eye, more specifically, the pupil through machine learning using a database including a large amount of images, for example. In this case, the image processing apparatus may improve precision by weighting the estimated eye position.

In operation 643 , the image processing apparatus may verify whether the position of the user's eye (or pupil) estimated by using the second number of vertices in operation 641 is correct. In operation 643 , the image processing apparatus may verify whether the user's eye pupil position estimated by using vertices corresponding to regions other than sunglasses except for vertices corresponding to sunglasses is correct, for example. .

If it is verified that the position of the user's eye (or pupil) estimated in operation 643 is correct, in operation 645 the image processing apparatus may output the eye coordinates from the estimated eye position. If it is verified that the position of the user's eye (or pupil) estimated in operation 643 is not accurate, the image processing apparatus may re-detect the face region in operation 610 . The image processing apparatus may track the user's eyes by reclassifying the type of occlusion in the redetected face region.

7 is a flowchart illustrating an image processing method according to another exemplary embodiment. Referring to FIG. 7 , a process in which a 3D HUD apparatus for a vehicle, which is an example of an image processing apparatus according to an embodiment, tracks a user's eyes through steps 710 to 740 and outputs a tracking result is illustrated. .

In step 710, the HUD device acquires an image frame. The image frame may correspond to, for example, a color image frame or an infrared image frame.

In step 720, the HUD device determines whether the user wears sunglasses in the image frame. For example, the HUD device may determine whether to wear sunglasses using a pre-trained detector to detect whether a user's eye region is covered by sunglasses.

In step 730, the HUD device tracks the user's eyes using different types of eye trackers according to the determination in step 720 whether to wear sunglasses.

For example, in response to the determination in step 720 that the sunglasses are not worn, the HUD device may track the user's eyes using the first type of eye tracker. A first type of eye tracker may track the user's eyes, for example, by aligning a first number of vertices to some shape of the user's facial region.

More specifically, according to the determination that the sunglasses are not worn in step 720 , the HUD device determines the size of the detection box for detecting the face region of the user by the first type of eye tracker in step 730 , the eye of the face region. And it can be reduced to correspond to the shape of the nose. The HUD device tracks the user's eyes by cropping the shape of the eyes and nose by the reduced detection box by the first type of eye tracker, and then aligning the first number of vertices to the cropped shape of the eyes and nose. can do.

Alternatively, in response to the determination in step 720 that the sunglasses are worn, the HUD device may use a second type of eye tracker to track the user's eyes. The second type of eye tracker may track the user's eyes, for example, by aligning a second number of vertices greater than the first number to the overall shape of the user's facial region.

More specifically, in accordance with the determination in step 720 that the sunglasses are worn, the HUD device determines the overall shape of the face region by a detection box that detects the face region of the user by the second type of eye tracker in step 730 . Two vertices can be aligned. The HUD device may track the user's eyes by estimating the position of the user's pupil in the face region by the aligned second number of vertices.

In operation 740, the HUD device outputs information on the face region including the tracked eye position.

8 is a block diagram of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 8 , the image processing apparatus 800 according to an embodiment includes a sensor 810 and a processor 830 . The image processing apparatus 800 may further include a memory 850 , a communication interface 870 , and a display panel 890 . The sensor 810 , the processor 830 , the memory 850 , the communication interface 870 , and the display panel 890 may communicate with each other through the communication bus 805 .

The sensor 810 acquires an image frame including the user's face. The sensor 810 may be, for example, an image sensor that captures an input image by infrared illumination, a vision sensor, or an infrared camera. The image frame may include, for example, a face image of a user or an image of a user driving in a vehicle.

The processor 830 detects the user's face region from the image frame. In this case, the image frame may be obtained by, for example, the sensor 810 or received from the outside of the image processing apparatus 800 through the communication interface 870 .

The processor 830 classifies the type of occlusion with respect to the face region according to whether occlusion exists in at least a portion of the face region. The processor 830 tracks the user's eyes based on the type of occlusion. The processor 830 outputs information on the face region including the tracked eye position.

The processor 830 may output, for example, information on the face region including the tracked eye position to the display panel 890 , or the outside of the image processing device 800 through the communication interface 870 . It can also be output as

There may be one processor 830 or a plurality of processors 830 . The processor 830 may perform the at least one method described above with reference to FIGS. 1 to 7 or an algorithm corresponding to the at least one method. The processor 830 may execute a program and control the image processing apparatus 800 . The program code executed by the processor 830 may be stored in the memory 850 . The processor 830 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU).

The memory 850 stores the image frame acquired by the sensor 810 , the user's face region detected from the image frame by the processor 830 , the occlusion type for the face region classified by the processor 830 , and the processor 830 . ) can store the coordinates of the user's eyes tracked. Also, the memory 850 may store information on the face region output by the processor. The memory 850 may be a volatile memory or a non-volatile memory.

The communication interface 870 may receive an image frame from the outside of the image processing apparatus 800 . The communication interface 870 may output information about the face region detected by the processor 830 and/or the face region including the position of the user's eyes tracked by the processor 830 . The communication interface 870 may receive an image frame captured from the outside of the image processing apparatus 800 or information of various sensors received from the outside of the image processing apparatus 800 .

The display panel 890 may display the processing result of the processor 830 , such as information on the user's face region including the eye position tracked by the processor 830 , for example. For example, when the image processing apparatus 800 is embedded in a vehicle, the display 890 may be configured as a head-up display (HUD) installed in the vehicle.

The image processing device 800 includes, for example, a Head Up Display (HUD) device, a 3D digital information display (DID), a navigation device, a 3D mobile device, a smart phone, a smart TV, and a smart vehicle. can, but is not necessarily limited thereto. The 3D mobile device is, for example, a display device for displaying Augmented Reality (AR), Virtual Reality (VR), and/or Mixed Reality (MR), a head mounted display (Head Mounted Display). Display (HMD) and face mounted display (FMD) may be understood as including both.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

obtaining an image frame including a user's face;
detecting the face region of the user from the image frame;
classifying the type of occlusion for the face region according to whether occlusion exists in at least a part of the face region;
tracking the user's eyes based on the type of occlusion; and
outputting information on the face region including the tracked eye position
containing,
Image processing method. According to claim 1,
The type of occlusion is
a first type corresponding to a bare face in which the occlusion does not exist;
a second type in which at least one of a forehead, an eyebrow, and at least a part of an eye of the user's face area is covered;
a third type in which an eye region of the user's face region is covered; and
A third type in which at least one of a mouth and a nose among the face regions of the user is covered
containing at least one of.
Image processing method. According to claim 1,
The type of occlusion is
A type corresponding to a bare face in which the occlusion does not exist; and
Type B in which the eye area is covered by sunglasses among the face area of the user
containing at least one of.
Image processing method. According to claim 1,
The step of tracking the user's eyes
tracking the user's eyes by aligning different numbers of vertices to at least some shapes of the facial region based on the type of occlusion;
containing,
Image processing method. 4. The method of claim 3,
The step of tracking the user's eyes
tracking the user's eyes by aligning a first number of vertices to a partial shape of the face region when the type of occlusion is the first type; and
tracking the user's eyes by aligning a second number of vertices greater than the first number to the overall shape of the face region when the type of occlusion is any one of the second to fourth types;
containing,
Image processing method. 6. The method of claim 5,
When the type of occlusion is the first type, the step of tracking the user's eyes
reducing a size of a detection box for detecting the face region to correspond to shapes of eyes and nose among the face regions;
cropping the shape of the eyes and nose by the reduced detection box; and
tracking the user's eyes by aligning the first number of vertices to the cropped shape of the eyes and nose.
containing,
Image processing method. 6. The method of claim 5,
If the type of occlusion is any one of the second type to the fourth type, the step of tracking the user's eye may include:
aligning the second number of vertices to the overall shape of the face region by a detection box for detecting the face region; and
tracking the user's eye by estimating the location of the user's pupil in the face region by the aligned second number of vertices
containing,
Image processing method. 4. The method of claim 3,
The step of tracking the user's eyes
re-detecting the face region according to a determination that tracking of the user's eyes by the aligned vertices has failed; and
Classifying the type of occlusion based on the re-detected face region, and repeatedly performing the steps of tracking the user's eyes
further comprising,
Image processing method. According to claim 1,
The step of classifying the type of occlusion is
Classifying the type of occlusion with respect to the face region using a pre-learned classifier to distinguish the region where the occlusion exists among the face regions by machine learning
containing,
Image processing method. obtaining an image frame;
determining whether the user wears sunglasses in the image frame;
tracking the user's eyes using different types of eye trackers according to the determination of whether to wear the sunglasses; and
outputting information on the face region including the tracked eye position
containing,
Image processing method of HUD (Head Up Display) device. 11. The method of claim 10,
The step of tracking the user's eyes
tracking the user's eyes using a first type of eye tracker in response to a determination that the sunglasses are not worn; and
tracking the user's eyes using a second type of eye tracker in accordance with a determination that the sunglasses are worn;
containing,
HUD device image processing method. 12. The method of claim 11,
The first type of eye tracker comprises
tracking the user's eyes by aligning a first number of vertices to some shape of the user's face region;
The second type of eye tracker is
tracking the user's eyes by aligning a second number of vertices greater than the first number to the overall shape of the user's face region;
HUD device image processing method. 12. The method of claim 11,
Using the first type of eye tracker to track the user's eye comprises:
reducing a size of a detection box for detecting the user's face region to correspond to shapes of eyes and nose among the face regions;
cropping the shape of the eyes and nose by the reduced detection box; and
tracking the user's eye by aligning a first number of vertices to the cropped eye and nose shape;
containing,
HUD device image processing method. 12. The method of claim 11,
Using the second type of eye tracker to track the user's eye comprises:
aligning a second number of vertices to the overall shape of the face region by a detection box for detecting the face region of the user; and
tracking the user's eye by estimating the location of the user's pupil in the face region by the aligned second number of vertices
containing,
HUD device image processing method. 11. The method of claim 10
The step of determining whether to wear the sunglasses
Determining whether to wear the sunglasses using a pre-learned detector to detect whether the user's eye region is covered by the sunglasses
containing,
HUD device image processing method. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 15. a sensor for acquiring an image frame including a user's face; and
Detects the user's face region from the image frame, classifies the type of occlusion with respect to the face region according to whether occlusion exists in at least a part of the face region, and based on the occlusion type A processor that tracks the user's eyes and outputs information on the face region including the tracked eye position
containing,
image processing device. 18. The method of claim 17,
the processor is
tracking the user's eyes by aligning different numbers of vertices to at least some shapes of the facial region based on the type of occlusion;
image processing device. 19. The method of claim 18,
the processor is
when the type of occlusion is the first type, tracking the user's eyes by aligning a first number of vertices to some shape of the face region;
tracking the user's eyes by aligning a second number of vertices greater than the first number to the overall shape of the face region when the type of occlusion is any one of the second to fourth types;
image processing device. 18. The method of claim 17,
The image processing device
At least one of a Head Up Display (HUD) device, a 3D Digital Information Display (DID), a navigation device, a 3D mobile device, a smart phone, a smart TV, and a smart vehicle,
image processing device.

Images