TWI603270B - Method and apparatus for detecting person to use handheld device - Google Patents

Description

Method and device for detecting personnel using handheld device

The present invention relates to an image recognition technology, and more particularly to a method and apparatus for utilizing image recognition technology to detect whether a person uses a handheld device.

With the rapid development of technologies such as mobile communication, people can use a handheld device such as a feature phone or a smart phone to make calls, send messages, and even browse the Internet. In addition, with the advancement of semiconductor process, materials, mechanism design and other technologies, handheld devices are gradually equipped with a thin and light design to facilitate handheld operations. Therefore, the convenience brought by the handheld device makes people's life gradually unable to get rid of the handheld device.

However, people often want to handle other things at the same time when operating the handheld device. When people are dealing with many things at the same time, it will be very easy to cause accidents. For example, when people use a handheld device while driving a vehicle, traffic accidents may occur due to factors such as distraction. So how do you have Effective and immediate monitoring of driving behavior or other situations where it is not appropriate to use a handheld device to avoid accidents will be an issue that needs to be addressed in this area.

The invention provides a method and a device for detecting a person using a handheld device, which can determine whether a specific gesture and a person's mouth movement appear on the ear side of the person through the image recognition technology, so as to accurately and quickly determine whether the person uses the handheld device.

The present invention provides a method for detecting a person using a handheld device, which is suitable for use in an electronic device. The method includes the following steps. Capture images of people. Analyze the image to get a facial object. The ear side search area and the mouth comparison area are determined according to the face object. It is judged whether the person is using the handheld device by searching for whether a specific gesture exists in the ear search area and comparing whether the detected mouth motion information meets the preset mouth information in the mouth comparison area.

In an embodiment of the invention, the above-mentioned searching for the presence of a specific gesture in the ear-side search area, and comparing the detected mouth motion information in the mouth comparison area with the preset mouth information, To determine if a person is using a handheld device includes the following steps. Detects whether a person's hand object appears in the ear side search area. A finger feature point of the hand object is obtained, wherein the finger feature point includes a foreground edge and a gradient value. The finger feature points are compared to determine if there is a specific gesture.

In an embodiment of the invention, the above-mentioned searching for the presence of a specific gesture in the ear-side search area, and comparing the detected mouth motion information in the mouth comparison area with the preset mouth information, To determine if the person is using the handheld After the device, the following steps are further included. During the preset time, it is judged whether the ratio of the number of hand-held events of the handheld device and the number of facial events of the detected face object is greater than the proportional threshold value to generate a prompt signal.

In an embodiment of the invention, the ear side searching area includes a first ear area and a second ear area, and determining the ear side searching area and the mouth comparing area according to the facial object includes the following steps. According to the face object, the shoulder feature in the image is detected to obtain the shoulder object. The ear side search area is determined according to the shoulder object and the face object. The interference features in the ear side search area are filtered to detect the first ear area and the second ear area.

In an embodiment of the invention, the determining the ear side search area and the mouth comparison area according to the facial object includes the following steps. The nostril positioning points are identified from the facial objects. The mouth area is set based on the nostril positioning point. The image of the mouth area is image processed to determine the mouth of the person. The mouth contrast region is determined in the mouth region according to the mouth object.

In an embodiment of the present invention, whether the person is being judged by searching for whether a specific gesture exists in the ear side search area, and whether the mouth movement is in accordance with the preset mouth information in the mouth comparison area The use of a handheld device includes the following steps. The mouth image is obtained, and the mouth feature is obtained according to the mouth image. According to the characteristics of the mouth, the image of the mouth is determined to be an open motion image or a closed motion image. During the recording time of the mouth, all closed motion images or open motion images detected in the mouth comparison region are sequentially recorded and converted into a coding sequence. The code sequence is stored in the mouth motion information.

In an embodiment of the present invention, whether the person is being judged by searching for whether a specific gesture exists in the ear side search area, and whether the mouth movement is in accordance with the preset mouth information in the mouth comparison area The use of a handheld device includes the following steps. During the mouth comparison time, the image of the mouth comparison area is compared with the template image to generate a mouth type code. The mouth type code is stored in the code sequence. The code sequence is stored in the mouth motion information.

The present invention provides a device for detecting a person using a handheld device. The device comprises an image capturing unit, a storage unit and a processing unit. The image capturing unit captures an image of the person. The storage unit stores images and presets mouth information. The processing unit is coupled to the storage unit to obtain an image. The processing unit analyzes the image to obtain a facial object. The processing unit determines the ear side search area and the mouth comparison area according to the face object. Moreover, the processing unit determines whether the person is using by searching for whether a specific gesture exists in the ear side search area, and comparing whether the detected mouth motion information meets the preset mouth information in the mouth comparison area. Handheld device.

In an embodiment of the invention, the processing unit detects whether a human hand object appears in the ear side search area, and obtains a finger feature point of the hand object, wherein the finger feature point includes a foreground edge and a gradient value, and the processing The unit compares the finger feature points to determine if a particular gesture exists.

In an embodiment of the invention, the processing unit determines whether the ratio of the number of handheld events that the person is using the handheld device to the number of facial events that detect the facial object is greater than a proportional threshold value within a preset time.

In an embodiment of the invention, the ear side search area includes the first Ear area and second ear area. The processing unit detects the shoulder features in the image according to the facial object, and obtains the shoulder object. The processing unit determines the ear side search area according to the shoulder object and the face object, and filters the interference feature in the ear side search area to detect the first ear area and the second ear area.

In an embodiment of the invention, the processing unit identifies a nostril positioning point from the facial object, sets a mouth searching area based on the nostril positioning point, and performs image processing on the image of the mouth searching area to determine the mouth of the person. The object, and the mouth region is taken out at the mouth alignment area according to the mouth object.

In an embodiment of the invention, the processing unit obtains the mouth image through the image capturing unit, and obtains the mouth feature according to the mouth image, and the processing unit determines the mouth image as the opening motion image or the closing according to the mouth feature. In the motion image, during the recording time of the mouth, the processing unit sequentially records all the closed motion images or the motion motion images detected in the mouth comparison region and converts them into a coding sequence, and stores the coding sequence in the mouth motion. News.

In an embodiment of the invention, in the mouth comparison time, the processing unit compares the image of the mouth comparison area with the template image to generate a mouth type code, and stores the mouth type code into the code. In the sequence, the code sequence is stored in the mouth motion information.

Based on the above, the image recognition technology can monitor whether a specific gesture of the ear side search area of the person and the mouth comparison area are present by the image recognition technology, and at the same time, whether the mouth movement of the matching person meets the preset mouth information, Determine if the person is using a handheld device. In this way, it is possible to effectively and accurately determine whether the person is making Use a handheld device.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧ device

110‧‧‧Image capture unit

130‧‧‧storage unit

150‧‧‧Processing unit

170‧‧‧Image Processing Module 170

190‧‧‧Warning unit

210‧‧‧Face Detection Module

230‧‧‧ear area definition module

240‧‧‧Mouth area definition module

250‧‧‧ gesture determination module

270‧‧‧ mouth comparison module

280‧‧‧Event Counting Module

290‧‧‧ Warning Module

S310~S370, S410~S490, S510~S590, S610~S670, S710 ~S790‧‧‧Steps

400, 500‧‧ images

410, 510‧‧‧Face objects

440, 450, 530, 540 ‧ ‧ ear side search area

520‧‧‧Face area

W1, w2, w3, w4‧‧‧ width

H1, h2, h3, h4‧‧‧ height

1 is a block diagram showing an apparatus for detecting a person using a handheld device according to an embodiment of the invention.

2 is a block diagram of an image processing module in accordance with an embodiment of the present invention.

FIG. 3 is a flow chart showing a method for detecting a person using a handheld device according to an embodiment of the invention.

4 is a schematic diagram of a person image in accordance with an embodiment of the present invention.

FIG. 5 is a schematic diagram of a person image according to another embodiment of the present invention.

FIG. 6 is a flow chart showing an example of determining a mouth matching region according to an embodiment of the present invention.

FIG. 7 is a flow chart showing an example of recording mouth movement information according to an embodiment of the present invention.

FIG. 8 is a flow chart showing an example of recording mouth movement information according to another embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of a detecting person using a handheld device according to an embodiment of the invention.

When a mobile phone picks up a mobile phone to answer an incoming call, the handheld device is usually brought to the vicinity of the face so that the handset of the handheld device faces the ear and the microphone of the handheld device is placed close to the mouth. In addition, people may have different gestures for grabbing handheld devices due to different usage habits. Accordingly, the embodiment of the present invention performs image monitoring on a person, and uses image recognition technology to determine whether a specific gesture appears near the ear of the person to determine whether the person holds the mobile phone. At the same time, the embodiment of the present invention further determines whether the mouth motion of the person meets the preset mouth information to determine whether the person speaks. Thereby, it is possible to effectively and accurately judge whether or not the person is using the handheld device. A plurality of embodiments in accordance with the spirit of the present invention are set forth below, and those applying the present embodiment can be appropriately adjusted according to their needs, and are not limited to the contents described in the following description.

1 is a block diagram showing an apparatus for detecting a person using a handheld device according to an embodiment of the invention. Referring to FIG. 1 , the device 100 includes an image capturing unit 110 , a storage unit 130 , a processing unit 150 , an image processing module 170 , and a warning sheet 190 . In one embodiment, the device 100 is, for example, disposed within the vehicle for detecting the driver. In other embodiments, the device 100 can also be used in an automated transaction device such as an Automated Teller Machine (ATM) to determine, for example, whether the operator is answering the handheld device and performing a transfer operation. It should be noted that the embodiment of the present invention can be used to set the device 100 to any electronic device or location that requires the monitoring personnel to use the handheld device.

The image capturing unit 110 may be a charge coupled device (Charge coupled Device; CCD) lens, complementary metal oxide semiconductor transistor (CMOS) lens, or infrared lens camera, camera. The image capturing unit 110 is configured to capture an image of a person and store the image in the storage unit 130.

The storage unit 130 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk. (Hard Disk Drive; HDD) or the like or a combination of the above elements.

The processing unit 150 is coupled to the image capturing unit 110 and the storage unit 130. The processing unit 150 may be a chip set, a microprocessor or a micro control unit (MCU) having a computing function of a central processing unit (CPU). The processing unit 150 of the embodiment of the present invention is configured to process all the operations of the apparatus 100 of the embodiment. The processing unit 150 can acquire images through the image capturing unit 110, store the images in the storage unit 130, and perform image processing on the images.

The image processing module 170 is, for example, a code segment written by a computer program language. The code segment can be stored in the storage unit 130 (or another storage unit) and includes a plurality of instructions, which are executed by the processing unit 150. The above code fragment. In addition, in other embodiments, the image processing module 170 may also be a hardware component composed of one or more circuits, which is not limited herein.

2 is a block diagram of an image processing module 170 in accordance with an embodiment of the present invention. Referring to FIG. 2, the image processing module 130 includes a face detection module 210 and an ear side. The area definition module 230, the mouth area definition module 240, the gesture determination module 250, the mouth comparison module 270, the event number determination module 280, and the warning module 290.

For example, after the image capturing unit 110 captures the image of the person, the face detecting module 210 of the image processing module 170 detects the facial features in the image to obtain the facial object. Moreover, the ear side region defining module 230 and the mouth region defining module 240 respectively determine the ear side searching area and the mouth matching area according to the face object. Then, the gesture determination module 250 searches for whether a specific gesture exists in the ear side search area, and whether the mouth comparison module 270 compares the preset mouth information in the mouth comparison area to determine whether the person is The handheld device is being used. The event number judgment module 280 determines whether the ratio of the number of hand-held events that the judgment person is using the hand-held device to the number of face events in which the face object is detected is greater than the proportional threshold value. Moreover, the alert module 290 prompts the person based on the result of the determination. The detailed operational steps of each functional module are further explained in the following embodiments.

The warning unit 190 is, for example, a display unit such as a liquid crystal display (LCD) or an organic electro-luminescence display (OELD), or a light module having at least one light emitting diode (LED). One or a combination of the group, the vibration module including the vibration motor, and the speaker module having a mono or stereo speaker.

It should be noted that, in other embodiments, the image capturing unit 110 further has a lighting component for correcting light when the light is insufficient to ensure the sharpness of the image captured by the image capturing unit 110.

In order to help understand the technology of the present invention, the following context illustrates the application of the present invention. Use the method. It is assumed that the device 100 of the embodiment of the present invention is installed in a car, and a driver sits in a driving position (not conveniently described, the following is a "person" as the driver), and the image capturing unit 110 on the device 100 can perform the personnel. Shooting. The image captured by the image capturing unit 110 may include a person's face, shoulders, or even a half body. In addition, it is assumed that the handheld device is placed anywhere near the gear or above the dashboard. Herein, the details will be described below in conjunction with various embodiments.

FIG. 3 is a flow chart showing a method for detecting a person using a handheld device (for example, a mobile phone of a type such as a smart phone or a smart phone) according to an embodiment of the invention. Referring to FIG. 2, the method of this embodiment is applicable to the device 100 of FIG. 1 and the image processing module 170 of FIG. Hereinafter, the methods described in the embodiments of the present invention will be described in conjunction with the components and modules in the device 100 and the image processing module 170. The various processes of the method can be adjusted accordingly according to the implementation situation, and are not limited thereto.

In step S310, the processing unit 150 captures the image of the person through the image capturing unit 110. For example, the image capturing unit 110 can set the shooting speed of 30 sheets, 45 sheets, or the like, or only one image to capture the person and store the captured image in the storage unit 130.

In other embodiments, the processing unit 150 may also set a startup condition in advance. When the start condition is met, the processing unit 150 can enable the image capture unit 110 to capture an image of the person. For example, a sensor (eg, an infrared sensor) may be disposed in the vicinity of the image capturing unit 110. The device 10 uses an infrared sensor to detect whether a person is located within a range in which the image capturing unit 110 can capture an image. If the infrared sensor detects that a person appears in front of the image capturing unit 110 (ie, matches When the condition is started, the processing unit 150 enables the image capturing unit 110 to start capturing images. In addition, a start button may be disposed on the device 100, and when the start button is pressed, the processing unit 150 activates the image capturing unit 110. It should be noted that the foregoing is merely illustrative, and the invention is not limited thereto.

In addition, the image processing module 170 can also perform background filtering on the captured image sequence. For example, the first image and the first +1 image are subjected to differential processing, and I is a positive integer. Thereafter, the image processing module 170 can convert the image filtered back to a grayscale image, thereby performing subsequent actions.

Then, the processing unit 150 starts to perform an image processing process on each image of the video sequence through the image processing module 170. In step S330, the processing unit 150 analyzes the image through the face detection module 210 to obtain a facial object. Specifically, the processing unit 150 analyzes one or more images to obtain facial features (eg, eyes, nose, lips, etc.), and then uses the alignment of the facial features to find facial objects in the image. For example, the storage unit 130 stores a feature database. This feature database includes facial feature samples. The face detection module 210 obtains the face object by comparing with the samples in the feature database. For the technique of detecting a face, the embodiment of the present invention can utilize an AdaBoost algorithm or other face detection algorithms (for example, Principal Component Analysis (PCA), Independent Component Analysis (ICA), etc. The algorithm or the Haar-like feature is used to perform face detection actions, etc.) to obtain facial objects in each image.

In other embodiments, the face detection mode is detected before the facial features are detected. The group 210 can perform the background filtering action first. For example, the processing unit 150 can capture at least one background image of the non-existing image through the image capturing unit 110, so that after obtaining the image of the person, the face detecting module 210 can perform the image including the portrait and the background image. Subtract, so you can filter out the background. Afterwards, the face detection module 210 can convert the image filtered back into a grayscale image and then convert it into a binarized image. At this time, the face detection module 210 can detect the facial features in the binarized image.

In step S350, the processing unit 150 determines at least one of the ear side search area and the mouth comparison area through the ear side area definition module 230 and the mouth area definition module 240 according to the face object.

In an embodiment, the ear side search area includes a first ear area and a second ear area. The ear side region definition module 230 detects the shoulder features in the image according to the face object, and obtains the shoulder object. The ear side region definition module 230 determines the ear side search area according to the shoulder object and the face object, and filters the interference feature in the ear side search area to detect the first ear area and the second ear area. In this embodiment, the interference feature is, for example, at least one of a glasses interference feature and a hair interference feature.

Specifically, in the area to be detected, the ear side area defining module 230 can detect whether two horizontally symmetric edge features exist under the face object, and there are two sides of the face object. When symmetrical edge features, this edge feature is considered a shoulder to obtain a shoulder object.

For example, FIG. 4 is a schematic diagram of a human image in accordance with an embodiment of the present invention. In this embodiment, it is assumed that the storage unit 130 of the device 100 that detects the person using the handheld device has a feature database. This feature database includes facial feature samples Samples such as (pattern), shoulder feature samples, and interference feature samples. The ear side region definition module 230 obtains the face object and the shoulder object by comparing the captured image with the sample in the feature database.

Referring to FIG. 4, the ear side region definition module 230 can obtain the face object 410 in the image 400, and search for the shoulder object 420, 430 according to the position of the face object 410. At this time, the ear side region defining module 230 can obtain the ear side searching regions 440, 450 according to the face object and the shoulder object. For example, the search area 440 is, for example, the distance between the highest point of the face object 410 on the vertical axis and the lowest point of the shoulder object 420 on the vertical axis as the height h1, and on the horizontal axis of the face object 410. The distance between the reference point (eg, at the 1/3 width of the face item 410) and the edge of the shoulder article 420 is taken as the width w1. The search area 450 is, for example, the distance between the highest point of the face object 410 on the vertical axis and the lowest point of the shoulder object 430 on the vertical axis as the height h2, and the other of the face object 410 on the horizontal axis. The distance between a reference point (eg, at a 2/3 width of the face item 410) and the edge of the shoulder article 430 is taken as the width w2. It should be noted that the height h1 and the height h2, the width w1 and the width w2 may be the same or different, and the position of the shoulders of the person is determined.

Alternatively, the ear side region definition module 230 can detect whether a person wears clothes under the face object, and determines the shoulder object according to the edge feature of the clothes. For example, a horizontally symmetrical edge feature of the highest point of the garment extending from the leftmost and rightmost ends of the garment to the vertical axis is used as the shoulder article.

In another embodiment, the ear side region definition module 230 can also be based only on the face. The length or width of the object determines the area of the ear side. For example, FIG. 5 is a schematic diagram of a person image according to another embodiment of the present invention. Referring to FIG. 5, the ear side region definition module 230 can obtain the face object 510 in the image 500, and according to the width between the rightmost end and the leftmost end of the face object 510 on the horizontal axis (or the face object on the vertical axis) The upper point and the length between the lowest points determine a face area 520. This face region 520 is, for example, a square region whose width and width are both widths w3. The ear side search area 530 is, for example, the highest point h3 of the face area 520 on the vertical axis downward height h3 (for example, a quarter length of the width w3) as the highest point of the ear side search area 530 on the vertical axis. And the height h4 (for example, the same length as the width w3) is used as the length of the ear side search region 530 on the vertical axis, and then the leftmost end of the face region 520 on the horizontal axis is used as the ear side search region 530 on the horizontal axis. The far right end of the face region 520 extends horizontally to the left at the leftmost end on the horizontal axis by a width w4 (for example, 0.7 times the width w3 length) as the leftmost end of the ear side search region 530 on the horizontal axis. The ear side search area 540 is symmetric with respect to the ear side search area 530. According to the ear side search area 530, the boundary of the ear side search area 540 can be determined, and details are not described herein again. It should be noted that the length ratios between the widths w3 and w4 and the heights h3 and h4 may be changed according to design requirements, and the embodiment of the present invention is not limited thereto.

In addition, in other embodiments, the ear side region definition module 230 may also determine only one ear side search region. For example, the distance between the highest point of the facial article 410 on the vertical axis in FIG. 4 and the lowest point on the vertical axis of the shoulder articles 420, 430 is taken as the height of the ear side search region. Also, the width (shoulder width) of the shoulder articles 420, 430 on the horizontal axis is taken as the width of the ear side search region.

On the other hand, in addition to determining the ear side search area according to the facial object, in another embodiment, the mouth area definition module 240 identifies the nostril positioning point from the nostril object, and sets the mouth area based on the nostril positioning point. The image of the mouth region is image-processed to determine the mouthpiece of the person, and the mouth-matching region is determined in the mouth region according to the mouthpiece.

For example, FIG. 6 is a flow chart showing an example of determining a mouth matching region according to an embodiment of the present invention. The mouth area defining module 240 can set the mouth area based on the nostril position information, and obtain a enhanced image by adjusting the contrast in the mouth area by, for example, a difference in the color of the lips and the color of the skin and the teeth (step S610). And further strengthen the image to remove the noise. For example, the pixels are filtered through the pixel matrix. The mouth area definition module 240 can obtain a clear denoised image with respect to the enhanced image (step S630). Next, the mouth region definition module 240 performs edge sharpening processing according to the degree of contrast of a certain color in the image with another color to determine an edge in the denoising image, thereby obtaining a sharpened image (step S550). Since the complexity of the image will determine the memory capacity occupied by the image, in order to improve the performance of the comparison, the mouth region definition module 240 performs binarization processing on the sharpened image. For example, the mouth area definition module 240 first sets the threshold value and divides the pixels in the image into two values exceeding or lower than the threshold value to obtain a binarized image (step S670). Finally, the mouth region definition module 240 performs edge sharpening processing on the binarized image again. At this time, the lip portion of the person in the binarized image is quite conspicuous, and the mouth region defining module 240 can take out the mouth matching region in the mouth region (step S690).

It should be noted that, in the embodiment of the present invention, the ear side position area and the mouth comparison area may be determined according to design requirements, for example, facial features for different people (for example, face width, ear size, lip width, etc.) Adjustments are made and the invention is not limited thereto.

In step S370, the processing unit 150 searches for the specific gesture in the ear side search area by the gesture determination module 250, and the mouth comparison module 270 compares the detected mouth in the mouth comparison area. Whether the action information conforms to the preset mouth information to determine whether the person is using the handheld device.

In one embodiment, the gesture determination module 250 detects whether a person's hand object appears in the ear side search area, and obtains a finger feature point of the hand object, wherein the finger feature point includes a foreground edge and a gradient value. Next, the gesture determination module 250 compares the finger feature points to determine whether a specific gesture exists.

For example, the gesture determination module 250 compares the current image with the reference image (which may be a previous image, such as the previous image or the first N images of the current image, or any of the preset images). The ear side search area (eg, the interest area R of FIG. 4) performs an image subtraction algorithm to obtain a foreground image. The gesture determination module 250 filters out the noise of the foreground image by using the ear side search area of the reference image to obtain the foreground edge. Alternatively, the gesture determination module 250 can utilize an edge detection algorithm to calculate the foreground edge. Thereby, the gesture determination module 250 can determine whether the hand object is present in the ear side search area according to the foreground image or the foreground edge. The gesture determination module 250 uses finger features (eg, nails, finger joints, finger distances, etc.) to determine the foreground edge of the hand item. Gesture determination module 250 can also Features such as the angle of the wrist or the size of the gesture block are used as the judgment of the foreground edge.

In addition, the gesture determination module 250 further calculates a gray scale value function of the image on the ear side search area, and calculates the hand object according to the gray scale change (for example, a step edge or a line edge). Gradient value.

In some embodiments, the storage unit 130 may store a skin color sample of the hand item in advance, and the gesture determination module 250 may determine the hand item according to the color. For example, the gesture determination module 250 can determine the skin color value of the hand object according to the face object or the skin color sample, and find whether the same or similar color value appears in other areas than the facial object in the image to identify the hand object. Alternatively, the gesture determination module 250 can determine the edge of the hand object according to the change in the skin color in the image frame.

In this embodiment, the storage unit 130 may also store a plurality of gesture samples corresponding to specific gestures, and the gesture samples have respective preset foreground edges and preset gradient values. The gesture determination module 250 can compare the preset foreground edge and the preset gradient value with the foreground edge and the gradient value determined by the image recognition, and if the comparison is matched, determine that the hand object in the ear side search area meets Specific gestures. Then, if the characteristics of the specific gesture are detected in the ear-side search area on each image in the sequence image detected within a predetermined time (for example, 1 second, 2 seconds, etc.), Confirm the condition of the person holding the handheld device.

It should be noted that, in some embodiments, the gesture determination module 250 can further determine the person's hand according to the specific gesture detected by the two ear regions (eg, the first ear region and the second ear region). The condition of the handheld device. For example, hand The potential determination module 250, when one of the first ear region and the second ear region searches for a particular gesture, will determine that the person is holding the handheld device forehand. The gesture determination module 250 determines that the person holds the handheld device when the first ear region and the second ear region simultaneously search for a specific gesture. When there is no specific gesture in the first ear region and the second ear region, the gesture determination module 250 also determines that the person is not using the handheld device.

In the process of the gesture determination module 250 searching for a specific gesture, the mouth comparison module 270 can be compared in the mouth comparison region at the same time or at a certain time interval (for example, 1 second, 2 seconds, etc.). Whether the detected mouth motion information conforms to the preset mouth information. For example, the preset mouth information is a code sequence, a picture change rate, or a pixel change rate. During a comparison time (for example, 2 seconds, 5 seconds, etc.), the mouth comparison module 270 can compare the mouth motion information obtained by the image sequence with a preset coding sequence, image variation rate, or pixel variation rate. . If the mouth comparison module 270 determines whether the mouth motion information conforms to the preset mouth information, the determination person is using the handheld device. On the contrary, the judger does not use the handheld device.

It should be noted that the mouth comparison module 270 records the mouth motion information for subsequent comparison before comparing whether the mouth motion information detected in the mouth comparison region meets the preset mouth information. . The embodiment will be described below.

In one embodiment, the mouth comparison module 270 obtains the mouth image, and obtains the mouth feature according to the mouth image, and determines the mouth image as the open motion image or the closed motion image according to the mouth feature. During the mouth recording time, the mouth comparison module 270 sequentially records all the closed motion images or the open motion images detected in the mouth comparison region and converts them into a coding sequence, and stores the coding sequence in the mouth. Department action information.

For example, FIG. 7 is a flow chart showing an example of recording mouth movement information according to an embodiment of the present invention. Referring to FIG. 7, the mouth comparison module 270 takes out a plurality of mouth features in the mouth comparison region (step S710). For example, the mouth features include an upper lip portion and a lower lip portion. Specifically, the method of extracting the mouth features by the mouth comparison module 270 can define the left and right corners of the mouth by finding the left and right borders of the mouth region. Similarly, the mouth comparison module 270 recognizes the upper and lower lip portions by the line connecting the upper and lower sides of the mouth comparison region and the line connecting the left and right corners. Next, the mouth comparison module 270 compares the distance between the upper lip portion and the lower lip portion with a gap value (for example, 0.5 cm, 1 cm, etc.) (step S720). It is judged whether or not the interval between the upper lip portion and the lower lip portion is larger than the gap value (step S730). If the distance between the upper lip portion and the lower lip portion is greater than the gap value, the user's mouth is opened, and an open motion image is obtained (step S740). On the contrary, the mouth comparison module 270 obtains a closed motion image (step S750).

The mouth comparison module 270 generates a code according to the closed motion image or the open motion image, and stores the code in the code sequence (for example, the Nth column, N is a positive integer) (step S760). The coding sequence can be binary coded or encoded in the form of a Morse code. For example, the open motion image is defined as 1 and the closed motion image is defined as 0. If the person opens his mouth for two unit hours and then closes his mouth for two unit hours, the coding sequence is (1, 1, 0, 0). Next, the mouth comparison module 270 determines whether or not the mouth recording time is reached (step S770). For example, the mouth comparison module 270 starts a timer at step S710, and determines whether the timer reaches the mouth record at step S770. between. If the mouth comparison module 270 determines that the timer has not reached the mouth recording time, N=N+1 is made (step S780), and the process returns to step S710 to continue to distinguish the opening condition of the mouth. And, the next generated code will be stored, for example, to the next field in the code sequence (for example, column N+1). Each of the Nth fields represents a unit time (for example, 200 milliseconds, 500 milliseconds, etc.), and the code stored in the field represents all open motion images recorded in a unit time and the sequence of the closed motion images. .

It should be noted that, in this example, a delay time (for example, 100 milliseconds, 200 milliseconds, etc.) may be added in the processes of steps S710 to S780, so that the time spent in the processes of steps S710 to S780 is equal to the unit time, so that each The N field represents a unit time. Finally, the mouth matching module 270 stores the code sequence in the mouth motion information (step S790).

In another embodiment, during the mouth comparison time, the mouth comparison module 270 compares the image of the mouth alignment area with the template image to generate a mouth pattern. The mouth comparison module 270 stores the mouth type code in the code sequence and stores the code sequence in the mouth motion information.

For example, FIG. 8 is a flow chart showing an example of recording mouth movement information according to another embodiment of the present invention. In this embodiment, the mouth motion information may also represent a combined sequence of a plurality of mouth types. Referring to FIG. 8, the mouth comparison module 270 compares the image of the mouth alignment area with a plurality of pattern images in the storage unit 130 (step S810). The sample image can be a specific specific mouth motion image or lip language, for example, reading the Japanese syllabary. , , , , "Chinese, "Hello, Hello, Please, I am" or English "hello" and other muscles in the mouth. These template images have a certain degree of flexibility, even if there is a slight difference between the mouth shape and the template image in the human face image, as long as the difference is within the allowable range of the variable elasticity, the mouth comparison module 270 can still be recognized as The sample image matches.

Next, the mouth matching module 270 determines that the image of the mouth matching area matches the template image (step S820). If the result of the comparison is consistent, the mouth comparison module 270 generates a mouth type code and stores the mouth type code in the code sequence (for example, the Mth column, M is a positive integer) (step S830). If the result of the comparison is not satisfied, the processing unit 150 sets M = M + 1 (step S840), and returns to step S810. Next, the mouth comparison module 270 determines whether or not the mouth comparison time is reached (step S850). For example, the mouth comparison module 270 starts the timer at step S810, and determines whether the timer reaches the mouth comparison time at step S840. When the timer reaches the mouth comparison time, the mouth comparison module 270 stores the code sequence in the mouth motion information (step S870). If the mouth comparison module 270 determines that the timer has not reached the mouth comparison time, then M=M+1 (step S860), and returns to step S810 to continue comparing the image of the mouth comparison area with the template image. .

In addition, before the event number determination module 280 determines whether the person is using the handheld device, the false positive condition is further avoided depending on, for example, the number of finger feature points and the number of events in which the handheld device is used. The following is a description of the examples.

In an embodiment, the event number determination module 280 determines whether the ratio of the number of hand-held events that the person is using the handheld device to the number of face events that detect the face object is greater than the proportional threshold value within the preset time. Specifically, the event number determination module 280 first determines whether the handheld device detection function has been activated for a preset period of time (for example, 5 seconds, 10 seconds, etc.), and then detects the detected face object during the detection time ( The number of facial events N _{F of the} event F) and the specific gesture of the ear side search area and the number of hand-held events N _{P of the} mouth movement information matching the preset mouth information (event P) are simultaneously detected. Next, the event number determination module 280 calculates the ratio of the number of face events N _P to the number of hand-held events N _F and determines whether the ratio is greater than a threshold (eg, 0.6, 0.7, etc.) to determine whether the person is using the handheld device. That is to say, the event number judging module 280 may detect the event P on the premise that the face object is detected.

In addition, in another embodiment, during the preset time, the event number determining module 280 further determines whether the number of facial events N _{P of the} detected facial object is greater than the event threshold (for example, 10 times, 15 times). Wait). It should be noted that when the above conditions are not met (for example, the ratio of the number of facial events N _P to the number of hand-held events N _F is not greater than the threshold or the number of facial events N _{P is} not greater than the threshold of the event), the number of events is determined. When the module 280 arrives at the next preset time, it is determined again whether the above conditions are met.

In addition, when the processing unit 150 determines that the person is using the handheld device through the mouth area definition module 240, the mouth comparison module 270, and the event number determination module 280, the processing module 150 further activates the warning program through the warning module 290. Specifically, the processing unit 150 generates a prompt signal to the alert module 290, and the alert module can alert the person according to the prompt signal. For example, the alert module 290 can display a text, image, or image through the alert unit 190 to indicate a warning (eg, be careful not to use the handheld device during driving!). The light module of the warning unit 190 can also flash or emit at a specific frequency. Lights of a specific color (for example, red, green, etc.). The vibration module of the warning unit 190 vibrates, for example, by a vibration motor to generate a fixed frequency or a variable frequency. Alternatively, the alert module 290 can also emit a prompt tone through the speaker module of the alert unit 190.

In some embodiments, the preset mouth information stored in the storage unit 130 may be a preset mouth type coding sequence obtained by combining the template image arrangement, and each preset mouth type coding sequence corresponds to the prompt signal. . For example, when the user is held hostage, the person can make a "salvation" action from the mouth, but does not have to make a sound. The processing unit 150 can cause the alert unit 190 to generate a distress signal and send it to the security center or the like for assistance in a manner that is difficult to detect (the alert module can include a communication module).

Therefore, the device 100 of the embodiment of the present invention can detect whether a person uses a handheld device through an accurate and fast image recognition technology, so as to reduce the incidence of accidents caused by answering an incoming call during driving. The following is an example.

FIG. 9 is a diagram illustrating an example of a detecting person using a handheld device according to an embodiment of the invention. Referring to FIG. 9, the processing unit 150 obtains a video image through the image capturing unit 110 (step S910), and the face detecting module 210 determines whether a human face appears in the video image (step S920). If the face detection module 210 determines that there is no face on the image, the process returns to S910 to continue to obtain the connected image. On the other hand, when the face detection module 210 detects a human face, the ear side region definition module 230 and the mouth region definition module 240 respectively determine the ear side search area and the mouth comparison area according to the face object (steps) S930). Then, the gesture determination module 250 detects subsequent image frames. Whether the hand item of the person appears, and the foreground edge and the gradient in the finger feature point are obtained (step S950), and the skin module (for example, the code part written by the computer programming language) pre-stored by the storage unit 130 is used to compare the opponent. And determining an appearance gesture in the image frame (step S960), and comparing the presence or absence of a specific gesture in the image frame by using a finger feature point detection module (for example, a code segment written in a computer programming language) (step S970), it is further determined whether the mouth movement in the mouth comparison area conforms to the preset mouth information. Then, the gesture determination module 250 determines whether the detection result passes a spatial filter and a temporal filter (step S980). Spatial filtering, for example, is to determine whether a particular gesture appears in the ear side search area determined by the facial object. The time filtering is, for example, determining whether the ratio of the number of hand-held events detected using the mobile phone to the number of face events detecting the face is greater than 0.6 within 5 seconds. If the detection result fails to pass spatial filtering or time filtering, it returns to S910. On the contrary, the warning module 290 issues an alert through the alert unit 190 (step S990).

It should be noted that the above situation of driving a car (which may also be an airplane, a ship, etc.) is for the purpose of assisting the description of the embodiment, but the embodiment of the present invention can also be applied to an electronic device of an automatic transaction device or other monitoring personnel who is using the handheld device. Or place.

In summary, the device according to the embodiment of the present invention can determine whether a specific gesture exists in the ear side search area by using image recognition technology, and determine whether the mouth motion of the person meets the preset mouth information. When the judgment results are all met, the device of the embodiment of the present invention can judge that the person is using the handheld device, and can also issue a prompt signal to alert the person. Thereby, the embodiment of the invention can effectively and immediately monitor the driving behavior or other situations that are not suitable for using the handheld device, for example, the driver can be more alert. Automated trading devices help police units quickly deal with issues such as phone fraud.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S310~S370‧‧‧Steps

Claims (14)

A method for detecting a person using a handheld device, which is applicable to an electronic device, comprising: capturing at least one image of a person; analyzing the at least one image to obtain a facial object; determining at least one ear side according to the facial object Searching area and a mouth matching area; and searching for whether a specific gesture exists in the at least one ear side search area, and simultaneously detecting a mouth motion information detected in the mouth matching area Compliance with a preset mouth information to determine whether the person is using a handheld device, wherein the mouth matching area is detected to generate a code sequence, and determining whether the mouth motion information conforms to the preset according to the code sequence Mouth information. The method of claim 1, wherein the mouth motion is detected in the mouth comparison region by searching for the specific gesture in the at least one ear search region. The step of determining whether the person is using the handheld device includes: detecting whether the first hand object of the person is present in the at least one ear side search area; obtaining at least one of the hand items a finger feature point, wherein the at least one finger feature point includes a foreground edge and at least one gradient value; and determining whether the particular gesture exists by comparing the at least one finger feature point. The method of claim 1, wherein at least one of Determining whether the specific gesture is present in the ear search area, and comparing whether the detected mouth motion information meets the preset mouth information in the mouth comparison area to determine whether the person is using the handheld After the step of the device, the method further includes: determining, within a preset time, whether the ratio of the number of handheld events that the person is using the handheld device to the number of facial events detecting the facial object is greater than a proportional threshold To generate a prompt signal. The method of claim 1, wherein the at least one ear side search area comprises a first ear area and a second ear area, and the at least one ear side search area and the mouth are determined according to the facial object The step of comparing the regions includes: detecting a shoulder feature of the at least one image according to the facial object, and obtaining a shoulder object; determining the ear side search according to the shoulder object and the facial object And filtering an interference feature in the ear side search area to detect the first ear area and the second ear area. The method of claim 1, wherein the step of determining the at least one ear side search area and the mouth comparison area according to the facial object comprises: identifying a nostril positioning point from the facial object; Setting a mouth region based on the nostril positioning point; and performing an image processing on the image of the mouth region to determine a mouth object of the person; The mouth matching region is determined in the mouth region according to the mouthpiece. The method of claim 1, wherein the presence or absence of the specific gesture is searched for in the ear-side search area, and whether the mouth movement corresponds to the preset mouth in the mouth comparison area The step of determining whether the person is using the handheld device comprises: obtaining at least one mouth image, and obtaining a plurality of mouth features according to the at least one mouth image; determining the at least one according to the mouth features The mouth image is an open motion image or a closed motion image; in the mouth recording time, all the closed motion images or the open motion images detected in the mouth comparison region are sequentially recorded and Converting to the code sequence; and storing the code sequence in the mouth motion information. The method of claim 1, wherein the presence or absence of the specific gesture is searched for in the ear-side search area, and whether the mouth movement corresponds to the preset mouth in the mouth comparison area The information to determine whether the person is using the handheld device includes: comparing the image of the mouth comparison area with the plurality of template images during a mouth comparison time to generate a mouth type code And storing the mouth type code in the code sequence; and storing the code sequence in the mouth motion information. A device for detecting a person using a handheld device includes: an image capturing unit that captures at least one image of a person; a storage unit that stores the image and a preset mouth information; and a processing unit coupled to The storage unit is configured to obtain the image, wherein the processing unit analyzes the at least one image to obtain a facial object, and the processing unit determines at least one of the ear side search area and a mouth comparison area according to the facial object, and borrows Determining the presence or absence of a specific gesture in the at least one ear side search area, and determining whether the mouth motion information detected in the mouth comparison area matches the preset mouth information Whether a handheld device is being used, wherein the processing unit detects the mouth matching area to generate a code sequence, and determines whether the mouth motion information conforms to the preset mouth information according to the code sequence. The apparatus of claim 8, wherein the processing unit detects whether a hand item of the person is present in the at least one ear side search area, and obtains at least one finger feature point of the hand item, wherein the at least one The one-finger feature point includes a foreground edge and at least one gradient value, and the processing unit determines whether the specific gesture exists if the at least one finger feature point is present. The device of claim 8, wherein the processing unit determines a number of hand-held events that the person is using the handheld device and detects a number of facial events of the face object within a predetermined time period. Whether the ratio is greater than a proportional threshold. The device of claim 8, wherein the at least one ear side The search area includes a first ear area and a second ear area, and the processing unit detects a shoulder feature of the at least one image according to the facial object to obtain a shoulder object, and the processing unit is configured according to the The shoulder object and the facial object determine the ear side search area, and filter an interference feature in the ear side search area to detect the first ear area and the second ear area. The device of claim 8, wherein the processing unit identifies a nostril positioning point from the facial object, and the mouth searching area is set based on the nostril positioning point, and the image of the mouth searching area is performed. An image processing is performed to determine one of the mouthpieces of the person, and a mouth region is taken out from the mouth portion in accordance with the mouthpiece. The apparatus of claim 8, wherein the processing unit obtains at least one mouth image through the image capturing unit, and obtains a plurality of mouth features according to the at least one mouth image, the processing unit according to the mouths The feature is used to determine that the at least one mouth image is an open motion image or a closed motion image. During a mouth recording time, the processing unit sequentially records all the detected objects in the mouth comparison region. The motion image or the motion image is closed and converted into the code sequence, and the code sequence is stored in the mouth motion information. The device of claim 8, wherein the processing unit compares the image of the mouth comparison area with the plurality of template images during a mouth comparison time to generate a mouth type code, The mouth type code is stored in the code sequence, and the code sequence is stored in the mouth motion information.