TW201227469A - Implement methods of an interactive multi-media bathroom mirror touch display system - Google Patents

Abstract

In the traditional bathroom mirror, we establish an interactive embedded multimedia man-machine interface to communicate with the outside world. People will have a pleasant bath time that promotes the economic value of the traditional mirror. Implementation methods of the system are to add the multimedia creativity designs and the improvements on the ordinary mirror, then providing services of time information, multimedia player, calendar information, online weather, news retrieval, automatic message boards via face image recognition and so on. In addition, the improved man-machine interface technologies including real-time tracking and recognition of the gesture and face images are designed. With the network monitoring, the system display bath environment control and warning information at any time so that people can enjoy the comfortable, convenient and safe multimedia and interactive digital life in the bathroom. Implementation of the present invention includes the following methods: 1. Improve the bathroom mirror to have a touch display, interaction, monitoring, multimedia functions. 2. To reduce the influence of the lamp luminous change, the posture change and the face expression change, the improved algorithm of real-time tracking and recognition of gesture and face image is proposed. 3. Automatic message boards via face image recognition 4. Using 1, 2, 3 improvements, the Linux operating system, OpenCV image processing platform and QT multimedia human-machine interface platform are built on the ARM-11 embedded system to realize an interactive multimedia bathroom mirror (called the magic mirror) production.

Description

201227469 VI. Description of the invention: [Technical field of the invention] The invention includes an embedded interactive mirror type bathroom mirror, an improved algorithm for real-time gesture face image tracking recognition, and an automatic message version of image recognition. With the production of interactive multi-body bathroom mirrors, _ for the bathroom, dressing table, Lin environment. Before using these three technologies, we will build a u-face job, a (10) image processing platform and a QT multimedia human-machine interface platform to realize the production and application of interactive multimedia bathroom mirrors. [Prior Art] Traditional bathroom mirrors - as many as basic functions such as washing and esthetics, due to the rise of the concept of digital home life, 'there are some new features, and currently related products such as Japan and Korea have related mirror display systems ( Mirr〇r TV, ViVi Mirror) and digital signage are available; bathroom mirror touchdown products' _ brothel development gestures interactive billboards and revelation and large interactive display framework, the function of these new products, or lack of integrity Multimedia features, or no image recognition from the _ words or no complete instant gesture face image tracking and recognition. The "Interactive Multimedia Guarding Mirror Display System" mentioned in this month's case contains interactive multimedia human-machine interface display mirrors, with touch display, interaction, monitoring, multimedia and other functions, and automatically records messages via image recognition. 'And improve (4) gestures and face image tracking and recognition to achieve a good human-machine interface; the fourth sub-item of the invention towel is based on the three technologies in the Embedded ARM_11 mirror display system, including providing time to check 9, multimedia video playback , calendar inquiry, network weather, news retrieval, automatic message recognition, face and face tracking and recognition, display of control and warning information of the bathroom environment. SUMMARY OF THE INVENTION The realization of the new interactive multimedia bathroom mirror touch display system includes: 1. The design of the new bathroom interactive touch display mirror, the general mirror becomes a touch display, interaction, monitoring, multimedia function. 201227469 2. A new algorithm for improving the real-time tracking recognition of gestures and face images, reducing the effects of changes in lighting brightness, posture changes and expression changes. 3. Automatic message board design via image recognition. 4. Using 1, 2, 3 improvements, constructing a 111^ operating system on the arm-h embedded system,

The OpenCV image processing platform and the Kenting multimedia human-machine interface platform enable the production of interactive multimedia bathroom mirrors (referred to as magic mirrors). Figure 1 is a schematic diagram of the system and function deployment diagram. Each sub-item is described in detail as follows:

1. New bathroom interactive touch display mirror design: The new bathroom interactive display mirror design, usually a mirror, after startup, becomes a multimedia information display panel. The whole (10) system specification includes ARM_U Μα^ controller core, 15"~30 or more semi-reflective mirror (mirr〇r) capacitive touch range Zhengqiu 8〇2.l5.4 (Zigbee) wireless communication transmission interface specification The mirror surface is made of glass (the thickness of the phase is 3mm) and the unidirectional light-transmissive coating (non-conducting) is applied to form a semi-reflective mirror surface. The characteristics of the ^-t-mirror are used in the mirror-side panel-surface panel, #镜后_ When the display panel is closed, since the rear of the mirror is dark, the overall device looks like a general mirror. When the mirror = square display panel is turned on, you can immediately see the __ private = reduced plate projection capacitive sugar field. The application of the capacitive touch panel consists of three parts: the touch panel, the touch controller and the driver, which have the advantages of degree, controllability and long life. The structure of the capacitive touch panel and the user relationship are as follows - ZlgBee technology is a short-distance, simple architecture, low-transmission rate wireless communication with power saving. Control (4) (4) contains touch screen: ARM-11 and Linux operating system, with multimedia micro bm_ed, news, weather, acting Calendar, message, _ Sense, Touch 2 screen and multimedia application as shown in Figure 3, ARm_ = system design architecture as shown in Figure 4. Soil hard & 遐 遐 control 2 · gestures and face image instant tracking recognition algorithm improved in the bathroom, job screen _ Read 'but sometimes it is easy to operate, if you can use gestures and 201227469 face image recognition will be more in line with the human-machine interface. At present, such image recognition, the actual application is about three to seventy percent 'how to improve its recognition rate and apply to this Design is one of the key points. Use the most economical Web-Camera to capture gestures and face images, use 〇penCV for image processing, use Adboosting to train classifiers and Haar-likeTransform, improved image pre-processing and improved PCA (Component Analysis The analysis method performs gesture and face recognition on the PC side and the ARM-11 board end respectively. (a) Gesture recognition part:

After the system uses the Webcam camera to obtain the image, the original color image is grayed out and the histogram is equalized (Equalized), and the Adaboosting-Adaptive algorithm is used to perform image detection on the mirror system platform. The image of the hand obtained by the rectangular frame is normalized as shown in Fig. 5. Through the meaning of the user's gestures, the action can be performed on the mirror. It is an instant gesture detection system as shown in Figure 6. The gesture detection and recognition process is shown in Figure 7.曰^ In other systems, the mosquito A gesture symbol, in general, the gesture mode can be divided into static gestures and dynamic gestures. Static gesture refers to the shape of the hand presented at a certain time, that is, considering only the spatiality of the hand, it can be distinguished by the state of the thumb, other fingers, and the palm, which is prone to malfunction. Dynamic gesture refers to the change of the shape in the continuous time, that is, not only the spatiality of the hand, but also the temporality, and it is not easy to misjudge 'but its recognition is more complex than the initial gesture, and finally The gestures and the trajectories of potential movements are distinguished. This algorithm is to make the bribe age, the rate can reach 90%. (b) Face recognition part: Face recognition is "the mutual screaming of the age, the swearing shirt is the green member considering the bathroom environment and the platform Guan, change # existing algorithm, propose the outline _ face _ and improved pcA feature record recognition" training and testing two stages, mainly for facial system, special reliance, its entire face (four) measurement and identification system as shown in Figure In the face _ square Γ & AdaBoost algorithm to detect facial images, its debt measurement flow chart is shown as round nine, _ before 201227469 using the Hist〇gram Equalization to improve the Detective Measure the quality, and then use the second-order horizontal and vertical projection to find the detected face area, as shown in Figure 10 (a). Feature extraction and facial compliance. Traditional pCA has been used, but there are still some shortcomings. It is easy to be affected by the brightness of the light, the change of face posture, the change of expression, etc. In this case, several improvement methods are proposed. When the face area is detected, the face is taken with a 1:1.2 rectangular frame, and the obtained face is re-executed for size. Normalization, as shown in Figure 10 (9), and improve the use of the eye center line, the face image is up Lower and tilt adjustment image positioning, as shown in Figure 10...Face feature area 'domain selection, considering the uniqueness and importance of personal characteristics, improved to select "full face, with eye, nose, mouth, face local features four Part of the identification as shown in Figure 12. From the experimental analysis of Figure 2 in Figure 22, the results are selected HSI (Hue, Saturati〇n, Intensit^ and .. Gray image (Gray) image as the face reference image recognition rate is better, grayscale image is 76= The Η is the highest 84%, so all the images in this case are based on the two analysis. The results of the analysis under the fixed light source, the recognition rate of the ^pcA in Figure 21, which selects the face, eyes, nose, mouth The personal characteristics are unique and adjust their weight to the influence of pCA. As a result, the recognition rate of WebCam is 87% and the average photo is 96%, which is better than the traditional 3%^ = 82% and 93%. Next, the recognition rate of the grayscale images (Gray), Η, S, and 丨 of the lion (5) image is shown in Figure 2 of Figure 22. The gray level is 76% and the Η is 84〇/〇 is the most. Good, the recognition rate of general pCA is close to fixed light source (82%). Further, in order to reduce the influence of light changes, we propose an EMSR improvement algorithm 'adjusting the influence of light source' using the histogram method (Equalized) and MSR (Mu 丨Shi Re|mex) feature enhancement method (referred to as EMSR in this case), improving the change of light intensity caused by p, picking The results of the three experiments show that the three strong different filaments are processed by the improved EMSR (EqUa ized MSR) algorithm in this case, and the ship brightness is one, °=Retmex algorithm uses Gaussian function to image The chopper of the convolution operation is divided into SSR (Single Scale Retinex), MSR (MuIti-Scale Retinex) and EMSR to improve the light sensitivity. EMSR is the method proposed in this case to improve msr. The image must be made directly. The graph is equalized (Equalized). The formula is as follows: SSR·· R = log I (Χ}γ) . log[F(x,y)<S)/ MSR: R^y) =Σΐν^(χ,γ) k^\ 201227469

EMSR. Histogram Equalization (Preprocessing) + MSR Next, the sample is based on the whole face and the "eye, nose, mouth" weighting PCA identification method for the 542 example (as shown in Figure 5, 5 cases) The results of training and case identification experiment are shown in Figure 3 of Figure 23. In the case of case 1, the training examples and the identification of ten cases are fixed with the same light source, and the recognition rate is 96〇/. However, cases 2 to 5 are variable light sources, and the recognition rate is 63% to 91% (mostly 85% or more), and the effect is better than before. Figure 24 of Figure 24 shows that the training and identification are compared and compared by different methods. As a result, the training is identified by EMSR training. When the EMSR is used for recognition, the recognition rate is 85.3%, which is much better than other methods. The improved algorithm ® has been implemented on PC and Embedded ARM-11, using family members as shown in Figure 15. Continuous use of WebCam for 30 photos at 30 price, the first one is used as a training sample, and the first one is used as a test sample. According to image pre-processing, image positioning, and facial feature extraction steps. The training results of the first 1 Zhang Zhang training samples, Eigenface, Eigeneyes, Eigennose, and Eigenm〇uth are shown in Figure 16. The test results can be detected and identified immediately. From the previous experimental results, it was found that the EMSR was used to adjust the variation of the light source, and then the PCA of the weighting was given according to the degree of influence of the characteristic. The identification effect of the method was proved to be significantly improved compared with the traditional method. • The concept of the PCA algorithm is to extract the main artifact image (Eigenface) coexisting in different human face images, and use Eigenface to simplify the representation of the restored face image, which is one of the most effective methods for face recognition. Since PCA is a feature of dimensionality reduction, it consumes less memory and has a faster recognition speed. Through the training example, the eigenvector of the common variable matrix of the sample data is calculated, the feature face Eigenface is obtained, and the feature vector of the larger eigen value is selected to constitute the Eigenspace space. Then the eigenvectors of all the tested examples are compared in Eigenspace. The closer the Euclidean distance is, the more similar the figure is. Figure 14 illustrates the PCA algorithm. 3. Automatic message board design through image recognition 201227469 Since the bathroom is the most common place for family members, it is most suitable for own camera, touch «screen, chase axis _ Wei 2 side... because of reading and smoothing the user, then The section is left in the _ front, the entire message contains the message and the message image - it is stored in the image. Retaining: As long as the message is automatically recognized before the screen, it is a convenient communication tool in the family. The system flow is shown in Figure 17.

4. The production and realization of the interactive multi-ship man-machine interface bathroom mirror (simplified mirror) With 1, 2, 3 improvements, we have built the operating system, OpenCV image processing platform and multimedia on the fine rhyme system. Human-machine interface platform to realize the production of interactive multimedia bathroom hiding. Guardian-reduced job-type person, township media, image recognition, network communication and human-machine interface integration, mainly including ARMU embedded system, mirror, Internet, multimedia applications, touch screen, gesture and face recognition, etc. interface. Multimedia features such as home security monitoring displays, audio and video playback, news, weather, calendars, messages and wireless Internet access. The soft and hard system of the mirror is shown in Figure 18. The main process and multimedia interface flow of the human-machine interface are shown in Figure 19. The application in the bathroom is shown in Figure 20. [Embodiment] The implementation part includes: 1. Interactive display touch mirror design; 2. ARM-11 circuit design and PC board production; 3. Linux job porting and Driver design; 4. Magic mirror multimedia man-machine interface design ; 5. Application in the bathroom. 1. Interactive display touch mirror design: Turn the general mirror into touch display, interactive, monitoring, multimedia functions. 201227469 2. ARM-11 circuit design and pc board production: The steps and processes can be divided into five stages, from hardware circuit design to PCB layout, PCB production, PCB SMT to the final system Debug. The hardware is divided into two upper and lower PCB boards, one is the peripheral bottom board, and the other is the cpu module board. Doing so can facilitate debugging. Both can effectively control costs. The cpu^ board printed circuit board uses a 6-layer board, and the peripheral board part adopts a two-layer board design, which reduces the overall cost. 3. Linux job porting and Driver design

The migration of Unux jobs includes the booting of the bootloader, kernel (kernel), Filesystem, development tools, API application 'qT and 〇penCV human-machine interface development platform. 4. Mirror multimedia man-machine interface design and application in the bathroom Mirror multimedia function design includes video playback, news, weather, calendar, message, Internet and supervision (4) tf, etc. 'immediate gestures and face speculation touch silk Design, automatic message board design for image recognition. 5. Application in the bathroom: physical testing.

[Simple diagram of the diagram] The first diagram, the second diagram, the third diagram, the fourth diagram, the fifth diagram, the sixth diagram, the seventh diagram, the eighth diagram, the ninth diagram, the tenth diagram, the system and the function deployment diagram, the embedded capacitors Illustrator (8) Capacitive touch screen and (9) Multimedia display application Touch screen ARM-li control system architecture Detects palm image Instant gesture detection system Gesture detection and recognition process Face detection and recognition process AdaBoost Face detection flow chart shows (8)-level face vertical and vertical projection; frame face and normalization 201227469 Η-图: face tilt adjustment twelfth picture: full face and eyes, nose, mouth The local feature area of the face is taken from the thirteenth picture: EMSR feature enhancement method to improve the instability of light intensity. Fourteenth figure: The fifteenth figure of PCA algorithm: Family member test sample tenth _/, figure · eigenface (Eigenface) Eigeneyes, Eigenmouth, Eigennose Figure 17: Image recognition automatic message board process Figure 18 Mirror body (4) Side bevel plus frame touch mirror; (b) Front ( Including multimedia touch mirror (c) Side Circuit Figure 19: Mirror Multimedia Human Machine Interface (a) Main Process; Main Menu; (〇 选 News 20th Illustrator: Application in Bathroom 21st Chart (Figure 1): Under fixed light source, the general full face PCA 'improves the face and nose of the nose, the PCA recognition rate of the face and nose mouth with the specific gravity is the twenty-second chart (Figure 2): the fixed light source, the general pCA pair (Gray), H recognition rate The twenty-third figure (Figure 3): fixed light source, improved PCA for Gray, Gray, EMSR recognition rate R, twenty-fourth figure (Figure 4): Gray, H, MSR, EMSR Comparison identification rate [main component symbol description] 7. Patent application scope: 1. A new bathroom interactive display mirror design, this improved mirror is a mirror but controls display, interaction, monitoring, multimedia functions. ~~ ' Usually Mirror, after startup, becomes a multimedia information display panel with image processing and

Claims (1)

201227469 Η Η 图 图 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸 脸: PCA algorithm fifteenth figure: Family member test sample tenth _ /, figure · Eigenface, Eigeneyes, Eigenmouth, Eigennose Figure 17: Image recognition Automatic message board process, the eighteenth mirror mirror body (4) side bevel plus frame touch mirror; (b) front (including) multimedia touch mirror; (c) side circuit nineteenth: magic mirror multimedia man-machine interface (a The main process; the main menu; (〇 选 election news twentieth: the application of the bathroom in the twenty-first chart (chart 1): fixed light source, generally full face PCA 'improve the face and nose, add weight to the face The PCA recognition rate of the nose and mouth is the twenty-second chart (Figure 2): the fixed light source, the general pCA pair (Gray), the recognition rate of H, the twenty-third figure (Figure 3): fixed light source, improved PCA pair Gray, Η EMSR recognition rate R, twenty-fourth figure (chart ): Gray, H, MSR, EMSR intersection and comparison identification rate [main component symbol description] Seven, the scope of application for patents: 1. A new bathroom interactive display mirror design, this improved mirror is a mirror but control display, interaction , monitoring, multimedia functions. ~~ ' Usually it is a mirror. After startup, it becomes a multimedia information display panel with image processing and 201227469. Multimedia functions such as audio and video playback, news, weather, action layer, Liugu, Internet and monitoring display, etc. 2. A new algorithm for improving the real-time tracking recognition of gestures and face images, reducing the effects of light changes, posture changes and expression changes. Its functions include: (1) Gesture part: refine the color image before system And the direct shed is equalized, after the instant detection of the image is obtained, the symbolic action of the gesture is defined by the symbol of the user's dynamic gesture, and the spatial and temporal characteristics of the gesture are considered (Ding Qing (4). The symbol of the dynamic gesture action The definition is differentiated by the initial gesture, the final gesture and the trajectory of the gesture. (2) Face recognition part: _ before Use the grayscale histogram to equalize the EquaHzation to improve the detection quality, and then use the eight her_ algorithm to detect the facial image. When the face area is detected, the face is obtained with a rectangle of 1 · 1.2. The face will be re-executed as a normalization of the size 'and use the eye center line to make the image of the face up and down and the tilt is adjusted. 'Select the whole face, and the importance of personal characteristics uniqueness and importance. The four parts of the face, the nose, the mouth and the mouth are used as the reference for the identification. The singularity of the Saturation, Intensity and the general grayscale image are used as the facial reference image. With _R (Equalized Multi_seaie Retinex method, the strength of the Wei line is unstable, and then the "full face" and "eye, nose, mouth, and importance" weighting PCA identification method is used for identification. 3. An automatic message board design via image recognition. Included · This new automatic message board design uses the camera to manipulate the user's image and automatically recognizes and records the user's message on the touch screen. The message is stored in the image picture mode, and the sever is received, as long as it is recognized before the screen to automatically get the message. 12 201227469 4. The realization and production of the interactive multimedia bathroom mirror (referred to as the magic mirror). This includes: = 1,2'3 Improvement, the construction of the Linux operating system, OpenCV image processing platform and QT multimedia human-machine interface platform on the ARM·11 embedded system, the design and production of the interactive multimedia bathroom mirror system copyright. The system works mainly include ARM11 embedded system , mirror, Internet, multimedia applications, control screens, gestures and face recognition and other human-machine interface integration. Multimedia features such as Display control, audio and video playback, news, weather, acts layer, message and wireless Internet access knot