CN102750555A - Expression robot applied to instant messaging tool - Google Patents

Abstract

The invention discloses an emoticon robot applied to an instant messaging tool, which can recognize emoticons used by users in real time, and respond with the same meaning as the emoticons represent. Moreover, the expression recognition method avoids the defects of high complexity, poor real-time performance, and the need to re-decipher the key after the upgrade of the instant messaging tool. In this system, the chat window monitoring module intercepts the image of the chat window as a picture when the instant messaging tool has a new chat message to display or regularly saves it as a picture; The position is sent to the emoticon recognition module; the emoticon recognition module compares the position of the emoticon with the emoticons in the known emoticon library, determines the meaning of the emoticon, and sends it to the response module; the response module receives the emoji recognition After the result sent by the module, it responds with the set expression method.

Description

An expression robot applied to instant messaging tools

technical field

The invention is an expression robot based on an instant messaging tool, and specifically relates to the technical fields of screen capture and image recognition.

Background technique

Instant messaging tool, also known as instant messaging tool, is an Internet-based service. After more than ten years of development, it now has a large number of regular users and penetrates into every aspect of our life and work. Emoji was born on the Internet. It was originally just an Internet subculture, but with the rapid development and popularization of the Internet, it has been widely accepted by people.

General instant messaging tools have the function of inserting emoticons, which greatly facilitates the expression of users and enhances the fun of communication and user experience. At present, emoticons are mainly developing in a more diverse, vivid and complex direction. The present invention proposes a creative development path: making emoticons more realistic. The solution can monitor the window interface of the instant messaging tool, and then recognize the meaning of the emoticon used by the user, and respond to the recognized emoticon according to the meaning of the emoticon. Through the emoticon robot, the user can more intuitively feel the emoticons used in the chat process, thereby increasing the experience and fun of the instant messaging user. This is an innovative product with little reference to previous experience.

When designing the emoticon recognition scheme, the first thing that comes to mind is to decrypt and analyze the data packets transmitted by the instant messaging tool, and extract the code of the emoticon from it, so as to realize the recognition. However, the current instant messaging tools will perform complex encryption on the chat content during the transmission process, and it will take a long time to decipher the key, which is difficult, and the instant messaging tools are usually upgraded from time to time, and each upgrade will be restarted. Set the key, so the defect that the instant messaging tool needs to decipher the key again after the upgrade, which brings about the defect that the program needs to continuously decipher the key, the complexity is high, and the real-time performance is not good, so it cannot be mass-produced and promote.

Contents of the invention

In view of this, the present invention provides an emoticon robot applied to instant messaging tools, which can recognize emoticons used by users in real time, and respond with the same meaning as the emoticons represent. Moreover, its expression recognition method does not use the method of decrypting and analyzing the data packets transmitted by the instant messaging tool, which avoids the high complexity, poor real-time performance, and the need to re-decipher the password after the instant messaging tool is upgraded. key defect.

The method is implemented like this:

An emoticon robot based on an instant messaging tool, including a chat window monitoring module, an emoticon positioning module, an emoticon recognition module and a response module;

The chat window monitoring module is used to monitor the chat window after determining that the current focus window is the chat window of the instant messaging tool, and intercept the image of the chat window regularly or when a new chat message is displayed by the user down, and save as a picture;

The emoticon positioning module is used to analyze the pictures intercepted by the chat window monitoring module, find the emoticons therein, and after finding the emoticons, send the position of the emoticons to the emoticon recognition module;

The emoticon recognition module is used to compare the emoticon at the position with the emoticon in the existing emoticon library after receiving the emoticon position sent by the emoticon positioning module, so as to determine the emoticon in the emoticon. The meaning of the representative, and then send the result to the response module;

The response module is used for responding through the set expression method after receiving the result sent by the emoticon recognition module.

Wherein, the response module responds through sound, image and/or action.

Preferably, the emoticon positioning module includes a cutting module, a graying module, a smoothing module and a Hough Hough detection module:

The cutting module is used to cut out the image of the dialogue bar from the image of the chat window according to the position of the dialogue bar in the chat window, and send the cut image to the grayscale module;

The grayscale module is used to grayscale the received image to obtain a grayscale image and send it to the smoothing module;

A smoothing module is used to perform Gaussian smoothing on the received grayscale image;

The Hough detection module is used to carry out Hough transform to the image after Gaussian smoothing, and detects the position of the circular emoticon; only outputs the position where coordinates x and y are the largest emoticon in the emoticon recognized; The x-axis and y-axis take the upper left corner of the dialogue bar as the zero point. Preferably, the Hough transform of the Hough detection module starts from the lower right corner of the dialogue column, and is calculated in the order from right to left and from bottom to top. When the first circle is detected, the position of the center of the circle is output, and the detection is stopped.

Preferably, the emoticon recognition module includes an extraction module, a matching module and a recognition module;

The extraction module is used to obtain the emoticon image from the image of the dialogue column obtained by the cutting module according to the position output by the emoticon positioning module, and send it to the matching module;

The matching module is used to use the received emoticon image as a template, and pre-stored images containing all default emoticons as a global image, then perform template matching, and finally find out the position of the emoticon image in the global image;

The recognition module is used to judge which expression meaning corresponds to the position of the emoticon image found by the matching module according to the position range and expression meaning of each default emoticon in the known global image, and send the expression meaning to the response module.

Beneficial effect:

The invention provides an emoticon robot applied to an instant messaging tool, which can recognize the emoticon used by the user in real time, and make a response that is the same as the meaning represented by the emoticon, and more importantly, the emoticon and its The facial expression recognition method does not adopt the method of decrypting and analyzing the data packets transmitted by the instant messaging tool. After the instant messaging tool is upgraded, it does not need to decipher the key again, so it has good versatility and is suitable for various instant messaging tools. Moreover, this method relies on screenshots for facial expression recognition, which is simple and can be implemented with the help of many existing image processing methods, and the calculation amount is not large, which is conducive to improving the real-time performance of facial expression recognition.

Secondly, after Hough circle detection, multiple circles at different positions may be obtained. According to the characteristics of the instant messaging tool, only the position of the circle with the largest x and y is output, thus avoiding subsequent matching calculations caused by the output of multiple circle positions Large amount, response confusion and other defects.

Again, when performing Hough detection calculations, start calculations directly from the lower right corner of the dialog box, and output the first detected circle position in the order from right to left, bottom to top, and do not need to perform subsequent calculations. This reduces computation.

Description of drawings

Fig. 1 is a system functional block diagram of the present invention.

Figure 2 is a flowchart of the screenshot module.

Figure 3 is the effect diagram of the screenshot module.

Figure 4 is a QQ default emoticon map.

Fig. 5 is a flow chart of emoticon location module processing.

Figure 6 is an exploded view of the QQ chat window.

Figure 7 is an effect diagram after grayscale.

Figure 8 is the effect diagram after Gaussian smoothing.

Fig. 9 is a diagram of emoticon positioning results.

Figure 10 is a flow chart of the emoticon recognition module.

Figure 11 is a division diagram of all default emoticons in QQ.

Fig. 12 is a graph of emoticon recognition results.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and examples.

The present invention adopts a method that does not need to decrypt and analyze the chat content, and analyzes the emoticons by monitoring the screenshots of the chat window, which is called the comparison method of screenshots.

The idea of this method is to first monitor the chat window of the user of the instant messaging tool, then select an appropriate time to take a screenshot and save it, then analyze and compare the images of the chat window to find out the position of the emoticon, and finally recognize the emoticon the meaning represented. Since the size of the emoticon is fixed, it can be realized by adopting the scheme of template matching in pattern recognition.

The screenshot comparison method has the advantage of bypassing the link of deciphering encrypted messages, so the method is simple, has good real-time performance, and can be easily implemented across instant messaging tools.

Fig. 1 is a composition block diagram of an emoticon robot of an instant messaging tool of the present invention. As shown in Fig. 1, the emoticon robot includes a chat window monitoring module, an emoticon positioning module, an emoticon recognition module and a response module. in,

The chat window monitoring module is used to monitor the chat window after determining that the current focus window is the chat window of the instant messaging tool, and to intercept the image of the chat window at regular intervals or when a new chat message is displayed by the user, And save it as a picture; wherein, the focus window refers to the window that the user is currently operating, and the focus window can be easily judged through the API function under Windows.

The emoticon positioning module is used to analyze the pictures intercepted by the chat window monitoring module, find emoticons therein, and after finding emoticons, send the position of emoticons to the emoticon recognition module;

The emoticon recognition module is used to compare the emoticon at the position with the emoticon in the existing emoticon library after receiving the position of the emoticon sent by the emoticon positioning module, so as to determine the emoticon represented by the emoticon meaning, and then send the result to the response module;

The response module is used for responding through the set expression method after receiving the result sent by the emoticon recognition module.

The function of each module is described in detail below.

●Chat window monitoring module

The main function of the chat window monitoring screenshot module is to monitor the user's chat window. When the user has a new chat message displayed, the chat window will be intercepted and saved as a picture for later processing and analysis. Chat window The workflow of the monitoring module is shown in Figure 2.

Taking Tencent's QQ as an example, the general process of the chat window monitoring module is as follows. When the system is running, it first judges whether the user's current focus window is a QQ chat window. If it is, it will enter the next step. If not, it will not respond. After it has been determined that the user's focus window is the QQ chat window, it is necessary to judge whether there is a new message entering, and if so, take a screenshot and save it. Or you can use the method of timing screenshots, and take screenshots of the chat window every fixed time. Whether new chat messages are added can be realized by monitoring the change of the image on the chat window. When the image in the chat window changes, it is considered that a new chat message has entered. The effect of the screenshot module is shown in Figure 3.

●Emoji positioning module

The screenshot of the QQ chat window is obtained through the chat window monitoring module. The next problem is how to analyze whether the captured picture contains emoticons, and how to determine the location of the emoticons.

First, analyze the default emoticons that come with QQ, as shown in Figure 4. For practical and technical reasons, this system currently only considers supporting the circular classic emoticons in QQ default emoticons. This series of emoji has two very prominent features:

1) It is circular in shape.

2) Both use yellow as the main color.

Considering that users mainly use Chinese characters, English letters, numbers, and emoticons when chatting, the first three items are interference items in the recognition process and do not have the circular shape of emoticons, but QQ users can customize chat text color, so it may also have the characteristic of yellow. After comprehensive consideration, it was decided to use the feature of circle to mark and extract emoji, which is also the feature of emoji used by most instant messaging tools.

Previously analyzed the characteristics of emoticons that are different from Chinese characters, English letters, and numbers, here we need to use some image processing techniques to find circular emoticons, as shown in Figure 5, first of all, we need to cut the image, remove Unnecessary interference areas can improve the efficiency of subsequent processing. After cutting, the image is grayscaled and image smoothed. The purpose of these two processes is to enhance the image in order to improve the accuracy of detection. Finally, it is to detect emoticons. Use When it comes to the Hough circle detection function, the result is to return the position of the emoji, that is, the coordinates of the circle.

Therefore, the emoticon localization module includes a cutting module, a graying module, a smoothing module and a Hough detection module.

The cutting module is configured to cut the image of the dialogue bar from the image of the chat window according to the position of the dialogue bar in the chat window, and send the cut image to the grayscale module.

Still taking QQ as an example, the screenshot of the QQ chat window is shown in Figure 6. First, let’s analyze the input screenshot. You can see that the QQ chat window can be divided into four areas, which are divided by red lines, which are function bars , chat record display window, input column, QQ show column, obviously what is needed in this system is the information of the chat record display window, and emoticons will not appear in the other three areas. Therefore, the purpose of cutting this step is to separate the chat record display window from the encirclement of the other three areas. Through experiments, I found that although the size of the QQ chat window can be changed according to the user's needs, the height of the function bar: " H _function ", the height of the input column: "H _input " and the width of the QQ show column: "W _show ", are a fixed value and will not change with the zoom of the window. After measuring H _function = 105 pixels, H _input = 155 pixels, W _show = 145 pixels. According to these three data, it is easy to deduce that the starting coordinates of the chat record display window are (0, 105), and the coordinates of the lower right corner are (W-145, H-155), where W is the width of the entire screenshot, and H is the entire The height of the screenshot. Through these data, it is easy to display the chat record display window separately, which eliminates a large interference area and makes subsequent processing more efficient.

The grayscale module is used to grayscale the received image to obtain a grayscale image and send it to the smoothing module.

Grayscaling is the conversion of a color image into a grayscale image. The image acquired by the screenshot module in this system is a color image. A color image is such an image that each pixel corresponds to three components (R, G, B), and these three components represent red and green. , blue, each component can take an integer value from 0 to 255, each group of values represents a color, such a pixel can have more than 16 million color values to choose from, even for a computer The numbers are also very large, so for the convenience of processing, it is necessary to grayscale the image. A grayscale image is such an image, and its pixels are also represented by three components of R, G, and B, but the three components of these three components The value is the same, it can be seen that the value range of a pixel in the grayscale image is only 255, and it can still reflect the distribution and characteristics of the overall and local chromaticity and brightness levels of the entire image just like the color image. Therefore, when processing digital images, the object to be processed is generally converted into a grayscale image first, which can greatly reduce the amount of calculation for subsequent processing. The grayscale result is shown in Figure 7.

The smoothing module is used to perform Gaussian smoothing on the received grayscale image.

The main purpose of image smoothing is to eliminate the noise in the original image and keep the edge contours and lines of the original image as much as possible. The noise in the image is not limited to the image distortion and deformation that can be perceived by human eyes. Many noises can only be found in computer image processing, and these noises are randomly distributed, and the size and shape are also irregular. . There are many methods of image smoothing, including linear and nonlinear smoothing and sharpening, false color processing, filtering, etc. When Gaussian smoothing is used to remove chaotic and randomly distributed noise, it will receive better results than other smoothing methods. The most important point is that it can get better image edges, so that it can be used in subsequent circle detection. Nice precision. Therefore, Gaussian smoothing is performed on the input image, and the effect after processing is shown in Figure 8.

The image after Gaussian smoothing is a denoised grayscale image, which already has the prerequisites for circle detection. When it is necessary to recognize simple geometric patterns from images in image processing, there is a basic method called Hough transform. Hough transform is a basic and effective method for edge detection. One of the basic methods for identifying simple geometric shapes. The basic principle of Hough transform is to use the duality of point and line in mathematics to transform the selected point in the original image space into a curve or surface in the parameter space, and the points with the same parameter characteristics are transformed in the parameter space Intersect, so that the detection of the characteristic curve can be completed by judging the degree of accumulation at the intersection point, that is, the peak value. Using the Hough transform, the problem of curve detection in the original image can be transformed into the problem of finding the peak in the parameter space, that is, the detection of the overall characteristics can be transformed into the detection of local characteristics. Based on the different properties of the parameters, the Hough transform can detect straight lines, circles, ellipses, curves, etc.

Hough transform is a global detection method, it is insensitive to random noise and partial cover phenomenon, has strong anti-interference ability, and can well suppress the interference caused by too concentrated data points. Hough transform has good fault tolerance and robustness in detecting objects with known shapes, even if the objects are damaged or polluted, they can be correctly identified. So the present invention adopts this method to detect the circle of the emoticon. Since the radius of the emoticon is known to be about 25 pixels, the emoticon can be found and separated as long as the center of the circle is located, and the effect is shown in Figure 9 .

In order to test the effectiveness and anti-interference ability of this method, the main interference items such as numbers, English letters, and Chinese characters are included in the screenshots. In order to facilitate the observation of the test results, the test results are output and marked here. Two circles are detected in the result image, and these two circles are exactly emojis. The center of the circle is marked with a green dot, and the coordinates of the center are output. The result is basically the same as expected.

After the positioning module detects a screenshot, there will be a few seconds to wait for the input of the new screenshot. Since this time is very short, the content of the chat history in the new screenshot will partially overlap with the previous screenshot. If If an emoji is contained in the overlapping part, then the emoji has already been located. If the detected result is output at this time, duplication will occur.

The reason for this problem is that the area where QQ displays chat records is fixed. When new chat records arrive, the old chat records will be topped up. If an emoji appears at the bottom of a screenshot, That is, the user has just received the emoticon, then the emoticon will appear in several subsequent screenshots until there are enough messages to push the emoticon out of the display area of the chat history.

Due to the above situation, this module cannot be set to store all the detected emoticons as results for output. In order to solve the problem of repeated positioning, this design is carried out: only the coordinates x and y of the recognized emoticons , are the positions of the largest emoticons for output.

Therefore, the function of the Hough detection module is to perform Hough transformation on the Gaussian smoothed image to detect the position of the circular emoji; only output the position of the largest emoji in the recognized emoji with coordinates x and y ; Among them, the x-axis and y-axis take the upper left corner of the dialogue bar as the zero point. In order to reduce the amount of calculation, preferably, the Hough transform of the Hough detection module starts from the lower right corner of the dialog box, and is calculated in the order from right to left and from bottom to top. When the first circle is detected, the position of the center of the circle is output, and stop detection.

The position of the emoticon can be obtained in the emoticon positioning module, so that the emoticon can be separated separately. But up to this point, the system only knows that there is an emoji in the chat window. It does not know the meaning of the emoji, so it cannot make a corresponding response. It is necessary to identify the meanings represented by the separated emoticons, and this process is the function that the emoticon recognition module can realize.

●Emoji recognition module

The main idea of the emoticon recognition module is to use the emoticon image acquired by the positioning module as a template, and use the image containing all default emoticons as a global image, then perform template matching, and finally find out the position of the emoticon in the global image. All emoticons are included in the global image and each emoticon has its own fixed position, so the meaning of the emoticon can be determined according to the position. The basic flow of this module is shown in Figure 10.

Therefore emoticon recognition module comprises extraction module, matching module and identification module; Wherein,

The extraction module is used to obtain the emoticon image from the image of the dialogue column obtained by the cutting module according to the position output by the emoticon positioning module, and send it to the matching module;

The matching module is used to use the received emoticon image as a template, and use the pre-stored image containing all default emoticons as a global image, then perform template matching, and finally find out the position of the emoticon image in the global image;

The identification module is used to determine which expression meaning the position of the emoticon image found by the matching module corresponds to according to the position range and expression meaning of each default emoticon in the known global image, and send the expression meaning to the response module.

Still taking QQ as an example, Figure 11 is a screenshot of all the default emoticons of QQ.

It can be seen from the figure that there are 8 rows and 14 columns of emoticons, a total of 105 emoticons, each emoticon occupies a fixed position, and has a fixed circumscribed rectangle. After measurement, the area occupied by the emoji rectangle is obtained. is 33×33 pixels, here we can divide Figure 11 into 112 small rectangular areas, and each area represents an emoji. To recognize an emoji, you only need to judge which area the coordinates of the emoji returned in template matching fall in. Yes, for example:

The area of the smiling emoji is 0<x<33, 0<y<33.

The region of curling mouth emoji is 33<x<66, 0<y<33.

The region of the surprised emoji is 0<x<33, 33<y<66.

The region of sad emoji is 33<x<66, 33<y<66.

The matching results are shown in Figure 12. In order to facilitate the observation of the recognition results, the images of the emoticons to be recognized are displayed together with the images of all default emoticons and the output of the recognition results, and the results of the template matching are marked with lines, which can be seen Using this method, template matching can be successfully performed and the meaning of emoticons can be recognized by using the location.

●Response module

The ideal response mode of the response module is accurate and interesting, that is to say, the module should be able to accurately respond to the emoticons, so that users can easily understand the meaning of the response, and require A certain amount of fun can improve the user experience.

The plan for this response module at the beginning of the dissertation research was to find a purpose-built peripheral that should have the following characteristics:

1) It can imitate human faces to make various expressions such as joy, anger, sorrow and joy.

2) It can be connected to the user's host in some way, and communicate with the user's host.

3) It can be programmed to control the peripheral to make different expressions.

The transmission method can be selected according to actual needs. The content of the transmitted data should be designed as follows. In the recognition module, 105 emoticons have been numbered, and the recognition result is the number of the emoticon, so the state of the emoticon robot can also be carried out. Number, this number should be consistent with the number of the emoji. In this way, you only need to send the number to the response module, and then you can make a corresponding expression response, so the content of the transmitted data can be designed as an integer between 1 and 105. When the response module receives the number, it can change to the corresponding state , then the state is exactly the state corresponding to the emoji.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (5)

1. A facial expression robot based on an instant messaging tool, comprising a chat window monitoring module, an emoticon location module, an emoticon recognition module and a response module; The chat window monitoring module is used to monitor the chat window after determining that the current focus window is the chat window of the instant messaging tool, and intercept the image of the chat window regularly or when a new chat message is displayed by the user down, and save as a picture; The emoticon positioning module is used to analyze the pictures intercepted by the chat window monitoring module, find the emoticons therein, and after finding the emoticons, send the position of the emoticons to the emoticon recognition module; The emoticon recognition module is used to compare the emoticon at the position with the emoticon in the known emoticon library after receiving the position of the emoticon sent by the emoticon positioning module, so as to determine the location of the emoticon. The meaning of the representative, and then send the result to the response module; The response module is used for responding through the set expression method after receiving the result sent by the emoticon recognition module. 2. The facial expression robot according to claim 1, wherein the response module responds through sound, image and/or action. 3. expression robot as claimed in claim 1, is characterized in that, described emoticon location module comprises cutting module, graying module, smoothing module and Hough Hough detection module: The cutting module is used to cut out the image of the dialogue bar from the image of the chat window according to the position of the dialogue bar in the chat window, and send the cut image to the grayscale module; The grayscale module is used to grayscale the received image to obtain a grayscale image and send it to the smoothing module; A smoothing module is used to perform Gaussian smoothing on the received grayscale image; The Hough detection module is used to carry out Hough transform to the image after Gaussian smoothing, and detects the position of the circular emoticon; only outputs the position where coordinates x and y are the largest emoticon in the emoticon recognized; wherein The x-axis and y-axis take the upper left corner of the dialogue bar as the zero point. 4. The emoticon robot according to claim 3, wherein the Hough transform of the Hough detection module starts from the lower right corner of the dialogue column, and calculates in the order from right to left and from bottom to top, when the first When a circle is formed, the position of the center of the circle is output and the detection is stopped. 5. the expression robot as claimed in claim 3 or 4, is characterized in that, described emoticon recognition module comprises extraction module, matching module and identification module; The extraction module is used to obtain the emoticon image from the image of the dialogue column obtained by the cutting module according to the position output by the emoticon positioning module, and send it to the matching module; The matching module is used to use the received emoticon image as a template, and pre-stored images containing all default emoticons as a global image, then perform template matching, and finally find out the position of the emoticon image in the global image; The recognition module is used to judge which expression meaning corresponds to the position of the emoticon image found by the matching module according to the position range and expression meaning of each default emoticon in the known global image, and send the expression meaning to the response module.

Images