CN114488838B

CN114488838B - Intelligent home control method, electronic equipment and system based on millimeter wave radar

Info

Publication number: CN114488838B
Application number: CN202210098522.4A
Authority: CN
Inventors: 刘光锋; 熊婷婷; 谢应平
Original assignee: Shenzhen Meizhi Huake Intelligent Technology Research Center Co ltd
Current assignee: Shenzhen Meizhi Huake Intelligent Technology Research Center Co ltd
Priority date: 2022-01-26
Filing date: 2022-01-26
Publication date: 2022-10-04
Anticipated expiration: 2042-01-26
Also published as: CN114488838A

Abstract

The invention discloses an intelligent household control method, electronic equipment and a system based on a millimeter wave radar. Furthermore, compared with the mode of preheating the camera according to the temperature of the camera in the prior art, the method and the device do not need to start the heating device frequently, only needs to preheat in the cooking time period of a user, and reduces the influence on the refrigerating environment in the refrigerator.

Description

Intelligent home control method, electronic equipment and system based on millimeter wave radar

Technical Field

The invention relates to the technical field of smart homes, in particular to a millimeter wave radar-based smart home control method, electronic equipment and system.

Background

In order to realize intelligent home furnishing, the traditional household appliances can be upgraded in the related art, if the intelligent function of the refrigerator is realized, a camera is required to be installed in the refrigerator so as to count the taking and placing conditions of the user on food in the refrigerator, and the intelligent control of replenishment reminding and the like is facilitated. In this kind of refrigerator use, because the camera lens temperature that is located the refrigerator compartment is lower, hot-air gushes into when the refrigerator door is opened, can make the camera lens surface of camera appear congeal the fog, leads to the camera to shoot the image fuzzy, can not get to put the food condition to the user and carry out accurate clear record, and it is the key problem in this field how to prevent that the camera lens from appearing congealing the fog.

In the prior art, various technical means for preventing fog of a camera lens are provided, for example, chinese patent CN105571238B discloses a refrigerator, when a door is opened, a control switch controls an air door to be opened, so that air flow flowing out of an air outlet can pass through the position of the camera to disperse water mist at the camera. The scheme is a defogging scheme provided under the condition that a camera lens is subjected to fog condensation, and the scheme still can lead to the situation that food is taken and placed in the process of opening the refrigerator door and cannot be clearly recorded. In order to solve the above problems, chinese patent CN106123475B discloses an anti-fog control method for a camera of a refrigerator, which provides a preheating and defogging function, specifically, when the refrigerator is started, a heating wire heats the camera with a power of P1; when a person is detected to approach and want to open the refrigerator door body, adjusting the power of the heating wire and heating the camera by P2 power; p2> P1. Since the heating is started before the user opens the refrigerator door, the camera has higher temperature when the user opens the refrigerator door, and the lens is prevented from being atomized. This scheme can both play certain defogging effect and also can prevent to congeal fog from appearing in the camera lens after opening the refrigerator door with regard to heating the camera lens before opening the refrigerator door. In the scheme disclosed in chinese patent CN106123475B, whether a user wants to open a refrigerator door is detected by the following method: the human body sensor detects that a person approaches the refrigerator, and after a second preset time period t2, the human body sensor detects again to confirm that the person approaches the refrigerator, the main control board judges that the person wants to open the door body of the refrigerator according to two detection results fed back by the human body sensor, then the power of the heating wire is adjusted and improved, and the camera is heated by P2 power.

The scheme for detecting whether the user wants to open the refrigerator door body has the following defects: in an actual use scene, even in a time period when the probability of the user opening the refrigerator is very low, the P1 power is required to be continuously adopted for preheating the camera, and the refrigerating environment inside the refrigerator is influenced.

Disclosure of Invention

The invention aims to solve at least one of technical problems in the prior art, and provides an intelligent household control method, electronic equipment and system based on a millimeter wave radar, which can reduce the influence of continuous preheating of a heating device of a camera on the refrigeration environment in a refrigerator.

In a first aspect, the present invention provides a millimeter wave radar-based smart home control method, where the method includes:

acquiring a first posture of a user in a cooking area in a cooking time period; the first posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar;

identifying the first gesture by using a first neural network model to obtain a behavior type corresponding to the first gesture; the first neural network model is obtained by training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture;

acquiring a traveling track of a user; the travel track is obtained by processing monitoring data acquired by a human body of a user based on a millimeter wave radar;

when the fact that the behavior type of the user belongs to the cooking behavior is detected, and the fact that the user leaves the cooking area is detected, under the condition that the trend of the travel track is detected to be a trend of traveling towards the position of the refrigerator, controlling the first power of the heating device to heat a camera in the refrigerator;

after the fact that the travelling track enters the trigger area is detected, when the trend of the travelling track is detected to be a trend of travelling towards the position of the refrigerator, the intention of a user is determined to be that the refrigerator door is intentionally opened; the trigger area comprises a trigger area outer area and a trigger area inner area, and the trigger area inner area is an area which is not more than a first distance away from the origin of the refrigerator; the outer region of the trigger region is a region which is more than a first distance and not more than a second distance away from the origin of the refrigerator;

when the user intends to open the refrigerator door intentionally, controlling the heating device to heat the camera in the refrigerator at a second power; the second power is greater than the first power.

In a second aspect, the present invention provides an electronic device comprising: the intelligent home control system comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the program, the intelligent home control method based on the millimeter wave radar is realized.

In a third aspect, the present invention provides an intelligent home control system based on millimeter wave radar, which is characterized by including millimeter wave radar and a control device, where the control device is configured to execute the above-mentioned intelligent home control method based on millimeter wave radar.

According to the intelligent home control method, the electronic device and the system based on the millimeter wave radar, the cooking time period of the user and the behavior of the cooking time period in the cooking area are identified, when the user cooks in the cooking area in the cooking time period, if the trend of the travel track of the user after leaving the cooking area is the trend of traveling towards the position of the refrigerator, the user has a high probability that the user goes to the position of the refrigerator and needs to open the refrigerator to take food materials, therefore, the camera is heated by using the first power when the user leaves the cooking area, and a good shooting effect can be achieved when the user opens the refrigerator. Furthermore, according to the historical cooking behavior data of the user, whether the user goes to the refrigerator immediately to take the food material or waits for a period of time to take the food material can be determined when the user is bound to open the refrigerator to take the food material during cooking, compared with the mode that the camera is preheated according to the temperature of the camera in the prior art, the method and the device do not need to start the heating device frequently, only needs to preheat in the cooking time period of the user, and reduces the influence on the refrigeration environment in the refrigerator. On the other hand, compared with the prior art that the user intention is easy to be misjudged, the method and the device have the advantages that the millimeter wave radar is used for obtaining the advancing track of the human body, after the advancing track is detected to enter the triggering area, whether the human body entering the triggering area of the refrigerator intends to open the refrigerator door can be accurately judged by detecting whether the trend of the advancing track is towards the position of the refrigerator, and the accuracy of intention judgment is improved.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The invention is further described below with reference to the accompanying drawings and examples;

FIG. 1 is a schematic diagram illustrating the construction of a coordinate map in an initialization mode according to an embodiment.

Fig. 2 is a schematic diagram illustrating a principle of a millimeter wave radar-based smart home control method in one embodiment.

Fig. 3 is a schematic diagram illustrating a principle of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 4 is a schematic diagram illustrating a principle of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 5 is a schematic diagram illustrating a principle of a millimeter wave radar-based smart home control method in an embodiment.

Fig. 6 is a schematic diagram illustrating a principle of a millimeter wave radar-based smart home control method in an embodiment.

Fig. 7 is a schematic diagram illustrating a principle of a millimeter wave radar-based smart home control method in an embodiment.

Fig. 8 is a schematic diagram illustrating a principle of a millimeter wave radar-based smart home control method in one embodiment.

Fig. 9 is a schematic flowchart of a smart home control method based on a millimeter wave radar in an embodiment.

Fig. 10 is a schematic flowchart of a smart home control method based on a millimeter wave radar in an embodiment.

Fig. 11 is a schematic flowchart of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 12 is a schematic flowchart of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 13 is a schematic flowchart of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 14 is a schematic flowchart of an intelligent home control method based on a millimeter wave radar in an embodiment.

Fig. 15 is a schematic flowchart of an intelligent home control method based on a millimeter wave radar in an embodiment.

FIG. 16 is a block diagram that illustrates the architecture of a computing device in one embodiment.

Reference numerals are as follows:

100. a millimeter wave radar; 210. a single-door refrigerator; 220. a double-door refrigerator.

Detailed Description

Reference will now be made in detail to the present embodiments of the present invention, preferred embodiments of which are illustrated in the accompanying drawings, wherein the drawings are provided for the purpose of visually supplementing the description in the specification and so forth, and which are not intended to limit the scope of the invention.

As shown in fig. 1, in an example, a place to which the millimeter wave radar-based smart home control method provided by the present invention is applied is a house, the house is divided into three major parts, namely a living room, a bedroom and a kitchen, the refrigerator can be placed at any position in the house according to the needs of a user, and the refrigerator can be a double-door refrigerator 220 or a single-door refrigerator 210, or other refrigerators, which is not limited herein. The millimeter wave radar 100 may be integrated in the refrigerator or distributed at other locations of the house, and one of the millimeter wave radars 100 is used for detecting the moving track of the human body. The method is applied to an intelligent home control system based on the millimeter wave radar, the control system comprises a control device and the millimeter wave radar 100 which is in communication connection with the control device, and the control device is used for receiving the traveling track and other data transmitted by the millimeter wave radar 100 so as to execute the method according to a set program.

Specifically, when a user uses the control system provided by the invention for the first time, the coordinate map of the house space needs to be constructed in an interactive mode. As shown in fig. 1, after the millimeter wave radar 100 is installed in the house space, in the initialization mode, in the space covered by the millimeter wave radar 100, the user sends from the point R, moves at a constant speed along the track with the arrow dotted line, and then returns to the point R, and the millimeter wave radar 100 can obtain the travel track of the user at that time, and use the travel track as the coordinate map of the house space. It is understood that the travel track of the user for constructing the coordinate map of the house space is determined by the user according to the specific layout of the house space, and in the initialization mode, the system will use the travel track of the user captured by the millimeter wave radar 100 as the outline of the house space, and determine the outline to include both the wall and the furniture appliances such as the cabinet and the refrigerator, etc. placed in fixed positions. The coordinate map is constructed after the travel track is obtained, the interactive interface displays the travel track of the user in the form of a line, all points on the line are known, and coordinates of all points in the space enclosed by the line are known through coordinate conversion, so that the coordinate ranges of the refrigerator, the cabinet and corresponding areas of all the spaces can be marked on the coordinate map in a manual mode. As shown in fig. 1, in this example, the coordinate range of the refrigerator position corresponding to the refrigerator can be marked manually, and since the user knows the structure of the refrigerator, the user can know that the handle of the refrigerator door is located at the lower left corner of the refrigerator position, so that a certain area of the lower left corner of the refrigerator is designated as the origin of the refrigerator in the coordinate range corresponding to the refrigerator position.

After a user specifies the coordinates of the refrigerator origin in the coordinate map, two parameters related to the refrigerator origin can be further determined, namely a trigger area and a non-trigger area, wherein the trigger area comprises a trigger area outer area and a trigger area inner area, and the trigger area inner area is an area which is not more than a first distance away from the refrigerator origin; the outer region of the trigger region is a region which is more than a first distance and not more than a second distance away from the origin of the refrigerator; the non-trigger area is an area which is more than a second distance away from the origin of the refrigerator. It is understood that the second distance may be a default parameter, or may be set by a user according to a habit of using the user or a position of the refrigerator, or may be set by other means, which is not limited herein.

Further, since the user is inevitably moving around the location of the refrigerator, that is, the user may pass around the refrigerator when the user opens the refrigerator door unintentionally, when the user thinks that the user needs to pass through the triggering area frequently due to the necessity of passing through the triggering area, the user may set the passing area parameters in the initialization mode, as shown in fig. 4, the refrigerator is set at a corner, the passage between the cabinet and the refrigerator is narrow, the user frequently enters the triggering area of the refrigerator when passing through the triggering area, and the user may walk a closed traveling track (not shown) between the cabinet and the refrigerator in the initialization mode as the corresponding passing area, where the hatched area shown in the figure is the area where the passing area intersects with the triggering area.

It should be noted that fig. 2 to 8 show a travel track of an intentionally opened refrigerator door by a solid line with letters and arrows, and a travel track of an unintentionally opened refrigerator door by a dot-dash line with letters and arrows, where an arrow is a travel direction of the travel track at the present time.

Hereinafter, the millimeter wave radar-based smart home control method provided by the embodiment of the present invention will be described and illustrated in detail through several specific embodiments.

As shown in fig. 12, in an embodiment, a smart home control method based on a millimeter wave radar is provided, the method includes:

step S402, acquiring a first posture of a user in a cooking area in a cooking time period; the first posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar.

Step S404, a first neural network model is used for identifying the first gesture to obtain a behavior type corresponding to the first gesture; the first neural network model is obtained after training by adopting training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture.

Step S406, acquiring a traveling track of a user; the travel track is obtained by processing monitoring data collected by a human body of a user based on a millimeter wave radar.

In this embodiment, the millimeter wave radar 100 is a radar that operates in a millimeter wave band (millimeter wave) for detection. Generally, the millimeter wave refers to the frequency domain (wavelength is 1-10 mm) of 30-300 GHz. The millimeter wave has a wavelength between microwave and centimeter waves, so the millimeter wave radar 100 has some advantages of both microwave radars and photoelectric radars, and the millimeter wave seeker has a small volume, light weight, high spatial resolution, can distinguish and identify very small targets, can simultaneously identify a plurality of targets, and has imaging capability.

Specifically, the millimeter wave radar 100 sends a radio frequency signal to a space through the antenna and receives an echo signal, multi-frame monitoring data of a user is obtained by receiving the echo signal, the distance and the azimuth angle between the user and the antenna on the millimeter wave radar 100 can be calculated from each frame of monitoring data to obtain the coordinates of the human body at the current moment, and the travelling track of the user can be obtained by processing the multi-frame monitoring data because each frame of data can obtain the coordinates of the human body at the corresponding moment.

And step S408, after detecting that the behavior type of the user belongs to the cooking behavior and detecting that the user leaves the cooking area, controlling the first power of the heating device to heat the camera in the refrigerator under the condition that the trend of the travel track is detected to be a trend of traveling towards the position of the refrigerator.

It should be noted that the technical contribution of the present invention lies in finding the internal association relationship between the cooking behavior of the user during the cooking time period and when the user is located in the cooking area and the intention of the user to open the refrigerator, that is, according to the statistics of the cooking time period of one family, the statistics of the records of the user taking food materials from the refrigerator during the cooking time period, and the recognition of the first gesture of the user in the cooking area marked by the user in the initialization mode, the recognition of the cooking behavior of the user can be automatically realized. Specifically, how to recognize the human body posture through the millimeter wave radar 100 belongs to the prior art, which is not an improvement point of the present invention, and specific implementation may refer to technical information disclosed in chinese patents with publication number CN110953681B, etc., and is not described herein again.

It can be understood that, in the history data, if there is a cooking behavior when the user is in the cooking time interval, in the cooking area, and there is a habit of opening the refrigerator to take the food in the cooking time interval, then there is a high probability that the user needs to open the refrigerator to take the food in the cooking time interval when the user is in the cooking time interval, in the cooking area, and there is the cooking behavior. Therefore, when the behavior type of the user is detected to belong to the cooking behavior and the user leaves the cooking area, under the condition that the trend of the moving track is detected to be the trend of moving towards the position of the refrigerator, the first power of the heating device is controlled to heat the camera in the refrigerator, namely, the user is considered to leave the cooking area, the food materials are required to be taken from the refrigerator with high probability, at the moment, the heating device is controlled to preheat the camera with the first power, so that when the user intends to open the refrigerator door in the following further judgment, the camera in the refrigerator can be quickly heated with the second power, and a better defogging effect is achieved. Due to the fact that the duration data statistics shows that the refrigerator can be opened by the user at a high probability in the cooking time period, even if misjudgment exists in subsequent intention judgment, namely the intention of the user for opening the refrigerator door intentionally cannot be recognized accurately, due to the fact that the camera is preheated by the first power in advance, when the user opens the refrigerator and does not heat the camera by the second power, the camera can have a good shooting effect.

Step S410, after the fact that the traveling track enters the trigger area is detected, when the fact that the trend of the traveling track is the trend of traveling towards the position of the refrigerator is detected, it is determined that the user intends to intentionally open the refrigerator door; the trigger area comprises a trigger area outer area and a trigger area inner area, and the trigger area inner area is an area which is not more than a first distance away from the origin of the refrigerator; the outer region of the trigger region is a region which is more than a first distance and less than a second distance away from the origin of the refrigerator.

Step S412, when the user intends to open the refrigerator door intentionally, controlling the heating device to heat the camera in the refrigerator with the second power; the second power is greater than the first power.

According to the method and the device, the cooking time period of the user and the behaviors of the cooking time period in the cooking area are identified, when the user conducts the cooking behaviors in the cooking area, if the trend of the travel track of the user after leaving the cooking area is the trend of traveling towards the position of the refrigerator, the user has a high probability of going to the position of the refrigerator and needing to open the refrigerator to take food materials, therefore, the camera is heated by using the first power when the user leaves the cooking area, and a good shooting effect can be achieved when the user opens the refrigerator. Furthermore, according to the historical cooking behavior data of the user, under the condition that the user inevitably needs to open the refrigerator to take the food materials during cooking, no matter whether the user goes to the refrigerator to take the food materials immediately after leaving the cooking area or waits for a period of time to take the food materials, compared with the method that the camera is preheated according to the temperature of the camera in the prior art, the method and the device do not need to start the heating device frequently, only needs to preheat in the cooking time period of the user, and reduces the influence on the refrigeration environment in the refrigerator. On the other hand, compared with the prior art that the intention of the user is easy to misjudge, the millimeter wave radar 100 is used for obtaining the advancing track of the human body, and after the fact that the advancing track enters the triggering area is detected, whether the human body entering the triggering area of the refrigerator intends to open the refrigerator door can be accurately judged by detecting whether the trend of the advancing track is towards the position of the refrigerator, so that the heating wire is prevented from being frequently in the starting state due to misjudgment, and the influence on the refrigerating environment inside the refrigerator is reduced.

As shown in fig. 9, in an embodiment, after detecting that the travel track enters the trigger area, when detecting that the trend of the travel track is a trend of traveling toward a location of the refrigerator, determining that the user intends to intentionally open the refrigerator door includes:

step S2041, detecting the travel direction of the process that the travel track enters a region where a trigger region and a passing region do not intersect from a non-trigger region; the triggering area is an area which is not more than a second distance away from the origin of the refrigerator; the non-trigger area is an area which is more than a second distance away from the origin of the refrigerator.

Step S2042, when it is detected that the traveling direction of the traveling trajectory in the process of entering the trigger area from the non-trigger area is the direction of continuously traveling toward the refrigerator position, determining that the user intends to intentionally open the refrigerator door.

Here, the travel direction of the travel track is the travel direction of each coordinate point on the travel track, and refers to the relationship between two coordinate points that are adjacent in time on the travel track of the user, that is, the travel direction of the coordinate point at the previous time is the direction in which the coordinate point at the previous time points to the coordinate point at the current time. If the direction points to any coordinate point in the coordinate area corresponding to the refrigerator position, that is, if the two coordinate points are in a line with any coordinate point in the coordinate area corresponding to the refrigerator position, the traveling direction at that time is considered to be the direction toward the refrigerator position. The fact that the traveling direction of the traveling track in the process of entering the triggering area from the non-triggering area is the direction of continuously traveling towards the refrigerator position means that the direction of each coordinate point on the traveling track entering the triggering area from the non-triggering area points to the coordinate point in the coordinate area corresponding to the refrigerator position. At this time, it is considered that the user has a strong intention to reach the refrigerator location.

As shown in fig. 2, in this scenario, there is no intersection region between the trigger region and the pass region, and when the user enters the trigger region from the non-trigger region along the travel trajectory a, the user inevitably enters the region where the trigger region and the pass region do not intersect. In this scenario, although there is an intersection region between the trigger region and the traffic region, when the travel locus D enters the trigger region from the non-trigger region, the trigger region and the traffic region do not intersect. In both cases, the process of the user moving from the non-triggering area to the triggering area is directly moving to the refrigerator position, and it can be determined that the approximate rate is to open the refrigerator door, that is, it is determined that the user intends to intentionally open the refrigerator door.

In the invention, in order to judge the intention of the user more accurately, the continuous advancing direction of the user in the process of entering the non-trigger area into the outer area of the trigger area is detected, obviously, if the intention of the user is to open the refrigerator door intentionally, the final target of the advancing track is necessarily the refrigerator, and the advancing tracks of the advancing track which are all moved from the outside of the trigger area into the trigger area are all towards the position direction of the refrigerator (such as the track A in fig. 2); if the final target of the user is not a refrigerator, the travel direction of the travel trajectory is less likely to be simultaneously directed to the refrigerator (e.g., the C trajectory in fig. 2) outside and inside the trigger area even if the travel trajectory passes through the trigger area. Due to the reasons, the invention can avoid the misjudgment of the user whose travel track only passes through the trigger area to a certain extent by analyzing the continuous direction of the travel track moving from the outside of the trigger area to the inside of the trigger area.

As shown in fig. 10, in another embodiment, after detecting that the travel track enters the trigger area, when detecting that the trend of the travel track is a trend of traveling toward a location of the refrigerator, determining that the user intends to intentionally open the refrigerator door specifically includes:

step S2043, after detecting that the travel track enters the area where the trigger area and the passing area intersect from the non-trigger area and passes through a set threshold distance, dividing the travel track entering the area where the trigger area and the passing area intersect into a plurality of sections of paths according to a preset time interval; the triggering area is an area which is not more than a second distance away from the origin of the refrigerator; the non-trigger area is an area which is more than a second distance away from the origin of the refrigerator.

Specifically, the trigger area comprises a trigger area outer area and a trigger area inner area, and the trigger area inner area is an area which is not more than a first distance away from the origin of the refrigerator; the outer region of the trigger region is a region which is more than a first distance and not more than a second distance away from the origin of the refrigerator; setting the first distance as a, the second distance as b and the threshold distance as theta; the set threshold distance θ is determined by the following equation:

(b-a)/3≤θ≤(b-a)/2。

it can be understood that, since the refrigerator is located at the corner, as shown by the travel track E in fig. 4, the travel track is directed to the position of the refrigerator when entering the region outside the trigger region, and since the user has two intentions in the intersected region, namely, opening the refrigerator door or passing through, the intention can not be finally determined after more detailed analysis and judgment on the travel track in the intersected region, so as to avoid misjudgment. However, if the travel track is too long, that is, the user has walked to a position very close to the refrigerator in the intersecting area, the user's intention is determined and the camera is heated at the second power, so that the heating effect is poor. Therefore, a proper set threshold distance theta is needed to be set to judge the intention of the user in the area of the camera, and in the invention, the first distance is the distance from the origin of the refrigerator (namely the handle of the refrigerator door) when the user opens the door, namely, when the user reaches or enters the first distance, the user can open the refrigerator door, and the reheating obviously has poor defogging effect. Therefore, the set threshold distance theta is set according to the first distance, so that the distance of the user walking in the intersected area is ensured to be between one third and one half of the difference between the second distance b and the first distance a, namely, the intention of the user is judged when the user does not walk to the triggering area in the intersected area (namely, the area with the distance from the origin of the refrigerator not larger than the first distance) so as to control heating with the second power, and the intention judgment accuracy is provided, and meanwhile, the better defogging effect is achieved. It is understood that the set threshold distance θ can also be set according to the usage habits of different users, for example, the length of the travel track that a user passes after entering the area where the trigger area intersects with the passing area and when the refrigerator door is opened is counted, and 1/3 of the length is used as the set threshold distance.

It should be noted that, the threshold distance θ is set as a start node of the determination intention, which is too long, and it is easy to cause that the travel track entering the intersection area may already enter the trigger area, and when the user is in the trigger area, the arm length of the user already meets the condition of opening the door, which may result in poor recognition effect of the camera. When the travel track enters the area outside the trigger area towards the refrigerator position, the user does not turn the corner when just entering the intersection area, the travel track still faces the refrigerator position, and the set threshold distance theta is too short, so that the intention coefficient is easily larger than the set value, misjudgment is easily caused, and the control device frequently heats the camera at the second power.

In step S2044, the direction from the start point to the end point of each segment of the path is taken as the direction of the segment of the path.

Step S2045, calculating a ratio of the number of paths oriented to the refrigerator position at the current time to the total number of the path segments as an intention coefficient.

In step S2046, when it is detected that the intention coefficient is greater than the set value, it is determined that the user intends to intentionally open the refrigerator door.

In one example, as shown in fig. 4, when the travel locus Z passes through (b-a)/3 from the non-trigger area to the area where the trigger area intersects with the pass area, the paths are divided into 40 segments in total at the present time, and if the number of paths oriented to point to the refrigerator location is calculated to be 7 and the number of paths oriented not to point to the refrigerator location is calculated to be 33, the intention factor is 7/40 and is smaller than a set value (the set value is 70% in this example), and it is determined that the user intends to open the refrigerator door unintentionally. In one example, as shown in fig. 4, when the travel trajectory E passes (b-a)/3 from the non-trigger area to the area where the trigger area intersects with the pass area, the paths are divided into 40 segments in total at the present time, and if the number of paths oriented to point to the refrigerator location is calculated to be 40 and the number of paths oriented not to point to the refrigerator location is calculated to be 0, the ratio of the two is 40/40 greater than the set value (the set value is 70% in this example), it is determined that the user intends to intentionally open the refrigerator door. It is understood that the setting value can be set in a statistical manner for different users, for example, an average value of the intention coefficients of a user when the user enters the area where the triggering area intersects with the passing area and does not open the refrigerator door is counted, and the corresponding setting value is set for different users according to the average value.

In the invention, considering that the difference of building layouts of different families exists, when the refrigerator is positioned in an area where a user must pass through, as shown in the figure, the refrigerator is positioned at the corner of a kitchen, because the corner area is narrow, the refrigerator enters the trigger area almost every time the user enters or exits the kitchen, and the refrigerator is positioned at the corner position, when the user enters the trigger area from the outside of the trigger area, the refrigerator is inevitably positioned towards the position direction of the refrigerator at a high probability, and at the moment, misjudgment is easily caused. Therefore, the invention provides a more accurate judgment method, which dynamically judges the trend (namely the intention coefficient) of the travel track of the user in the passing area at each moment by judging the trend (namely the intention coefficient) of the travel track entering the area where the trigger area and the passing area are intersected in real time, and reduces the misjudgment probability.

In one embodiment, the method further comprises:

and counting the average distance between the user and the refrigerator origin when the user opens the refrigerator door as the first distance corresponding to the user.

In the invention, the differences among different users, such as height, arm length, use habits and the like, are considered, so that the corresponding first distances of different users when the refrigerator is opened are different, and the accuracy of the intention judgment of different users in the area where the trigger area and the passing area are intersected depends on the first distances, therefore, the intention judgment accuracy can be further improved by matching the corresponding first distances of the different users and the user. It can be understood that the technical contribution of the present invention lies in that corresponding first lengths are adopted for different users, so that the accuracy of intention judgment can be improved, how to use the millimeter wave radar 100 to identify the user identity belongs to the prior art, and details are not described here, and specific implementation manners may refer to technical information disclosed in chinese patents with publication numbers CN112686094B, CN111738060A, and the like.

In one embodiment, after detecting that the travel track enters the trigger zone, the method further comprises:

when the traveling direction of the traveling track entering the outer region of the trigger region is detected to be the direction of traveling towards the refrigerator position, it is determined that the user intends to open the refrigerator door intentionally.

As shown in fig. 2, the traveling direction of the traveling track B in the non-trigger area is not toward the refrigerator position, and the traveling track B does not face the refrigerator position immediately after entering the trigger area outer area from the non-trigger area, and the user temporarily intends to open the refrigerator door.

When the traveling direction of the traveling track entering the area in the trigger area is detected to be the direction of traveling towards the position of the refrigerator, the intention of the user is determined to be that the refrigerator door is opened intentionally.

As shown in fig. 2, in this example, since the triggering area inner region is directly bordered by the non-triggering area, there is a case where the user directly enters the triggering area inner region from the non-triggering area, and when it is detected that the traveling direction of the traveling trajectory entering the triggering area inner region is a direction traveling toward the refrigerator position, it is determined that the user intends to intentionally open the refrigerator door.

Considering that the final target of the user is not the refrigerator when the user initially passes through the refrigerator trigger zone (including the trigger zone inner area and the trigger zone outer area), the refrigerator is not triggered to heat the camera at the second power when the user does not travel towards the refrigerator location through the trigger zone of the refrigerator. However, when a user is in the triggering area, there may be an intention to temporarily want to open the refrigerator door, at which time the user may advance toward the refrigerator position in the triggering area. In the invention, the behavior of temporarily changing the travel track of the user in the trigger area outer area and the trigger area inner area is monitored in real time, and when the condition that the user changes the travel direction in the trigger area (namely, the user travels to the position of the refrigerator) is monitored, the intention of the user can be judged in time, and the function of heating the camera is realized.

In one embodiment, a method for determining a trigger area inner region and a trigger area outer region is provided, as shown in fig. 3, a straight line OPQ is drawn with an origin O of a refrigerator as a center, where OP is equal to a first distance and a second distance, such as OQ, a small circle is drawn with OP as a radius, a large circle is drawn with OQ as a radius, the small circle is a trigger area inner region, and an area enclosed by an inner portion of the large circle and an outer portion of the small circle is a trigger area outer region.

In one embodiment, the refrigerator origin is coordinates of a refrigerator door handle; the method further comprises the following steps:

and excluding the coordinates corresponding to the refrigerator and the coordinates of a region which is not more than the first distance away from the refrigerator origin and passes through the refrigerator by connecting the refrigerator origin from the triggering region.

And excluding the coordinates corresponding to the refrigerator and the coordinates of a connecting line with the refrigerator origin and passing through the refrigerator in an area which is away from the refrigerator origin by more than a first distance and not more than a second distance from the refrigerator origin outside the trigger area.

As shown in fig. 2 and 5, after partial areas of the small circle and the large circle drawn by taking the origin of the refrigerator as the center of the circle are removed, the corresponding trigger areas are obtained. As shown in fig. 4 and 6, when the refrigerator is located at a corner, a part of the activation region thereof is located at the other side of the wall, and it is apparent that this part needs to be excluded from the activation region as well.

The present invention considers that a user stands at a position where the user can touch and see the door handle of the refrigerator when opening the door of the refrigerator, and obviously, it is impossible to open the refrigerator when the user stands at a position where the user cannot see and cannot touch the door handle of the refrigerator, for example, when the user stands at a side of the refrigerator away from the door handle, so that it is impossible to open the door of the refrigerator even if the user is in close proximity to the door handle of the refrigerator and travels toward the door handle of the refrigerator, which may be doing something else, such as putting an article on the top of the refrigerator, etc. Therefore, the probability of erroneous judgment can be reduced by excluding the coordinates from the trigger region inner area and the trigger region outer area.

As shown in fig. 11, in one embodiment, the method further comprises:

step S302, a plurality of passing tracks when the user does not open the refrigerator door under the condition of passing through the trigger area inner area and/or the trigger area outer area are counted.

Step S304, taking an area surrounded by the plurality of passing tracks as the passing area corresponding to the user.

As shown in fig. 3, as an example, the travel tracks X, Y, and W in the drawing are three tracks with the highest frequency when the user a passes through the area within one month without opening the refrigerator door, if the frequencies corresponding to the X, Y, and W within one month are 120 times, 30 times, and 130 times, the two ends of the X and the two ends of the W are respectively connected to enclose a passing area corresponding to the user a, and the shaded area in the drawing is an area where the triggering area and the passing area intersect. If the corresponding frequencies of X, Y and W in a month are 100 times, 120 times and 30 times respectively, two ends of X and two ends of Y are connected to form a passing area corresponding to the user A respectively. That is, in this embodiment, an area surrounded by two traffic tracks with the highest occurrence frequency within a period of time (in this example, one month) is selected from the multiple traffic tracks as the traffic area corresponding to the user.

In the invention, the refrigerator is positioned at the corner, and the triggering area (comprising the triggering area inner area and the triggering area outer area) of the refrigerator is positioned on the road where the user is bound to pass, so that the probability difference of opening the refrigerator door or passing when the user walks in the area where the triggering area and the passing area are intersected is not large, the passing area corresponding to the corresponding user is determined by combining the historical passing habits of different users, and the accuracy of the intention judgment of the user can be improved.

Referring to fig. 13, in an embodiment, after detecting that the behavior type of the user belongs to a cooking behavior and detecting that the user leaves the cooking area, in a case that a trend of a travel track is detected as a trend of traveling toward a refrigerator position, controlling the heating device to heat the camera in the refrigerator at the first power specifically includes:

step S4081, when the behavior type of the user is detected to belong to the cooking behavior and the user is detected to leave the cooking area, whether the user and the refrigerator are located in the same space area is detected; the same space area is an area where a connecting line between the current time coordinate of the user and the position of the refrigerator does not pass through a wall.

Step S4081, when the user is located in the same spatial area as the refrigerator, detects the traveling direction of the traveling locus at the current time.

And step S4081, when the traveling direction of the traveling track at the current moment is detected to point to the position of the refrigerator, controlling the first power of the heating device to heat the camera in the refrigerator.

As shown in fig. 8, in an example, coordinates of an arrow portion of the travel track G are located in a living room area, a connection line between the arrow portion and the refrigerator position does not pass through a wall, it is considered that the user and the refrigerator are located in the same space area at this time, and when it is detected that the travel direction of the travel track at the current time points to the refrigerator position, the heating device is controlled to heat the camera in the refrigerator at the first power. In one example, when the coordinates of the part of the travel track F located in the living room area are located in the same space area as the refrigerator, but the travel direction of the travel track at the current time is detected not to be directed to the refrigerator, it is considered that the user is not directed to the refrigerator at other places, and thus the heating device is not controlled to heat the camera in the refrigerator at the first power.

In a specific application scenario, the refrigerator may be located in a different space region from the cooking region, i.e. separated from each other by a wall, and the range of the user's activity may also be separated from the space where the refrigerator is located by a wall, at this time, although the user travels to the refrigerator position in another space separated by a wall, it does not mean that the user wants to open the refrigerator door. The connecting lines between the coordinates of all parts of the travel track H and the position of the refrigerator in fig. 8 pass through the wall, and it is considered that the user and the refrigerator are not located in the same spatial region at this time. For example, a user needs to watch a tv drama in a living room and wait for cooked rice to be cooked, and then needs to take dishes out of the refrigerator for cooking, so that the user does not need to control the first power of the heating device to heat the camera in the refrigerator, and the user behavior in a different space from the refrigerator needs to avoid interfering with the timing of controlling the heating device to heat the camera at the first power. Therefore, the method and the device can only respond to the user behavior in the same area with the refrigerator by identifying whether the user and the refrigerator are positioned in the space areas separated by the wall and correspondingly processing the behaviors in different space areas, thereby avoiding the misjudgment of the user intention.

Referring to fig. 14, in one embodiment, the method further comprises:

step S502, after detecting that the travel track enters the trigger area, detecting the travel direction of the travel track entering the trigger area from the non-trigger area.

Step S504, according to the advancing direction, when the advancing direction of the advancing track in the process of entering the outer area of the trigger area from the non-trigger area is detected to be the direction of continuously advancing towards the refrigerator position, a second gesture of the user is obtained; the second posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar.

Step S506, a second neural network model is used for identifying the second posture to obtain a behavior type corresponding to the second posture; the second neural network model is obtained after training by adopting training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture.

Step S508, when it is detected that the behavior type of the user belongs to the unintentional behavior and the unintentional behavior is maintained to enter the region outside the trigger region from the non-trigger region, it is determined that the user intends to open the refrigerator door unintentionally.

In one example, the method further comprises:

when detecting that the advancing direction of the advancing track entering the outer area of the trigger area is the advancing direction towards the position of the refrigerator, acquiring a third gesture of the user; the third posture is obtained by processing monitoring data acquired by the human body based on a millimeter wave radar;

identifying the third posture by using a second neural network model to obtain a behavior type corresponding to the third posture;

when the type of behavior of the user is detected as being unintentional behavior and the unintentional behavior is kept traveling toward the refrigerator position, it is determined that the user intends to open the refrigerator door unintentionally.

In one example, the method further comprises:

when the traveling direction of the traveling track entering the triggering area is detected to be the traveling direction towards the refrigerator position, acquiring a fourth gesture of the user; the fourth posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar;

identifying the fourth posture by using a second neural network model to obtain a behavior type corresponding to the fourth posture;

Given that a refrigerator is placed in a home environment, and that the user is a member of the family, certain actions by the user are avoided to be taken near the location of the refrigerator. For example, in the floor mopping action in the household practice, when the user drags the floor, the mop is dragged or pushed to continuously travel from the non-trigger area to the outer trigger area or even to the inner trigger area, obviously, simply judging the travel track of the user in this case is likely to cause misjudgment, resulting in heating the camera at the second power. In the invention, in order to solve the problems, a gesture recognition technology is introduced, and the relation between the gesture and the behavior type of the user can be recognized through the training of a neural network model. By adding the recognition of the continuous behavior of the user when the user travels towards the refrigerator position, the misjudgment probability is reduced.

Referring to fig. 15, in one embodiment, the method further comprises:

step S602, after detecting that the travel track of the first user enters the trigger area, detecting a travel direction of the travel track of the first user entering the trigger area from the non-trigger area.

Step S604, according to the advancing direction, when the advancing direction of the advancing track of the first user in the process of entering the outer region of the trigger region from the non-trigger region is detected to be the direction of continuously advancing towards the refrigerator position, acquiring a fifth gesture of the first user; and the fifth posture is obtained by processing monitoring data acquired by the human body based on the millimeter wave radar.

Step S606, identifying the fifth posture by using a third neural network model to obtain a behavior type corresponding to the fifth posture; the third neural network model is obtained by training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture.

Step S608, when it is detected that the behavior type of the first user belongs to an avoidance behavior and the first user keeps the avoidance behavior entering the outside area of the trigger area from the non-trigger area, and it is detected that the travel track of the second user enters the area where the trigger area intersects with the pass area, it is determined that the user of the first user intends to open the refrigerator door unintentionally.

As shown in fig. 7, the travel track I is a travel track of the first user when the first user is avoiding, and J is a travel track of the second user when the second user meets the first user at a corner, at this time, the method of the present invention can accurately recognize the intention of the first user, and although the travel track of the first user travels straight toward the refrigerator position, since the fifth gesture is recognized and the travel track J of the second user is recognized, the erroneous judgment of the intention of the first user is avoided.

In an actual application scene, when the refrigerator is located in a passing area at a corner, because the passing area is narrow, when two persons need to pass at the same time, one person needs to avoid, when a user close to the refrigerator carries out avoidance, the user generally moves from an area where a triggering area and the passing area are intersected to the refrigerator to avoid other passing users, and at the moment, misjudgment is easily generated. According to the invention, the intention judgment scheme is further optimized, and the behavior corresponding to the fifth posture of the first user can be identified through training of the neural network model, so that whether the first user belongs to the avoidance behavior is determined. After the first user is identified as the avoidance behavior, whether an area where the second user passes through the triggering area and the passing area is intersected is further judged, and if the two conditions are met, the user intention of the first user is judged to be that the refrigerator door is opened unintentionally. It should be understood that, at this time, the user intention judgment of the second user can still be performed synchronously, and the intention judgment of the second user is not affected.

FIG. 16 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may particularly be a control apparatus. As shown in fig. 16, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and also stores a computer program, and when the computer program is executed by a processor, the processor can realize the intelligent home control method based on the millimeter wave radar. The internal memory can also store a computer program, and when the computer program is executed by the processor, the processor can execute the intelligent home control method based on the millimeter wave radar. It will be appreciated by those skilled in the art that the configuration shown in fig. 16 is a block diagram of only a portion of the configuration associated with the inventive arrangements and is not intended to limit the computing devices to which the inventive arrangements may be applied, and that a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, there is provided an electronic device including: the intelligent home control method based on the millimeter wave radar comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the steps of the intelligent home control method based on the millimeter wave radar when executing the program. Here, the steps of the millimeter wave radar-based smart home control method may be steps in the millimeter wave radar-based smart home control methods of the foregoing embodiments.

In an embodiment, an intelligent home control system based on a millimeter wave radar is further provided, and includes a millimeter wave radar 100 and a control device, where the control device is configured to execute the intelligent home control method based on a millimeter wave radar.

In one embodiment, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions for causing a computer to execute the steps of the above-mentioned millimeter wave radar-based smart home control method. Here, the steps of the millimeter wave radar-based smart home control method may be steps in the millimeter wave radar-based smart home control methods of the foregoing embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRA), rambus (Rambus) direct RAM (RDRA), direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM), among others.

All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.

Claims

1. A smart home control method based on a millimeter wave radar is characterized by comprising the following steps:

after the fact that the travel track enters the trigger area is detected, when the trend of the travel track is detected to be the trend of traveling towards the position of the refrigerator, it is determined that the user intends to open the refrigerator door intentionally; the triggering area comprises a triggering area outer area and a triggering area inner area, and the triggering area inner area is an area which is not more than a first distance away from the origin of the refrigerator; the outer region of the trigger region is a region which is more than a first distance and less than a second distance away from the origin of the refrigerator;

when the user intends to open the refrigerator door, controlling a second power of the heating device to heat the camera in the refrigerator; the second power is greater than the first power;

after the advancing track is detected to enter the trigger area, the advancing direction of the advancing track in the process of entering the outer area of the trigger area from the non-trigger area is detected;

according to the advancing direction, when the advancing direction of the advancing track in the process of entering the outer region of the trigger region from the non-trigger region is detected to be the direction of continuously advancing towards the refrigerator position, acquiring a second gesture of the user; the second posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar;

identifying the second posture by using a second neural network model to obtain a behavior type corresponding to the second posture; the second neural network model is obtained after training by adopting training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture;

when the behavior type of the user is detected to be unintentional behavior and the unintentional behavior is kept entering the area outside the trigger area from the non-trigger area, the fact that the user intends to open the refrigerator door unintentionally is determined.

2. The smart home control method based on the millimeter wave radar according to claim 1, wherein after it is detected that the behavior type of the user belongs to a cooking behavior and it is detected that the user leaves the cooking area, and in a case that it is detected that a trend of a travel track is a trend of traveling toward a refrigerator position, controlling a first power of a heating device to heat a camera in the refrigerator specifically comprises:

when the behavior type of the user is detected to belong to the cooking behavior and the user is detected to leave the cooking area, detecting whether the user and the refrigerator are located in the same space area; the same space area is an area where a connecting line between the current time coordinate of the user and the refrigerator position does not pass through a wall;

when a user and the refrigerator are positioned in the same space region, detecting the traveling direction of the traveling track at the current moment;

and when the fact that the traveling direction of the traveling track at the current moment points to the position of the refrigerator is detected, controlling the heating device to heat the camera in the refrigerator at the first power.

3. The millimeter wave radar-based smart home control method according to claim 1, further comprising:

when detecting that the advancing direction of the advancing track entering the outer area of the trigger area is the advancing direction towards the position of the refrigerator, acquiring a third gesture of the user; the third posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar;

4. The millimeter wave radar-based smart home control method according to claim 1, further comprising:

5. The millimeter wave radar-based smart home control method according to claim 1, further comprising:

after detecting that the traveling track of the first user enters the trigger area, detecting the traveling direction of the traveling track of the first user entering the process of the outer area of the trigger area from the non-trigger area;

according to the advancing direction, when the advancing direction of the advancing track of the first user in the process of entering the outer region of the trigger region from the non-trigger region is detected to be the direction of continuously advancing towards the position of the refrigerator, acquiring a fifth gesture of the first user; the fifth posture is obtained by processing monitoring data collected by a human body based on a millimeter wave radar;

identifying the fifth posture by using a third neural network model to obtain a behavior type corresponding to the fifth posture; the third neural network model is obtained by training data with marking information, and the marking information is used for marking the human body posture in the training data and the behavior type matched with the human body posture;

and under the condition that the behavior type of the first user is detected to belong to an avoidance behavior and the first user keeps the avoidance behavior entering the outer region of the trigger region from the non-trigger region, when the travel track of the second user is detected to enter the region where the trigger region and the pass region intersect, determining that the user of the first user intends to open the refrigerator door unintentionally.

6. The millimeter wave radar-based smart home control method according to claim 1, wherein when the trend of the travel track is detected as a trend of traveling towards a refrigerator position, determining that the user intends to intentionally open a refrigerator door specifically comprises:

detecting the travel direction of the process that the travel track enters a region where the trigger region and the passing region do not intersect from the non-trigger region; the non-trigger area is an area which is more than a second distance away from the origin of the refrigerator;

and according to the travelling direction, when the travelling direction of the travelling track in the process of entering the triggering area from the non-triggering area is detected to be the direction of continuously travelling towards the position of the refrigerator, determining that the user intends to intentionally open the refrigerator door.

7. The millimeter wave radar-based smart home control method according to claim 1, wherein when the trend of the travel track is detected as a trend of traveling towards a refrigerator position, determining that the user intends to intentionally open a refrigerator door specifically comprises:

after detecting that the travel track enters a region where the trigger region and the passing region intersect from the non-trigger region and passes through a set threshold distance, dividing the travel track entering the region where the trigger region and the passing region intersect into a plurality of sections of paths according to a preset time interval; the non-trigger area is an area which is more than a second distance away from the origin of the refrigerator;

taking the direction from the starting point to the end point of each section of the path as the direction of the section of the path;

calculating the ratio of the number of the paths pointing to the refrigerator at the current moment to the total number of the path segments as an intention coefficient;

when the intention coefficient is detected to be larger than the set value, the intention of the user is determined to be the intention of opening the refrigerator door.

8. An electronic device, comprising: the millimeter-wave radar-based smart home control method is characterized in that the processor executes the program to realize the millimeter-wave radar-based smart home control method according to any one of claims 1 to 7.

9. An intelligent home control system based on millimeter wave radar is characterized by comprising the millimeter wave radar and a control device, wherein the control device is used for executing the intelligent home control method based on millimeter wave radar according to any one of claims 1 to 7.