CN114063572A - Non-sensing intelligent device control method, electronic device and control system - Google Patents

Non-sensing intelligent device control method, electronic device and control system Download PDF

Info

Publication number
CN114063572A
CN114063572A CN202010760967.5A CN202010760967A CN114063572A CN 114063572 A CN114063572 A CN 114063572A CN 202010760967 A CN202010760967 A CN 202010760967A CN 114063572 A CN114063572 A CN 114063572A
Authority
CN
China
Prior art keywords
information
point cloud
human body
person
radio frequency
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010760967.5A
Other languages
Chinese (zh)
Inventor
唐志刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Entropy Technology Co ltd
Original Assignee
Beijing Entropy Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Entropy Technology Co ltd filed Critical Beijing Entropy Technology Co ltd
Priority to CN202010760967.5A priority Critical patent/CN114063572A/en
Publication of CN114063572A publication Critical patent/CN114063572A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00Programme-control systems
    • G05B19/02Programme-control systems electric
    • G05B19/418Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM]
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B15/00Systems controlled by a computer
    • G05B15/02Systems controlled by a computer electric
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/10Machine learning using kernel methods, e.g. support vector machines [SVM]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/20Ensemble learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00Computing arrangements using knowledge-based models
    • G06N5/01Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/20Pc systems
    • G05B2219/26Pc applications
    • G05B2219/2642Domotique, domestic, home control, automation, smart house
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02PCLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A control method, electronic equipment and a control system of a non-sensing intelligent device belong to the technical field of intelligent home, wherein the method comprises the following steps: acquiring a radio frequency scanning signal; judging whether a person is in a controlled scene or not based on the radio frequency scanning signal, and calculating to obtain any one or more of the following human body information: a position of the person, a posture of the person, physiological information of the person; collecting and storing control information, wherein the control information comprises whether a person exists in a scene or not and human body information acquired when the person exists in the scene; and matching the collected control information according to the preset corresponding relation between the control information and the control instruction of the intelligent equipment, determining the corresponding control instruction and outputting the control instruction.

Description

Non-sensing intelligent device control method, electronic device and control system
Technical Field
The invention belongs to the technical field of intelligent home furnishing, and particularly relates to a control method of a non-inductive intelligent device.
Background
With the progress of technology, intelligent household appliances are rapidly developed in recent years and enter thousands of households, so that the family life of people is more comfortable, simpler, more convenient and more happy. How to let users manage home devices by more convenient means is a topic of interest in the industry. For this reason, CN 111158246A provides an intelligent household appliance control system, which uses a microwave radar to detect gestures in a designated area and sends the detected gesture information to an intelligent household appliance control application program; and the intelligent household appliance control client determines a control instruction according to the received gesture information, and then controls the intelligent household appliance according to the determined control instruction. However, the control system needs to preset a gesture instruction through the client and requires the user to make a specific gesture in the designated area, and the whole control process still needs the user to make an active operation, and no sensorless control can be realized.
Disclosure of Invention
The invention aims to provide a method for controlling an intelligent device without sensing, and provides corresponding electronic equipment and a control system.
Based on the above purpose, the invention provides the following technical solutions:
in a first aspect, a method for sensorless smart device control includes the steps of,
s101: acquiring a radio frequency scanning signal;
s102: judging whether a person is in a controlled scene or not based on the radio frequency scanning signal, and calculating to obtain any one or more of the following human body information: a position of the person, a posture of the person, physiological information of the person;
s103: collecting and storing control information, wherein the control information comprises whether a person exists in a scene or not and human body information acquired when the person exists in the scene;
s104: and matching the collected control information according to the preset corresponding relation between the control information and the control instruction of the intelligent equipment, determining the corresponding control instruction and outputting the control instruction. According to the method, whether a person exists in a scene, the position of the person, the posture of the person, the physiological information of the person and other objective information are collected to further match a control instruction, the person does not need to make active control action, and non-perception control can be achieved.
In order to further meet the active control will of the person, the human body information may further include gesture information of the person.
In order to save the computation workload, step S102 first determines whether there is a person in the controlled scene according to the radio frequency scanning signal; and when the person is judged to be present, the human body information is obtained based on the point cloud calculation.
In order to further save the calculation amount, whether a moving object exists can be judged by simple calculation, and then whether a person exists can be further confirmed. The step S102 is to determine whether there is a person in the scene:
performing FFT signal processing on the acquired radio frequency scanning signal to judge whether a moving object exists in a scene; if no moving object is found and no person exists in the scene in the previous scanning, directly judging that no person exists in the scene in the current scanning;
if a moving object is found or the moving object is not found but people are judged in the scene in the previous scanning, the radio frequency scanning signal is further resolved into point cloud information, and whether people exist in the scene is judged based on the point cloud information.
Step S102, judging whether a person exists by using a machine learning model MO obtained by training, comprising the following steps:
s201: based on radio frequency scanning, point cloud information is obtained: taking a group of point cloud data obtained by N times of radio frequency scanning as 1 point cloud data group, and calculating point cloud information corresponding to each point cloud data group, wherein the information corresponding to each reflection point in the point cloud information at least comprises the spatial position, the speed and the signal intensity information of the reflection point; in order to improve the efficiency and accuracy of machine learning, the point cloud information corresponding to the information corresponding to each reflection point can further comprise acceleration and noise amplitude information;
s202: inputting the point cloud information calculated in step S201 into the model MO, and outputting the point cloud information to the output destination of O = { (Pr)m,Psm),m=1,2,3,……,M},PrmIs the probability, Ps, of the presence of the mth human target to be detectedmThe spatial position of a representative point of the mth human target to be detected is shown, and M is the number of people in the scene; the space position of the human body representative point corresponds to the position of a person, M =0 represents that no person exists in the scene, and M > 0 represents that a person exists in the scene;
the model MO is obtained through the following steps:
s301: performing point cloud data acquisition on a scene based on radio frequency scanning;
s302: taking a group of point cloud data obtained by N times of radio frequency scanning as 1 point cloud data group, and calculating point cloud information corresponding to each point cloud data group, wherein the information corresponding to each reflection point in the point cloud information at least comprises the spatial position, the speed and the signal intensity information of the reflection point; n is an integer greater than or equal to 2;
s303: marking spatial position information of human body representative points corresponding to point cloud information obtained by each group of radio frequency scanning in a scene according to reference information recorded in the data acquisition process; collecting point cloud information obtained by a plurality of groups of radio frequency scanning and corresponding human body representative point space position information to form a first sample set; training a model MO capable of identifying the number M of the people in the scene and the space position of each human body representative point by using a machine learning method based on the first sample set; the reference information is a video record or an audio and video record synchronously acquired in the radio frequency scanning process; the marks of the training set can be marked manually, and as a more preferable scheme, the positions of representative points, key points and human behavior information of a human body can be extracted from the reference information by using the existing artificial intelligence recognition method, so that point cloud information is marked automatically based on the same time axis.
And acquiring the posture of the person and the gesture of the person by using the models MK and MA obtained by training according to the output result of the MO, including,
s203: when the number M of the human bodies output by the MO is larger than or equal to 1, filtering point cloud information obtained by radio frequency scanning according to the spatial position information of the human body representative points output by the model MO, and only keeping the point cloud information of a specific distance range near the human body representative points; inputting the filtered point cloud information into a model MK, scanning and identifying the input point cloud information by the model MK by using a sliding window method, wherein the window length corresponds to NpkGroup radio frequency scanning is carried out, and a plurality of key point information of M human bodies is output;
s204: inputting the output result of the model MK into the model MA, scanning and identifying the input information by the model MA by using a sliding window method, wherein the window length corresponds to NmaThe secondary MK continuously outputs the result, and outputs the specific human posture and the human gesture;
continuing to execute step S304 on the basis of the training of the model MO to obtain a model MK:
will NpkThe point cloud information obtained by the group radio frequency scanning is regarded as a point cloud information sequence, the point cloud information sequence is filtered according to the output result of the model MO, only the point cloud information in a specific distance range near the human body representative point is reserved, and the filtered point cloud information sequence is obtained; n is a radical ofpkIs an integer greater than 1;
selecting a plurality of key points on a human body based on human body joint points, marking the spatial position information of the human body key points corresponding to each filtered point cloud information sequence according to reference information recorded in the formation of a data set, collecting a plurality of information sequences, filtering and marking to form a second sample set;
training a model MK capable of identifying a plurality of key point information of M human bodies in a scene by using a machine learning method based on the second sample set; the output target of the model MK is key point information OK = { (Pr) of each human body target to be detectedk,Psk) K =1, 2, 3, … …, K }, where K is the number of selected human body key points; pr (Pr) ofkThe probability of the k-th key point of a certain human target to be detected; pskThe spatial position of the kth key point of a certain human body target to be detected exists;
continuing to execute step S305 on the basis of the training of the model MK to obtain the model MA:
obtaining a third sample set, wherein the third sample set comprises a positive sample and a negative sample, the positive sample comprises a specific human body posture or gesture obtained from the reference information, and N corresponding to the specific human body posture or gesturemaContinuously outputting results of the secondary MK, and taking other output results of the secondary MK as counterexample samples without certain specific human body posture or gesture; n is a radical ofmaIs an integer greater than 1;
training a model MA capable of identifying a specific human body posture or gesture according to a plurality of point cloud information sequences by using a machine learning method based on a third sample set; the output target of the model MA presents the probability of a certain specific gesture or gesture for a certain human body target to be detected.
The method for controlling the sensorless intelligent device further comprises the steps of obtaining environment information and collecting the environment information as control information, wherein the environment information comprises environment temperature and/or light intensity.
In step S105, for each type of control information, the controlled device in which the type of control information is registered is acquired, the type of control information is distributed to each registered controlled device control information group, then each control information group is matched with the corresponding controlled device according to the preset correspondence between the control information and the intelligent device control instruction, the corresponding control instruction is determined, and the controlled device is controlled by using the determined control instruction.
In a second aspect, an electronic device includes:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of the first aspect.
In a third aspect, a sensorless home control system includes:
the environment monitoring module is used for monitoring the environment temperature and/or the light intensity to obtain environment information;
the radio frequency detection module is used for carrying out radio frequency scanning on a controlled scene to acquire a radio frequency scanning signal;
the electronic device of claim 7, configured to obtain environmental information; the system comprises a radio frequency scanning signal acquisition unit, a display unit and a display unit, wherein the radio frequency scanning signal acquisition unit is used for acquiring a radio frequency scanning signal, judging whether a person exists in a scene according to the radio frequency scanning signal and acquiring various human body information; determining an equipment control instruction according to control information including environment information, and information about whether a person and a human body exist;
and the communication module is used for sending the control instruction determined by the electronic equipment to the control module of the controlled equipment.
In order to facilitate installation, reduce wiring investment of users and improve the signal acquisition effect of radio frequency scanning, the environment monitoring module, the radio frequency detection module, the electronic equipment and the communication module are integrated in the lamp or the smoke sensor.
Further, the non-sensing home control system further comprises a wireless signal receiving module for receiving other control signals acquired by sensors installed at other positions.
The intermediate frequency signals obtained by radio frequency scanning generally need to be subjected to FFT and CFAR algorithms twice to obtain point cloud information, and the calculation amount is large, and the requirement on calculation equipment is high. The method for controlling the non-inductive intelligent device provided by the invention firstly carries out preliminary judgment on whether a person exists in a controlled scene according to the collected radio frequency scanning signal, if the person exists, further operation is carried out to obtain richer human body information; if the judgment result is nobody, the judgment result is directly collected without doing more operations, so as to save the operation capability. Further, when the judgment of 'whether people exist' is made, whether a moving object exists in the scene is judged through FFT (fast Fourier transform) simple calculation, and accordingly, the judgment result is directly obtained or whether people exist in the scene is judged through point cloud calculation, so that the calculation amount can be further reduced. In the process of acquiring various information by utilizing MO, MK and MA, the invention also implements the design idea of step-by-step screening to save the computing power so as to reduce the volume of corresponding electronic equipment and control system, so that the corresponding control system has the condition of being integrated in a certain small household appliance such as a lamp or a smoke sensor.
Drawings
FIG. 1 is a flow chart of example 1;
FIG. 2 is a schematic diagram of a layout of key points of a human body;
FIG. 3 is an overall flow chart of example 2;
FIG. 4 is a partial flowchart of embodiment 2;
fig. 5 is a schematic diagram of a sliding window.
Detailed Description
The present application is further described with reference to the following figures and specific examples.
Example 1
The method for obtaining the human perception model of the models MO, MK and MA is shown in fig. 1, and may include the following steps:
s301: based on radio frequency scanning, point cloud data acquisition is carried out on a certain scene. The frequency range of the acceptable radio frequency signal is 3-90 Ghz, and the bandwidth is 500Mhz-20 Ghz. The transmission and reception of radio frequency signals can be realized by MIMO antennas pre-installed in a scene. To obtain a stereo signal, multiple sets of antennas may be laid out in the scene to obtain gridded point cloud data. And synchronously recording the scene or adopting other marking means in the radio frequency scanning process to acquire the reference information.
S302: taking a group of point cloud data obtained by N times of radio frequency scanning (N is an integer more than or equal to 2) as 1 point cloud data group, and solving point cloud information corresponding to each point cloud data group; the information corresponding to each reflection point P in the point cloud information at least includes spatial position (x, y, z), velocity v (velocity information is obtained when N is greater than or equal to 2), and signal intensity g information of the reflection point, and may further include acceleration a (acceleration information is obtained when N is greater than or equal to 3) and noise amplitude N, which are recorded as P { (x, y, z), v, g, a, N }.
Acquiring spatial position information of a target to be detected, wherein equipment is required to linearly scan a B bandwidth frequency band in a Tc time period, transmit a radio frequency signal and simultaneously receive the radio frequency signal, filter the radio frequency signal after mixing the radio frequency signal and the radio frequency signal to obtain an intermediate frequency signal, and then sample the intermediate frequency signal, wherein the scanning frequency is linearly increased to obtain an intermediate frequency signal
Figure 471781DEST_PATH_IMAGE001
Wherein τ is the time required for transmitting a signal from the device to the target to be detected to go back and forth, Tc is the time period, ƒ τ is the frequency of the received intermediate frequency signal, B is the bandwidth frequency band, and d = distance between the target to be detected and the device is obtained
Figure 475509DEST_PATH_IMAGE002
And C is the light velocity, the ƒ tau value of the reflection point is obtained by performing Fourier transform on the sampling signal, and further the distance information of the reflection point, namely the target to be detected is obtained.
Acceleration information of the target to be detected is obtained, the target to be detected is in a moving state, and the phase of the radio frequency received by twice detection can be greatly changed due to the Doppler phenomenonThe displacement between two reflecting points of the target to be detected can be obtained by phase change
Figure 995483DEST_PATH_IMAGE003
Instantaneous velocity of
Figure 451872DEST_PATH_IMAGE004
Wherein
Figure 879442DEST_PATH_IMAGE005
Is the wavelength of the radio frequency used,
Figure 347464DEST_PATH_IMAGE006
and acquiring the acceleration of the target to be detected at each reflecting point through at least three times of scanning for the phase difference of the two times of scanning. Typically, one RF scan period Tc=20~3500μs。
In order to reduce the burden of subsequent data operation, in the resolving process, point cloud data obtained by radio frequency scanning can be filtered according to data obtained by radio frequency scanning of an unmanned scene, and fixed scene information is filtered.
S303: according to reference information recorded in the data acquisition process, marking spatial position information of human body representative points corresponding to point cloud information acquired by each group of radio frequency scanning in a scene, and as an implementation mode, selecting a central point of a human body trunk as a representative point; when marking, the position of a representative point of a human body can be extracted from the reference information by using the existing artificial intelligence recognition method, and then the point cloud information is automatically marked based on the same time axis;
collecting point cloud information obtained by a plurality of groups of radio frequency scanning and corresponding human body representative point space position information to form a first sample set;
training a model MO capable of identifying the number M of the people in the scene and the spatial position of each human body representative point by utilizing a machine learning method such as a random forest, a support vector machine, AdaBoost or Gradient Tree Boosting based on a decision Tree, a neural network and the like based on the first sample set, wherein the output target of the model MO is O = { (Pr { (R) }m,Psm),m=1,2,3,……,M},PrmIs the probability, Ps, of the presence of the mth human target to be detectedmThe space position of the representative point of the mth human target to be detected, and M is the number of people in the scene. Depending on the algorithm chosen, numerical-like loss functions such as MSE, Mhattan distance between input and output values, etc., are used as evaluation methods to improve model accuracy.
S304: will NpkGroup (N)pkX times N, NpkThe number of the points is more than 1, preferably 2-25) point cloud information obtained by radio frequency scanning is regarded as a point cloud information sequence, the point cloud information sequence is filtered according to an output result of the model MO, only the point cloud information in a specific distance range and even a specific speed range (data in a human body size range are used as effective data to be beneficial to further reducing the data calculation amount) near a human body representative point is reserved, and the filtered point cloud information sequence is obtained;
selecting a plurality of key points on a human body based on human body joint points, marking the spatial position information of the human body key points corresponding to each filtered point cloud information sequence according to reference information recorded in the formation of a data set, collecting a plurality of information sequences, filtering and marking to form a second sample set; selection of human body keypoints may be referred to fig. 2, where in the embodiment shown in fig. 2, K =8, and the numbers 1-8 in fig. 2 indicate 8 human body keypoints, respectively, the torso 1 (coinciding with the human body representative point positions), the head 2, the elbows 3 and 4, the knee joints 5 and 6, and the hands 7 and 8;
training a model MK capable of identifying information of a plurality of key points of M human bodies in a scene by using a machine learning method such as a random forest, a support vector machine, AdaBoost or Gradient Tree Boosting based on a decision Tree, a neural network and the like based on a second sample set; the output target of the model MK is the key point information OK = { (Pr) of each human target to be detectedk,Psk) K =1, 2, 3, … …, K }, where K is the number of selected human body key points; pr (Pr) ofkThe probability of the k-th key point of a certain human target to be detected; pskIs the spatial position of the kth key point of a certain human target to be detected. Using a numerical class penalty function according to the selected algorithmSuch as MSE, mahhattan distance between input and output values, etc., as evaluation methods to improve model accuracy.
S305: obtaining a third sample set, wherein the third sample set comprises a positive sample and a negative sample, the positive sample comprises a posture of a specific person and a gesture (such as a fall) of the specific person obtained from the reference information, and N corresponding to the posture and the gesturemaSecond (N)maIs an integer larger than 1, preferably 18-750), and taking the rest MK output results as counter-example samples without certain specific behaviors;
training a model MA capable of identifying the posture of a specific person and the gesture of the specific person according to a plurality of point cloud information sequences by using a machine learning method, such as a random forest, a support vector machine, AdaBoost or Gradient Tree Boosting based on a decision Tree, a neural network and the like, based on a third sample set; the output target of the model MA is the probability that a certain human target to be detected has certain specific behavior. Depending on the algorithm chosen, a class loss function such as cross entropy in a neural network or Hinge in a support vector machine is used as an evaluation method to improve model accuracy.
Similarly, referring to step S305, other specific postures or gestures, such as sitting, standing, walking, running, jumping, waving, clapping, etc., are set, and parameter N is adjusted according to the duration of actionmaRepeating step S305, a plurality of models MA capable of recognizing different gestures or gestures may be obtained.
As an embodiment, Tc=1000μs,N=3,Npk=10,Nma=50, monitor fall behaviour by MA, monitor NmaThe time corresponding to the secondary data is 1500ms, which is basically the same as the time required for a fall to occur.
In order to improve the application universality of the model, different settings can be made on the scene, and different activities can be performed on different numbers of people in the scene, so that a richer sample set can be obtained.
Example 2
A sensorless smart device control method, as shown in fig. 3 and 4, includes the steps of,
S101:
acquiring a radio frequency scanning signal; acquiring environmental information including an environmental temperature and a light intensity;
S102:
judging whether a person is in a controlled scene according to the acquired radio frequency scanning signal:
acquiring a radio frequency scanning signal, processing the signal acquired through radio frequency scanning through FFT (fast Fourier transform) according to the description of the step S302 in the embodiment 1, calculating the speed of all reflection points in a scene, and searching the reflection points generating the speed to judge whether a moving object exists in the scene;
if no moving object is found and no person exists in the scene in the previous scanning, directly judging that no person exists in the scene in the current scanning (no further operation is performed on the signal acquired by the current scanning);
if a moving object is found or no moving object is found but people are in the scene in the previous scanning judgment, further resolving the radio frequency scanning signal into point cloud information, and calculating, identifying and judging whether people are in the scene based on the point cloud information by using a model MO, wherein the method comprises the following steps:
s201: acquiring point cloud information: taking a group of point cloud data obtained by N times of radio frequency scanning as 1 point cloud data group, and calculating point cloud information corresponding to each point cloud data group, wherein the information corresponding to each reflection point P in the point cloud information comprises the spatial position, the speed, the acceleration, the signal intensity and the noise amplitude information of the reflection point;
s202: inputting the point cloud information calculated in step S201 into the model MO, and outputting the point cloud information to the output destination of O = { (Pr)m,Psm),m=1,2,3,……,M},PrmIs the probability, Ps, of the presence of the mth human target to be detectedmThe spatial position of a representative point of the mth human target to be detected is shown, and M is the number of people in the scene; the spatial position of the human body representative point corresponds to the position of a person, M =0 represents no person in the scene, and M > 0 represents a person in the scene.
When the person is judged to be present, acquiring the position information of the person according to the output result of the MO; acquiring the posture and the gesture of a person through deployment models MK and MA, wherein the steps comprise:
s203: to MOJudging an output result, when the number M of human bodies output by the MO is larger than or equal to 1, filtering point cloud information obtained by radio frequency scanning according to the space position information of the representative points of the human bodies output by the model MO, only keeping the point cloud information in a specific distance range or even a specific speed range near the representative points of the human bodies, inputting the filtered point cloud information into the model MK, and starting the model MK; the model MK uses a sliding window method to scan and identify the input point cloud information, and the window length corresponds to NpkAnd performing group radio frequency scanning, and outputting a plurality of key point information of M human bodies. The principle of sliding window is shown in fig. 5, where the window length corresponds to N in the embodiment shown in fig. 5pk=10, pane width Spk=2, that is, when the model MK receives the point cloud information obtained by 4 sets of radio frequency scanning, a window initial state is formed, and corresponding to step S401, scanning identification is performed once; the MK continues to receive the point cloud information obtained by the 2 groups of radio frequency scanning, the window slides forwards for 1 time, the first 2 groups of failure information are subtracted, and 2 groups of latest information are added to form a current window, and corresponding to the step S402, the MK carries out secondary scanning identification on the current window information; the MK continues to receive the point cloud information obtained by the 2 groups of radio frequency scanning, the window slides forwards for 1 time, 2 groups of failure information in front of the window of S402 are subtracted, 2 groups of latest information are added to form a new current window again, and corresponding to the step S403, third scanning identification is carried out on the information corresponding to the new window; and in the same way, traversing all the received information. Of course, SpkMay take on a smaller value, e.g. Spk=1, or larger integer, SpkThe larger the device, the lower the computational burden, but the accuracy of the recognition is also reduced. When the human body is in a state with small activity amplitude such as sleep, S can be properly increasedpkValue of (2), e.g. Spk=N pk2=5 or Spk=Npk=10。
S204: inputting the output result of the model MK into the model MA, scanning and identifying the input information by the model MA by using a sliding window method, wherein the window length corresponds to NmaThe secondary MK continuously outputs the result, and outputs the specific human posture and the human gesture;
inputting the output of model MK to one or more models MA, model MA using slidingThe window method scans and identifies the input information, and the window length corresponds to NmaThe secondary MK continuously outputs the result and outputs a specific human body posture or gesture. The operation principle of the sliding window is the same as that described in S40, but as an embodiment, the window width S is the same as that of step S50maPreferably Nma/2 (in agreement with example 1, N)ma= 50). It will be understood that SmaThe accuracy can be obtained by calculating the quantity; or increased, possibly traded for speed with a sacrifice in accuracy. The models MA used for recognizing a plurality of different human body postures or gestures run synchronously, each model MA carries out scanning recognition at respectively set intervals, and the probability of different behaviors is deduced and recognized and output.
Further, a step of judging the human behavior type output by the model MA can be added, and the window width S is given according to different human behaviorspkAppointing values and feeding back to step S203, and giving window pane width S according to different human body behaviorsmaA value is assigned and fed back to step S204.
The acquisition of physiological information of a person has been described in a number of prior art methods, for example, with reference to CN 109729632A.
S103:
Collecting control information, wherein the control information comprises a judgment result of whether a person exists in a previous scanning scene, a judgment result of whether a person exists in a current scanning scene, and human body information and environment information acquired when the person exists in the current scanning scene;
S104:
for each type of control information, acquiring the controlled equipment registered with the type of control information, distributing the type of control information to each registered controlled equipment control information group, and then respectively matching each group of information: and matching each control information group with the corresponding controlled equipment according to the preset corresponding relation between the control information and the control instruction of the controlled equipment, determining the corresponding control instruction and outputting the control instruction to the corresponding controlled equipment.
As another embodiment, each type of control information is distributed to the control module of the controlled device in which the controlled information is registered, and the control module of the controlled device matches the control instruction and sends the control instruction to the execution mechanism.
Example 3
An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method described in embodiment 2.
Example 4
An sensorless home control system comprising:
the environment monitoring module is used for monitoring the environment temperature and the light intensity to obtain environment information; the radio frequency detection module is used for carrying out radio frequency scanning on a controlled scene to acquire a radio frequency scanning signal; the electronic device of embodiment 3, configured to obtain the acquisition environment information; the system comprises a radio frequency scanning device, a data acquisition module and a data processing module, wherein the radio frequency scanning device is used for acquiring a radio frequency scanning signal, resolving the radio frequency scanning signal into point cloud information, and performing point cloud calculation based on the point cloud information to acquire human body information; determining an equipment control instruction according to control information containing environment information and human body information; and the communication module is used for sending the control instruction determined by the electronic equipment to the control module of the controlled equipment.
The environment monitoring module, the radio frequency detection module, the electronic equipment and the communication module are integrated in the lamp or the smoke sensor.
The controlled device may include: electric lights, air conditioners, door locks, switch cabinets, patch panels, speakers, horns, smoke detectors, showers, curtains, bathroom heaters, computers, electric fans, cameras, television boxes, routers, electric windows, and the like.
In the following, taking the control of an intelligent lamp, an air conditioner and a switch cabinet (especially, the switch cabinet does not need to be normally opened, or called an energy-saving control switch) as an example, a reference setting is provided for the distribution and matching of control information:
firstly, equipment numbering:
1-intelligent lamp, 2-air conditioner, 3-energy-saving control switch
II, control information classification numbering and description:
0-environmental information, 1-presence or absence, 2-position of person, 3-posture of person, 4-physiological information of person, 5-gesture of person.
0-environmental information provides values of ambient temperature, light intensity, etc.; 1-the presence or absence information provides the presence or absence state of the whole family space; 2-the position information of the person is used for distinguishing different positions of the person on the sofa, the bed and the like; 3, distinguishing the postures of the person such as sitting, standing (including standing/walking/running and other standing postures), lying or falling by the posture information of the person; 4-the physiological information of the person provides a value of the breathing or heartbeat frequency of the person; the 5-human gesture refers to a preset specific gesture expressing a specific control instruction, such as waving a hand, drawing a circle in space, and the like. When the respiratory or heartbeat frequency value of the person reaches a low threshold value range for a period of time, indicating that the person enters a sleep state; when the respiratory or heartbeat frequency value of the person reaches a high threshold value range for a period of time, the person is in a waking activity state; if the threshold range is exceeded, the person is considered to be in an unhealthy hazardous state. If people exist in the scanning space of the previous round and no people exist in the scanning space of the current round, the system considers that people just leave; if no person exists in the scanning space of the previous round, and a person exists in the scanning space of the current round, the person is considered to have just entered.
The control information obtained from the previous scanning round is recorded in the database, and if necessary, the control information obtained from the previous scanning round is stored in the database, retrieved and compared.
After the six kinds of control information are output, how to combine and correspond to a specific control instruction can be preset according to the needs of the controlled equipment. For example, when the controlled device is a 3-energy-saving control switch, only the presence or absence of a person in a room (1-presence or absence) needs to be known, and the position information of a 2-person does not need to be known, so that the device 3-energy-saving control switch only needs to register the control information 1-presence or absence. When the controlled device is a 1-intelligent lamp, the user may request to be very dark (0-environmental information) in the room, the user lies (3-human posture) on the bed (2-human position) and the heart rate and breathing (4-human physiological information) frequency decreases, and after 15 minutes, the light is dimmed, so that the control is more accurate. Even requiring the user to make a specific gesture (a 5-person gesture) to issue an active control instruction. And the control information 1, namely the existence of the person, is registered, and the states of the person coming in or the person leaving and the like can be quickly known by comparing the existence information in the time space before and after, so that control instructions of turning on or turning off the light and the like can be conveniently given.
Table 1 shows one way of registering different devices and control information, such as 1-smart lights, 2-air conditioners, 3-energy saving control switches, etc.
Figure 162973DEST_PATH_IMAGE007
Table 1-control information for smart lamp registration: 0-environmental information (light intensity), 1-presence or absence, 2-position of person, 3-posture of person, 4-physiological information of person, 5-gesture of person; 2-information of air conditioner registration: 0-environmental information (temperature), 1-presence or absence, 2-location of person, 4-physiological information of person; 3-energy saving control switch registered control information only: 1-existence or non-existence of human. When some kind of control information is acquired, after the system inquires the equipment number through the table 1, whether to distribute the control signal acquired by the scanning in the current round is judged according to the control frequency and the sending time; if the time interval has not yet arrived, it needs to be sent again. Thus, the controlled equipment can adjust the control time according to the requirement.
The control information is distributed to each registered controlled device control information group according to the table 1, and then the information of each group is matched: and matching each control information group with the corresponding controlled equipment according to the preset corresponding relation between the control information and the control instruction of the controlled equipment, determining the corresponding control instruction, outputting the control instruction to the corresponding controlled equipment, and executing the corresponding instruction by the controlled equipment.
Taking an energy-saving control switch as an example, the matching relationship of the control commands may be: when the current round of information given by the acquired control information 1- "existence or nonexistence" item is "existence" and the previous round of information is "nonexistence", matching a control instruction of "opening a control circuit"; and when the current round of information given by the acquired control information 1- 'existence of people' item is 'no people' and the previous round of information is 'existence of people', matching the control instruction of 'closing the control circuit'. Other information is ignored.
Taking an intelligent air conditioner with an automatic switch, a portable air blower and an automatic temperature regulation function during sleeping as an example, the matching relation of control instructions can be as follows: when the current round of information given by the acquired control information 1- 'existence or not' item is 'existence' and the previous round of information is 'no-people', matching a control instruction of 'starting an air conditioner'; and when the current round of information given by the acquired control information 1- 'existence of people' item is 'no people' and the previous round of information is 'existence of people', matching a control instruction of 'closing the air conditioner'.
And 2, the position information corresponds to a control instruction of adjusting the air blowing direction of the air conditioner.
2-position information gives information that the person is in bed, 4-physiological information gives that the heart rate or breath of the person becomes slow and reaches a certain time; at the moment, if the air conditioner is in a refrigeration mode, matching a control instruction of 'increasing temperature'; at this time, if the air conditioner is in the heating mode, the control instruction of 'turn down the temperature' is matched.
Other information is ignored.

Claims (11)

1. A non-sensing intelligent device control method is characterized by comprising the following steps,
s101: acquiring a radio frequency scanning signal;
s102: judging whether a person is in a controlled scene or not based on the radio frequency scanning signal, and calculating to obtain any one or more of the following human body information: a position of the person, a posture of the person, physiological information of the person;
s103: collecting and storing control information, wherein the control information comprises whether a person exists in a scene or not and human body information acquired when the person exists in the scene;
s104: and matching the collected control information according to the preset corresponding relation between the control information and the control instruction of the intelligent equipment, determining the corresponding control instruction and outputting the control instruction.
2. The sensorless intelligent device control method of claim 1, wherein the human body information further comprises human gestures.
3. The method for controlling the sensorless intelligent device according to claim 1 or 2, wherein step S102 is to determine whether there is a person in the controlled scene according to the rf scanning signal; and when the person is judged to be present, the human body information is obtained based on the point cloud calculation.
4. The sensorless intelligent device control method according to claim 3, wherein the process of step S102 determining whether there is a person in the scene is:
performing FFT signal processing on the acquired radio frequency scanning signal to judge whether a moving object exists in a scene;
if no moving object is found and no person exists in the scene in the previous scanning, directly judging that no person exists in the scene in the current scanning;
if a moving object is found or the moving object is not found but people are judged in the scene in the previous scanning, the radio frequency scanning signal is further resolved into point cloud information, and whether people exist in the scene is judged based on the point cloud information.
5. The sensorless intelligent device control method according to claim 4, wherein the step S102 judges whether there is a person based on the point cloud information by using a machine learning model MO obtained by training, including,
s201: acquiring point cloud information: taking a group of point cloud data obtained by N times of radio frequency scanning as 1 point cloud data group, and calculating point cloud information corresponding to each point cloud data group, wherein the information corresponding to each reflection point in the point cloud information at least comprises the spatial position, the speed and the signal intensity information of the reflection point;
s202: inputting the point cloud information calculated in step S201 into the model MO, and outputting the point cloud information to the output destination of O = { (Pr)m,Psm),m=1,2,3,……,M},PrmIs the probability, Ps, of the presence of the mth human target to be detectedmThe spatial position of a representative point of the mth human target to be detected is shown, and M is the number of people in the scene;the space position of the human body representative point corresponds to the position of a person, M =0 represents that no person exists in the scene, and M > 0 represents that a person exists in the scene;
the model MO is obtained through the following steps:
s301: performing point cloud data acquisition on a scene based on radio frequency scanning;
s302: taking a group of point cloud data obtained by N times of radio frequency scanning as 1 point cloud data group, and calculating point cloud information corresponding to each point cloud data group, wherein the information corresponding to each reflection point in the point cloud information at least comprises the spatial position, the speed and the signal intensity information of the reflection point; n is an integer greater than or equal to 2;
s303: marking spatial position information of human body representative points corresponding to point cloud information obtained by each group of radio frequency scanning in a scene according to reference information recorded in the data acquisition process; collecting point cloud information obtained by a plurality of groups of radio frequency scanning and corresponding human body representative point space position information to form a first sample set; and training a model MO capable of identifying the number M of the people in the scene and the spatial position of each human body representative point by using a machine learning method based on the first sample set.
6. The sensorless smart device control method of claim 5, wherein the posture of the person and the gesture of the person are obtained using models MK and MA obtained by training according to the result of the MO output, including,
s203: when the number M of the human bodies output by the MO is larger than or equal to 1, filtering point cloud information obtained by radio frequency scanning according to the spatial position information of the human body representative points output by the model MO, and only keeping the point cloud information of a specific distance range near the human body representative points; inputting the filtered point cloud information into a model MK, scanning and identifying the input point cloud information by the model MK by using a sliding window method, wherein the window length corresponds to NpkGroup radio frequency scanning is carried out, and a plurality of key point information of M human bodies is output;
s204: inputting the output result of the model MK into the model MA, scanning and identifying the input information by the model MA by using a sliding window method, wherein the window length corresponds to NmaThe sub-MK continuously outputs the result, and outputs the posture of the specific personA gesture of a person and a state;
continuing to execute step S304 on the basis of the training of the model MO to obtain a model MK:
will NpkThe point cloud information obtained by the group radio frequency scanning is regarded as a point cloud information sequence, the point cloud information sequence is filtered according to the output result of the model MO, only the point cloud information in a specific distance range near the human body representative point is reserved, and the filtered point cloud information sequence is obtained; n is a radical ofpkIs an integer greater than 1;
selecting a plurality of key points on a human body based on human body joint points, marking the spatial position information of the human body key points corresponding to each filtered point cloud information sequence according to reference information recorded in the formation of a data set, collecting a plurality of information sequences, filtering and marking to form a second sample set;
training a model MK capable of identifying a plurality of key point information of M human bodies in a scene by using a machine learning method based on the second sample set; the output target of the model MK is key point information OK = { (Pr) of each human body target to be detectedk,Psk) K =1, 2, 3, … …, K }, where K is the number of selected human body key points; pr (Pr) ofkThe probability of the k-th key point of a certain human target to be detected; pskThe spatial position of the kth key point of a certain human body target to be detected exists;
continuing to execute step S305 on the basis of the training of the model MK to obtain the model MA:
obtaining a third sample set, wherein the third sample set comprises a positive sample and a negative sample, the positive sample comprises a specific human body posture or gesture obtained from the reference information, and N corresponding to the specific human body posture or gesturemaContinuously outputting results of the secondary MK, and taking other output results of the secondary MK as counterexample samples without certain specific human body posture or gesture; n is a radical ofmaIs an integer greater than 1;
training a model MA capable of identifying a specific human body posture or gesture according to a plurality of point cloud information sequences by using a machine learning method based on a third sample set; the output target of the model MA presents the probability of a certain specific gesture or gesture for a certain human body target to be detected.
7. The sensorless smart device control method of claim 1 further comprising the step of obtaining environmental information and collecting the environmental information as control information, the environmental information including ambient temperature and/or light intensity.
8. The method of claim 1, wherein in step S105, for each type of control information, the controlled device that has registered the type of control information is acquired, the type of control information is distributed to each registered controlled device control information group, each control information group is matched with the corresponding controlled device according to a preset correspondence between the control information and the controlled device control instruction, the corresponding control instruction is determined, and the intelligent device is controlled by the determined control instruction.
9. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 or 2 or 4-8.
10. An sensorless home control system comprising:
the environment monitoring module is used for monitoring the environment temperature and/or the light intensity to obtain environment information;
the radio frequency detection module is used for carrying out radio frequency scanning on a controlled scene to acquire a radio frequency scanning signal;
the electronic device of claim 9, configured to obtain environmental information; the system comprises a radio frequency scanning signal acquisition unit, a display unit and a display unit, wherein the radio frequency scanning signal acquisition unit is used for acquiring a radio frequency scanning signal, judging whether a person exists in a scene according to the radio frequency scanning signal and acquiring various human body information; determining an equipment control instruction according to control information including environment information, and information about whether a person and a human body exist;
and the communication module is used for sending the control instruction determined by the electronic equipment to the control module of the controlled equipment.
11. The sensorless home control system of claim 10, wherein the environmental monitoring module, the radio frequency detection module, the electronics, and the communication module are integrated into a light fixture or a smoke sensor.
CN202010760967.5A 2020-07-31 2020-07-31 Non-sensing intelligent device control method, electronic device and control system Pending CN114063572A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010760967.5A CN114063572A (en) 2020-07-31 2020-07-31 Non-sensing intelligent device control method, electronic device and control system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010760967.5A CN114063572A (en) 2020-07-31 2020-07-31 Non-sensing intelligent device control method, electronic device and control system

Publications (1)

Publication Number Publication Date
CN114063572A true CN114063572A (en) 2022-02-18

Family

ID=80227711

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010760967.5A Pending CN114063572A (en) 2020-07-31 2020-07-31 Non-sensing intelligent device control method, electronic device and control system

Country Status (1)

Country Link
CN (1) CN114063572A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116991089A (en) * 2023-09-28 2023-11-03 深圳市微琪思网络有限公司 Intelligent control method and system for electric iron based on wireless connection
WO2023236848A1 (en) * 2022-06-08 2023-12-14 深圳绿米联创科技有限公司 Device control method, apparatus and system, and electronic device and readable storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090190798A1 (en) * 2008-01-25 2009-07-30 Sungkyunkwan University Foundation For Corporate Collaboration System and method for real-time object recognition and pose estimation using in-situ monitoring
CN109484935A (en) * 2017-09-13 2019-03-19 杭州海康威视数字技术股份有限公司 A kind of lift car monitoring method, apparatus and system
CN110045370A (en) * 2019-05-10 2019-07-23 成都宋元科技有限公司 Human perception method and its system based on millimetre-wave radar
CN110632849A (en) * 2019-08-23 2019-12-31 珠海格力电器股份有限公司 Intelligent household appliance, control method and device thereof and storage medium
CN110686376A (en) * 2019-09-18 2020-01-14 珠海格力电器股份有限公司 Air conditioner and fan combined control method based on human body sleeping posture recognition, computer readable storage medium and air conditioner
CN110728213A (en) * 2019-09-26 2020-01-24 浙江大学 Fine-grained human body posture estimation method based on wireless radio frequency signals
CN111126314A (en) * 2019-12-26 2020-05-08 杭州中科先进技术研究院有限公司 Passenger flow statistical visual method and system based on 3D point cloud data
CN111360819A (en) * 2020-02-13 2020-07-03 平安科技(深圳)有限公司 Robot control method and device, computer device and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090190798A1 (en) * 2008-01-25 2009-07-30 Sungkyunkwan University Foundation For Corporate Collaboration System and method for real-time object recognition and pose estimation using in-situ monitoring
CN109484935A (en) * 2017-09-13 2019-03-19 杭州海康威视数字技术股份有限公司 A kind of lift car monitoring method, apparatus and system
CN110045370A (en) * 2019-05-10 2019-07-23 成都宋元科技有限公司 Human perception method and its system based on millimetre-wave radar
CN110632849A (en) * 2019-08-23 2019-12-31 珠海格力电器股份有限公司 Intelligent household appliance, control method and device thereof and storage medium
CN110686376A (en) * 2019-09-18 2020-01-14 珠海格力电器股份有限公司 Air conditioner and fan combined control method based on human body sleeping posture recognition, computer readable storage medium and air conditioner
CN110728213A (en) * 2019-09-26 2020-01-24 浙江大学 Fine-grained human body posture estimation method based on wireless radio frequency signals
CN111126314A (en) * 2019-12-26 2020-05-08 杭州中科先进技术研究院有限公司 Passenger flow statistical visual method and system based on 3D point cloud data
CN111360819A (en) * 2020-02-13 2020-07-03 平安科技(深圳)有限公司 Robot control method and device, computer device and storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023236848A1 (en) * 2022-06-08 2023-12-14 深圳绿米联创科技有限公司 Device control method, apparatus and system, and electronic device and readable storage medium
CN116991089A (en) * 2023-09-28 2023-11-03 深圳市微琪思网络有限公司 Intelligent control method and system for electric iron based on wireless connection
CN116991089B (en) * 2023-09-28 2023-12-05 深圳市微琪思网络有限公司 Intelligent control method and system for electric iron based on wireless connection

Similar Documents

Publication Publication Date Title
CN105357842B (en) Master control intelligent lamp
CN105446162B (en) A kind of intelligent home furnishing control method of smart home system and robot
CN105785782B (en) Intelligent home equipment control method and device
CN109974235B (en) Method and device for controlling household appliance and household appliance
Kim et al. Resident location-recognition algorithm using a Bayesian classifier in the PIR sensor-based indoor location-aware system
CN107576022A (en) Control method, air conditioner and the storage medium of air conditioner
CN205581626U (en) Steerable intelligent household equipment's intelligent floor
CN108231079A (en) For the method, apparatus, equipment and computer readable storage medium of control electronics
CN109945421A (en) Progress control method, operating control device and household appliance
CN109489212B (en) Intelligent sleep control method, adjustment system and equipment for air conditioner
CN105223826B (en) Home equipment control method, apparatus and system
CN106325250A (en) Electric device coordinated control method and electric device coordinated control system based on information detection
CN106444415A (en) Smart home control method and system
CN109631245A (en) Air conditioning control method, device, storage medium and processor
CN105045140A (en) Method and device for intelligently controlling controlled equipment
CN108717271A (en) Man-machine interaction control method, device, system and readable storage medium storing program for executing
CN114063572A (en) Non-sensing intelligent device control method, electronic device and control system
US20180363937A1 (en) Control apparatus, control method, and program
CN106556111A (en) Household electric appliance control method, device, server and air-conditioner
CN110426962A (en) A kind of control method and system of smart home device
CN109916010A (en) Progress control method, module, household appliance, system and computer storage medium
CN107229262A (en) A kind of intelligent domestic system
CN105094298A (en) Terminal and terminal based gesture recognition method
CN108549251A (en) A kind of the self study user custom control system and its control method of smart home
CN115345908A (en) Human body posture recognition method based on millimeter wave radar

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination