CN115223089A - Children dangerous behavior detection method and device, intelligent terminal and storage medium - Google Patents

Children dangerous behavior detection method and device, intelligent terminal and storage medium Download PDF

Info

Publication number
CN115223089A
CN115223089A CN202210712422.6A CN202210712422A CN115223089A CN 115223089 A CN115223089 A CN 115223089A CN 202210712422 A CN202210712422 A CN 202210712422A CN 115223089 A CN115223089 A CN 115223089A
Authority
CN
China
Prior art keywords
detection
risk
child
target
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210712422.6A
Other languages
Chinese (zh)
Other versions
CN115223089B (en
Inventor
罗书立
史宇航
刘洪海
林熹
任卫红
李素芳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Graduate School Harbin Institute of Technology
Shenzhen Childrens Hospital
Original Assignee
Shenzhen Graduate School Harbin Institute of Technology
Shenzhen Childrens Hospital
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Graduate School Harbin Institute of Technology, Shenzhen Childrens Hospital filed Critical Shenzhen Graduate School Harbin Institute of Technology
Priority to CN202210712422.6A priority Critical patent/CN115223089B/en
Publication of CN115223089A publication Critical patent/CN115223089A/en
Application granted granted Critical
Publication of CN115223089B publication Critical patent/CN115223089B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/52Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10Complex mathematical operations
    • G06F17/18Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/004Artificial life, i.e. computing arrangements simulating life
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0635Risk analysis of enterprise or organisation activities
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0639Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06393Score-carding, benchmarking or key performance indicator [KPI] analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/46Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • GPHYSICS
    • G08SIGNALLING
    • G08BSIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02Alarms for ensuring the safety of persons
    • G08B21/04Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
    • G08B21/0407Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis
    • G08B21/043Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis detecting an emergency event, e.g. a fall
    • GPHYSICS
    • G08SIGNALLING
    • G08BSIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02Alarms for ensuring the safety of persons
    • G08B21/04Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
    • G08B21/0438Sensor means for detecting
    • G08B21/0469Presence detectors to detect unsafe condition, e.g. infrared sensor, microphone
    • GPHYSICS
    • G08SIGNALLING
    • G08BSIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02Alarms for ensuring the safety of persons
    • G08B21/04Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
    • G08B21/0438Sensor means for detecting
    • G08B21/0476Cameras to detect unsafe condition, e.g. video cameras
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07Target detection
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02ATECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • Emergency Management (AREA)
  • Mathematical Physics (AREA)
  • Operations Research (AREA)
  • Evolutionary Computation (AREA)
  • Development Economics (AREA)
  • Educational Administration (AREA)
  • Software Systems (AREA)
  • Gerontology & Geriatric Medicine (AREA)
  • Game Theory and Decision Science (AREA)
  • Computational Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Human Computer Interaction (AREA)
  • Marketing (AREA)
  • Social Psychology (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Pure & Applied Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Psychiatry (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)

Abstract

The invention discloses a method and a device for detecting dangerous behaviors of children, an intelligent terminal and a storage medium, wherein the method comprises the following steps: monitoring a target area in real time to obtain video data; inputting the video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area; acquiring pre-stored clinical data of a target child; obtaining a clinical risk based on the clinical data; calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk; obtaining a risk score according to a risk score calculation model based on the clinical risk and the detected risk; and when the danger score exceeds a set threshold value, outputting warning information. Compared with the prior art, the method has the advantages that the children dangerous behaviors are judged on the low-dimensional data through simple calculation, the data processing is simple, the speed is high, and the target area can be monitored in real time.

Description

Children dangerous behavior detection method and device, intelligent terminal and storage medium
Technical Field
The invention relates to the technical field of intelligent monitoring, in particular to a method and a device for detecting dangerous behaviors of children, an intelligent terminal and a storage medium.
Background
In the pediatric hospitalization, as children are lively and active and lack of identification capability for dangerous environments, dangerous behaviors such as falling down and falling into a bed are easy to happen, physical and psychological health of the children is affected, and even medical care disputes are brought. And the workload of nurses is large, so that each child is difficult to intervene in time. Whether dangerous behaviors such as falling off a bed, touching a kettle and the like exist in the ward of the child or not is judged through online monitoring, and early warning is timely performed in advance so as to reduce secondary injuries of the child in the ward of the child hospital.
The existing abnormal detection system calculates the key points of limbs based on depth information, and judges whether abnormal behaviors exist or not according to the movement speed and the movement frequency of the key points of the limbs and the fact whether the key points cross the boundary or not. The detection processing content is complex, the detection speed is slow, and real-time detection cannot be realized.
Thus, the prior art is in need of improvement and enhancement.
Disclosure of Invention
The invention mainly aims to provide a method and a device for detecting dangerous behaviors of children, an intelligent terminal and a storage medium, and aims to solve the problems that an abnormality detection system in the prior art is low in processing speed and cannot detect in real time.
In order to achieve the above object, a first aspect of the present invention provides a method for detecting dangerous behaviors of children, wherein the method comprises:
monitoring a target area in real time to obtain video data;
inputting the video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
acquiring pre-stored clinical data of a target child;
obtaining a clinical risk based on the clinical data;
calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
obtaining a risk score according to a risk score calculation model based on the clinical risk and the detected risk;
and when the danger score exceeds a set threshold value, outputting warning information.
Optionally, the associating objects include people and objects associated with dangerous behaviors of children in the target area, and the inputting the video data into the target detection model to obtain the detection data of the detection object includes:
inputting the video data into the target detection model to obtain a preset number of image frames;
sequentially acquiring the area and the position of the target child and the area and the position of the associated object in each image frame;
and saving the area and the position of the target child and the area and the position of the associated object based on the sequence of the image frames to obtain the detection data.
Optionally, the calculating, based on the detection data, an area intersection ratio of the target child and the associated object to obtain a detection risk includes:
obtaining an area intersection comparison sequence of the target child and the associated object according to the position and the area of the target child and the position and the area of the associated object;
obtaining the variation trend of the area intersection ratio of the target child and the associated object based on the area intersection ratio sequence;
obtaining the detection risk based on all the variation trends.
Optionally, the expression of the risk score calculation model is as follows:
y=(ω 1 f 1 (a,g,d,m)+b 1 )×[(ω 2 max{x 1 ,x 2 }+b 2 )×f 2 (x 3 ,x 4 ,x 5 ,t)+b 3 ]f 3 (t),
wherein, w 1 ,w 2 A, g, d and m are weight coefficients, the age, sex, dangerous behavior history and medication information of the target child are respectively, and b 1 ,b 2 ,b 3 Is a constant offset, x 1 ,x 2 Area intersection ratio, x, between the target child and the parents, respectively, the physician 3 ,x 4 ,x 5 The area intersection ratio of the target child to the sickbed, the chair and the bedside table, t is duration, f 1 For the clinical risk calculation formula, f 2 Formula for risk detection calculation, f 3 Is a duration weight formula.
Optionally, the clinical data includes: age, sex, history of dangerous behavior and medication information of the target child, based on the clinical data, obtaining clinical risks comprising:
respectively acquiring an expected value and a standard deviation of each item of data in the clinical data;
and obtaining the clinical risk according to the clinical data, the expected value and the standard deviation based on the preset physiological information weight and the preset medical information weight.
Optionally, after obtaining the risk score according to the risk score calculation model, the method further includes:
obtaining an assessment score according to a child fall assessment scale based on the clinical data;
and performing weighted accumulation on the evaluation score and the danger score and updating the danger score.
Optionally, the monitoring the target area in real time to obtain video data includes:
and synchronously acquiring two video data of the target area, and preprocessing the two video data to acquire the video data.
In a second aspect, the present invention provides a device for detecting dangerous behaviors of children, wherein the device comprises:
the video data acquisition module is used for monitoring a target area in real time to acquire video data;
the detection data acquisition module is used for inputting the video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
the clinical data acquisition module is used for acquiring pre-stored clinical data of the target child;
a clinical risk calculation module for obtaining a clinical risk based on the clinical data;
the detection risk calculation module is used for calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
a risk score calculation module for obtaining a risk score according to a risk score calculation model based on the clinical risk and the detection risk;
and the alarm module is used for outputting alarm information when the danger score exceeds a set threshold value.
A third aspect of the present invention provides an intelligent terminal, where the intelligent terminal includes a memory, a processor, and a child dangerous behavior detection program stored in the memory and executable on the processor, and the child dangerous behavior detection program implements any one of the steps of the child dangerous behavior detection method when executed by the processor.
A fourth aspect of the present invention provides a computer-readable storage medium, where a child risk behavior detection program is stored on the computer-readable storage medium, and when being executed by a processor, the child risk behavior detection program implements any one of the steps of the child risk behavior detection method.
Therefore, compared with the prior art, the method and the device have the advantages that the target detection model is adopted to analyze video data, the target child and the related objects related to the child dangerous behaviors in the target area are identified, the area intersection ratio of the target child and the related objects is calculated, the detection risk is obtained according to the area intersection ratio, the clinical risk is obtained according to the clinical data of the target child, then the danger score calculation model is adopted to comprehensively score, the child dangerous behaviors are identified, and early warning is carried out. The method and the device have the advantages that the children dangerous behaviors are judged on the low-dimensional data through simple calculation, the data processing is simple, the speed is high, and the real-time monitoring on the target area can be realized.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the embodiments or the prior art description will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive labor.
FIG. 1 is a schematic flow chart of a method for detecting dangerous behaviors of children according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating the detailed step of step S500 in the embodiment of FIG. 1;
fig. 3 is a schematic structural diagram of a device for detecting dangerous behaviors of children according to an embodiment of the present invention;
fig. 4 is a schematic block diagram of an internal structure of an intelligent terminal according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when 8230," or "once" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted depending on the context to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings of the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
For children patients, falling and falling from bed are one of common accidents, and serious falling or falling from bed not only injures the children patients themselves to aggravate the illness condition, but also easily causes doctor-patient disputes. The prevention of falling and falling of children is one of the quality indexes of hospital safety target management.
The existing anomaly detection system cannot effectively and reliably detect children and early warn dangerous actions of the children, meanwhile, the detection processing content of the human skeleton points calculated based on depth information is large, the general detection speed is low, and real-time detection cannot be achieved.
Therefore, the invention provides a method for detecting dangerous behaviors of children, which comprises the steps of identifying a detection object in video data through a target detection model, then calculating an area intersection ratio, dynamically identifying dangerous behaviors, comprehensively judging the existing dangerous behaviors by combining input clinical data of children, and giving an alarm in time. The detection speed is high, the precision is high, and the dangerous behaviors of children can be detected in real time.
Exemplary method
As shown in fig. 1, an embodiment of the present invention provides a method for detecting dangerous behaviors of children, which is configured in a system of a hospital and is used for identifying dangerous behaviors of children in a ward in real time. Specifically, the method comprises the following steps:
step S100: monitoring a target area in real time to obtain video data;
specifically, this embodiment is used for carrying out real-time inspection to the dangerous behavior of children in the children ward. Correspondingly, the monitored target area is a child ward, and the video data of the child ward are acquired in real time by installing a red, green and blue (RGB) camera on the wall of the child ward. The red, green and blue camera can represent the object according to the shape and the color of the object, and is suitable for target identification occasions.
It should be noted that the method of the present invention can also be used in other areas in a hospital, such as an infusion area, a waiting area, etc. Even for real-time assessment of child-risk behavior in a home room.
Step S200: inputting video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
the target detection model is a neural network model, key frames can be obtained from the obtained video data, each key frame image is automatically identified, and people and objects in the key frame images are identified. Because the invention is used for monitoring and judging the dangerous behaviors of the children, the invention identifies the target children, the people and objects related to the dangerous behaviors of the children in the target area, and other detection objects according to the target detection model. The Yolov5n neural network has simple network architecture and high calculation speed, and is particularly suitable for rapid identification and detection, so the target detection model is preferably the Yolov5n network model.
Specifically, video data obtained through monitoring is input into a target detection model, and a preset number of image frames output by the target detection model are obtained; then, sequentially acquiring position coordinate information of the target child and position coordinate information of the associated object in each image frame; and carrying out target area calculation on the identified target children and the associated objects to obtain corresponding area information, and storing the position coordinates and the area information according to the sequence of the image frames to obtain detection data.
In the embodiment, two red, green and blue cameras are installed, video images of a child on a bed and near the bed in a ward are synchronously acquired, the pixel size of the video image is adjusted to 640 multiplied by 640 and then the video image is input into a Yolov5n network model according to the input image requirement of the Yolov5n network model, and the position coordinate and the area information of the child on the current sickbed in an image frame are output in real time, and similarly, the position coordinate and the area information of related objects such as parents, doctors/nurses, beds, chairs and bedside tables are also output.
In order to improve the accuracy of recognition, the present embodiment also trains the target detection model in advance. Based on the collected clinical dangerous behavior data set, according to the prior knowledge of dangerous actions in medicine (including the fact that the medical dangerous behavior data set is close to a bed edge without guardrails, the medical dangerous behavior data set is used for climbing a bed fence for a low-age child, getting on or off the bed by oneself and the like), the marked ward target position data is used for training a target detection model.
Optionally, the target detection model may also be combined with an onnx (Open Neural Network Exchange) model to accelerate the speed and efficiency of target detection.
Step S300: acquiring pre-stored clinical data of a target child;
specifically, each time a child enters the ward, clinical information of the target child is input on an input interface of the hospital information system, such as: the age, height, sex, health status, medication condition and history of dangerous behaviors of the children. Such clinical data may be obtained from a hospital information system.
Step S400: obtaining a clinical risk based on the clinical data;
specifically, first, an expected value and a standard deviation of each item of data in the clinical data are respectively obtained; and according to the clinical data, the expected value and the standard deviation, performing weighted calculation based on preset physiological information weight related to age and gender and preset medical information weight related to dangerous behavior history and medication information to obtain clinical risk. The formula for calculating clinical risk in this embodiment is:
Figure BDA0003708558100000081
wherein rho is a correlation coefficient of the dangerous behavior history and the medication information; sigma a σ g σ d σ m Respectively the age, sex and risk of childrenStandard deviation parameters for history and medication information; a, g, d and m are respectively age, sex, dangerous behavior history and medication information of the target child; a is 0 g 0 d 0 m 0 Respectively the expected parameters of the age, sex, dangerous behavior history and medication information of the children; lambda 1 To a physiological information weight; lambda 2 Is a medical information weight.
Step S500: calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
specifically, the area intersection ratio of the target child and the associated object can be calculated according to the coordinate position and the area of the target child and the coordinate position and the area of the associated object in the detection data. The specific calculation formula is as follows:
Figure BDA0003708558100000082
wherein p is 0 Is the location area of the child, p 1 ,p 2 ,p 3 ,p 4 ,p 5 The location areas of the parents, the doctor, the hospital bed, the chair and the bedside table, respectively. According to the time sequence of the image frames, after area intersection ratios of different time sequences are obtained, the change trend of the area intersection ratios is obtained, whether behaviors of the target children are dangerous or not is judged in advance, and then the detection risk is obtained. For example: the target child intersects with the area of the bed and the area ratio becomes smaller, which indicates that the target child has the possibility of falling into the bed.
In this embodiment, a plurality of associated objects are provided, and a transformation trend of the area intersection ratio between the target child and each associated object is obtained first, and then comprehensive judgment is performed to obtain the detection risk. For example: the intersection ratio of the target child to the hospital bed is smaller, and the intersection ratio to the area of the parent or the doctor is smaller, which means that the parent or the doctor is far away from the target child, and the target child may not have time to rescue when falling down, that is, the target child has the risk of falling down.
Step S600: based on the clinical risk and the detection risk, obtaining a risk score according to a risk score calculation model;
step S700: and when the danger score exceeds a set threshold value, outputting alarm information.
Specifically, after clinical risk and detection risk are obtained, weighted calculation is performed by combining the duration of the current behavior of the infant patient, and a risk score is obtained. When the danger score exceeds a set threshold, meaning that the target child is at risk, an alarm signal is output to prompt a nurse or a parent. For example: various alarm modes such as voice prompt, red indicator light display on the sickbed and the like can be adopted.
The present invention performs the above-described weighting calculation according to the risk score calculation model. The specific expression of the risk score calculation model is as follows:
y=(ω 1 f 1 (a,g,d,m)+b 1 )×[(ω 2 max{x 1 ,x 2 }+b 2 )×f 2 (x 3 ,x 4 ,x 5 ,t)+b 3 ]f 3 (t),
wherein, w 1 ,w 2 A, g, d and m are weight coefficients, the age, sex, dangerous behavior history and medication information of the target child are respectively, and b 1 ,b 2 ,b 3 Is a constant offset, x 1 ,x 2 Area intersection ratio, x, between the target child and the parents, respectively, the physician 3 ,x 4 ,x 5 The area intersection ratio of the target child to the sickbed, the chair and the bedside table, t is duration, f 1 For the clinical risk calculation formula, f 2 Formula for risk detection calculation, f 3 Is a duration weight formula.
Figure BDA0003708558100000091
Wherein λ 4 Influencing a weight parameter for the duration; xi is a time-influencing bias constant; t is t i For the activation time
Figure BDA0003708558100000092
It should be noted that, the number and content of the area intersection ratio parameters in the risk score calculation model may be changed correspondingly, if the associated objects are different. If the associated object comprises a kettle, the early warning can be performed on dangerous behaviors such as boiling water scalding.
Further, in some embodiments, after the risk score is obtained according to the risk score calculation model, the evaluation score can be obtained by scoring according to a child falling/falling bed evaluation scale according to clinical data of the target child, wherein the child falling/falling bed evaluation scale obtains the target child falling/falling bed score from the clinical medicine perspective according to the age, sex, diagnosis, cognitive disorder, environmental factors, surgery/anesthesia/sedation response and medicine use information of the child. And then, carrying out weighted accumulation on the evaluation score and the danger score, and taking the obtained accumulated score as the danger score. And the risk score is evaluated in two dimensions, so that the evaluation accuracy can be further verified, and misjudgment is prevented.
As described above, in this embodiment, objects such as children, parents, medical care personnel, chairs, and bedside tables are detected in real time based on the yolov5 network model, and the position areas of the beds in the images are segmented; and judging the risk score of the current child frame by utilizing the position coordinates and the area information, weighting and combining the risk score with the clinical risk score obtained by the clinical data, and giving an alarm to the dangerous state of the child in real time if the risk score of the child is higher than a set threshold value.
In some embodiments, as shown in fig. 2, the step S500 specifically includes the following steps:
step S510: obtaining an area intersection comparison sequence of the target child and the associated object based on the position and the area of the target child and the position and the area of the associated object;
step S520: obtaining the variation trend of the area intersection ratio of the target child and the associated object based on the area intersection ratio sequence;
step S530: and obtaining the detection risk based on the variation trend.
Specifically, the area intersection ratio of the target child and each associated object is calculated according to the position and the area of the target child and the position and the area of the associated object in the image frame output by the target detection model. After the area intersection ratio is obtained, the embodiment judges the variation trend of the area intersection ratio through a detection risk calculation formula and outputs the detection risk. The detection risk calculation formula is specifically as follows:
Figure BDA0003708558100000101
Figure BDA0003708558100000102
Figure BDA0003708558100000103
wherein λ is 3 Is the current risk weight;
Figure BDA0003708558100000104
the weights of the cross-correlation activation functions for the corresponding areas; xi shape 1 ξ 2 Time-affecting bias constants of the low-order threshold and the high-order threshold, respectively; ζ represents a unit 1 ζ 2 Respectively a lower threshold and an upper threshold of the area intersection ratio.
According to the method, the detection risk is directly calculated through the detection risk calculation formula, the calculation is simple and quick, and the real-time detection of the target area is realized.
It should be noted that, although the embodiment is mainly used for early warning of a dangerous behavior of falling/falling of a child, the invention can also be used for early warning of other dangerous behaviors, such as real-time detection and alarm of dangerous behaviors of getting on and off a bed by a child in a ward, crossing a rail, touching a kettle, falling from a chair and the like.
Exemplary device
As shown in fig. 3, an embodiment of the present invention further provides a children dangerous behavior detection apparatus corresponding to the above children dangerous behavior detection method, where the children dangerous behavior detection apparatus includes:
a video data acquisition module 600, configured to monitor a target area in real time to obtain video data;
a detection data obtaining module 610, configured to input the video data into a target detection model, and obtain detection data of a detection object, where the detection object includes a target child and an associated object associated with a child dangerous behavior in a target area;
a clinical data obtaining module 620, configured to obtain pre-stored clinical data of a target child;
a clinical risk calculation module 630 for obtaining a clinical risk based on the clinical data;
the detection risk calculation module 640 is used for calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
a risk score calculation module 650 for obtaining a risk score according to a risk score calculation model based on the clinical risk and the detected risk;
and the alarm module 660 is used for outputting alarm information when the danger score exceeds a set threshold value.
Specifically, in this embodiment, the specific functions of each module of the child dangerous behavior detection apparatus may refer to the corresponding descriptions in the child dangerous behavior detection method, and are not described herein again.
Based on the above embodiment, the present invention further provides an intelligent terminal, and a schematic block diagram thereof may be as shown in fig. 4. The intelligent terminal comprises a processor, a memory, a network interface and a display screen which are connected through a system bus. Wherein, the processor of the intelligent terminal is used for providing calculation and control capability. The memory of the intelligent terminal comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a child-risk behavior detection program. The internal memory provides an environment for the operation of an operating system and a child-risk behavior detection program in the nonvolatile storage medium. The network interface of the intelligent terminal is used for being connected and communicated with an external terminal through a network. When being executed by a processor, the children dangerous behavior detection program realizes the steps of any one of the children dangerous behavior detection methods. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.
It will be understood by those skilled in the art that the block diagram of fig. 4 is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation to the intelligent terminal to which the solution of the present invention is applied, and a specific intelligent terminal may include more or less components than those shown in the figure, or combine some components, or have different arrangements of components.
In one embodiment, a smart terminal is provided, where the smart terminal includes a memory, a processor, and a child risk behavior detection program stored in the memory and executable on the processor, and the child risk behavior detection program performs the following operations when executed by the processor:
monitoring a target area in real time to obtain video data;
inputting the video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
acquiring pre-stored clinical data of a target child;
obtaining a clinical risk based on the clinical data;
calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
obtaining a risk score according to a risk score calculation model based on the clinical risk and the detected risk;
and when the danger score exceeds a set threshold value, outputting warning information.
Optionally, the associating objects include people and objects associated with dangerous behaviors of children in the target area, and the inputting the video data into the target detection model to obtain the detection data of the detection object includes:
inputting the video data into the target detection model to obtain a preset number of image frames;
sequentially acquiring the area and the position of the target child and the area and the position of the associated object in each image frame;
and saving the area and the position of the target child and the area and the position of the associated object based on the sequence of the image frames to obtain the detection data.
Optionally, the calculating, based on the detection data, an area intersection ratio of the target child and the associated object to obtain a detection risk includes:
obtaining an area intersection comparison sequence of the target child and the associated object according to the position and the area of the target child and the position and the area of the associated object;
obtaining the variation trend of the area intersection ratio of the target child and the associated object based on the area intersection ratio sequence;
and obtaining the detection risk based on all the variation trends.
Optionally, the expression of the risk score calculation model is as follows:
y=(ω 1 f 1 (a,g,d,m)+b 1 )×[(ω 2 max{x 1 ,x 2 }+b 2 )×f 2 (x 3 ,x 4 ,x 5 ,t)+b 3 ]f 3 (t),
wherein, w 1 ,w 2 A, g, d and m are weight coefficients, the age, the sex, the dangerous behavior history and the medication information of the target child are respectively, b 1 ,b 2 ,b 3 Is a constant offset, x 1 ,x 2 The area intersection ratio of the target child to the parents and the physician, x3, x4 and x5 are the area intersection ratio of the target child to the sickbed, the chair and the bedside table, t is duration, f is 1 Formula for clinical risk calculation, f 2 To detect the risk calculation formula, f 3 Is a duration weight formula.
Optionally, the clinical data comprises: the age, sex, dangerous behavior history and medication information of the target child, based on the clinical data, obtaining clinical risks comprising:
respectively acquiring an expected value and a standard deviation of each item of data in the clinical data;
and obtaining the clinical risk according to the clinical data, the expected value and the standard deviation based on the preset physiological information weight and the preset medical information weight.
Optionally, after obtaining the risk score according to the risk score calculation model, the method further includes:
obtaining an assessment score according to a child fall assessment scale based on the clinical data;
and performing weighted accumulation on the evaluation score and the danger score and updating the danger score.
Optionally, the monitoring the target area in real time to obtain video data includes:
and synchronously acquiring two video data of the target area, and preprocessing the two video data to acquire the video data.
The embodiment of the present invention further provides a computer-readable storage medium, where a child dangerous behavior detection program is stored in the computer-readable storage medium, and when the child dangerous behavior detection program is executed by a processor, the steps of any one of the child dangerous behavior detection methods provided in the embodiments of the present invention are implemented.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the apparatus may be divided into different functional units or modules to implement all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the above modules or units is only one type of logical function division, and the actual implementation may be implemented by another division manner, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
The integrated modules/units described above, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium and can implement the steps of the embodiments of the method when the computer program is executed by a processor. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file or some intermediate form. The computer readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunication signal, software distribution medium, etc. It should be noted that the contents contained in the computer-readable storage medium can be increased or decreased as required by legislation and patent practice in the jurisdiction.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art; the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein.

Claims (10)

1. A method for detecting dangerous behaviors of a child, the method comprising:
monitoring a target area in real time to obtain video data;
inputting the video data into a target detection model to obtain detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
acquiring pre-stored clinical data of a target child;
obtaining a clinical risk based on the clinical data;
calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
obtaining a risk score according to a risk score calculation model based on the clinical risk and the detected risk;
and when the danger score exceeds a set threshold value, outputting warning information.
2. The method for detecting dangerous behaviors of children as claimed in claim 1, wherein said associated objects include persons and objects associated with dangerous behaviors of children in a target area, and said inputting said video data into a target detection model to obtain detection data of a detection object comprises:
inputting the video data into the target detection model to obtain a preset number of image frames;
sequentially acquiring the area and the position of the target child and the area and the position of the associated object in each image frame;
and saving the area and the position of the target child and the area and the position of the associated object based on the sequence of the image frames to obtain the detection data.
3. The method for detecting dangerous behaviors of children according to claim 2, wherein the calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain the detection risk comprises:
according to the position and the area of the target child and the position and the area of the associated object, obtaining an area intersection and comparison sequence of the target child and the associated object;
obtaining the change trend of the area intersection ratio of the target child and the associated object based on the area intersection ratio sequence;
obtaining the detection risk based on all the variation trends.
4. The method for detecting dangerous behaviors of children according to claim 1, wherein the risk score calculation model has the expression:
y=(ω 1 f 1 (a,g,d,m)+b 1 )×[(ω 2 max{x 1 ,x 2 }+b 2 )×f 2 (x 3 ,x 4 ,x 5 ,t)+b 3 ]f 3 (t),
wherein, w 1 ,w 2 A, g, d, m are weight coefficientsAge, sex, history of dangerous behavior and medication information of the target child, b 1 ,b 2 ,b 3 Is a constant offset, x 1 ,x 2 Area intersection ratio, x, between the target child and the parents, respectively, the physician 3 ,x 4 ,x 5 The area intersection ratio of the target child to the sickbed, the chair and the bedside table, t is duration, f 1 For the clinical risk calculation formula, f 2 Formula for risk detection calculation, f 3 Is a duration weight formula.
5. The method for detecting dangerous behavior of children as claimed in claim 1, wherein said clinical data comprises: the age, sex, dangerous behavior history and medication information of the target child, based on the clinical data, obtaining clinical risks comprising:
respectively acquiring an expected value and a standard deviation of each item of data in the clinical data;
and acquiring the clinical risk according to the clinical data, the expected value and the standard deviation based on a preset physiological information weight and a preset medical information weight.
6. The method for detecting dangerous behaviors of children according to claim 1, wherein after obtaining the danger score according to the danger score calculation model, the method further comprises:
obtaining an assessment score from a child fall assessment scale based on the clinical data;
and performing weighted accumulation on the evaluation score and the danger score and updating the danger score.
7. The method for detecting dangerous behaviors of children as claimed in claim 1, wherein said monitoring the target area in real time and obtaining video data comprises:
and synchronously acquiring two video data of the target area, and preprocessing the two video data to acquire the video data.
8. Child dangerous behaviour detection apparatus, characterised in that the apparatus comprises:
the video data acquisition module is used for monitoring a target area in real time to acquire video data;
the detection data acquisition module is used for inputting the video data into a target detection model to acquire detection data of a detection object, wherein the detection object comprises a target child and an associated object associated with child dangerous behaviors in a target area;
the clinical data acquisition module is used for acquiring pre-stored clinical data of the target child;
a clinical risk calculation module for obtaining a clinical risk based on the clinical data;
the detection risk calculation module is used for calculating the area intersection ratio of the target child and the associated object based on the detection data to obtain a detection risk;
a risk score calculation module for obtaining a risk score according to a risk score calculation model based on the clinical risk and the detection risk;
and the alarm module is used for outputting alarm information when the danger score exceeds a set threshold value.
9. An intelligent terminal, characterized in that the intelligent terminal comprises a memory, a processor and a child dangerous behavior detection program stored on the memory and operable on the processor, and when executed by the processor, the steps of the child dangerous behavior detection method according to any one of claims 1 to 7 are implemented.
10. Computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a child risk behavior detection program, which when executed by a processor implements the steps of the child risk behavior detection method according to any one of claims 1-7.
CN202210712422.6A 2022-06-22 2022-06-22 Children dangerous behavior detection method and device, intelligent terminal and storage medium Active CN115223089B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210712422.6A CN115223089B (en) 2022-06-22 2022-06-22 Children dangerous behavior detection method and device, intelligent terminal and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210712422.6A CN115223089B (en) 2022-06-22 2022-06-22 Children dangerous behavior detection method and device, intelligent terminal and storage medium

Publications (2)

Publication Number Publication Date
CN115223089A true CN115223089A (en) 2022-10-21
CN115223089B CN115223089B (en) 2023-06-06

Family

ID=83608737

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210712422.6A Active CN115223089B (en) 2022-06-22 2022-06-22 Children dangerous behavior detection method and device, intelligent terminal and storage medium

Country Status (1)

Country Link
CN (1) CN115223089B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107358783A (en) * 2017-07-26 2017-11-17 深圳市盛路物联通讯技术有限公司 A kind of long distance monitoring method and device
US20180165938A1 (en) * 2016-12-09 2018-06-14 Fuji Xerox Co., Ltd. Monitoring apparatus and non-transitory computer readable medium
CN108540777A (en) * 2018-04-28 2018-09-14 上海与德科技有限公司 A kind of intelligent control method, device, equipment and storage medium
CN113251558A (en) * 2021-05-13 2021-08-13 重庆海尔空调器有限公司 Infant anti-falling monitoring control method and device, storage medium and air conditioner
CN113450534A (en) * 2020-03-27 2021-09-28 海信集团有限公司 Device and method for detecting approach of children to dangerous goods
CN113780255A (en) * 2021-11-12 2021-12-10 北京世纪好未来教育科技有限公司 Danger assessment method, device, equipment and storage medium
CN113971864A (en) * 2020-07-23 2022-01-25 易程融创信息科技有限公司 Children home safety monitoring method and device
CN114202775A (en) * 2021-12-16 2022-03-18 福建省海峡智汇科技有限公司 Transformer substation dangerous area pedestrian intrusion detection method and system based on infrared image

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180165938A1 (en) * 2016-12-09 2018-06-14 Fuji Xerox Co., Ltd. Monitoring apparatus and non-transitory computer readable medium
CN107358783A (en) * 2017-07-26 2017-11-17 深圳市盛路物联通讯技术有限公司 A kind of long distance monitoring method and device
CN108540777A (en) * 2018-04-28 2018-09-14 上海与德科技有限公司 A kind of intelligent control method, device, equipment and storage medium
CN113450534A (en) * 2020-03-27 2021-09-28 海信集团有限公司 Device and method for detecting approach of children to dangerous goods
CN113971864A (en) * 2020-07-23 2022-01-25 易程融创信息科技有限公司 Children home safety monitoring method and device
CN113251558A (en) * 2021-05-13 2021-08-13 重庆海尔空调器有限公司 Infant anti-falling monitoring control method and device, storage medium and air conditioner
CN113780255A (en) * 2021-11-12 2021-12-10 北京世纪好未来教育科技有限公司 Danger assessment method, device, equipment and storage medium
CN114202775A (en) * 2021-12-16 2022-03-18 福建省海峡智汇科技有限公司 Transformer substation dangerous area pedestrian intrusion detection method and system based on infrared image

Also Published As

Publication number Publication date
CN115223089B (en) 2023-06-06

Similar Documents

Publication Publication Date Title
US11544953B2 (en) Methods and systems for identifying the crossing of a virtual barrier
JP6823160B2 (en) Devices, systems, and methods for patient monitoring to predict and prevent bed falls
Rantz et al. Sensor technology to support aging in place
JP6975230B2 (en) Patient monitoring system and method
US9060681B2 (en) Trend monitoring system with multiple access levels
US11129549B2 (en) Device, system and method for patient monitoring to predict and prevent bed falls
US20190214146A1 (en) Device, system and method for patient monitoring to predict and prevent bed falls
WO2018086990A1 (en) Patient monitoring systems and methods
EP3504649B1 (en) Device, system and method for patient monitoring to predict and prevent bed falls
WO2021044520A1 (en) Software, state-of-health determination device, and state-of-health determination method
Lloyd-Jukes et al. Vision-based patient monitoring and management in mental health settings
CN107516019A (en) Noninvasive health forecast system and method
Inoue et al. Bed-exit prediction applying neural network combining bed position detection and patient posture estimation
CN113226154A (en) Display method, monitoring equipment and system for early warning score
US20230317269A1 (en) System and Method For Evaluating, Monitoring, Assessing and Predicting Ambient and Health Conditions
CN115223089B (en) Children dangerous behavior detection method and device, intelligent terminal and storage medium
Inoue et al. Bed exit action detection based on patient posture with long short-term memory
KR20230069363A (en) Abnormal behavior detecting system using artificial intelligence
TWI809864B (en) Sensing method and sensing system
TW202020890A (en) Display system for medical information and method for producing display content
TWI809612B (en) Fall Assessment Risk Warning System
CN114171146B (en) Intelligent medical monitoring method, system and readable storage medium
US20230301602A1 (en) Information processing device, information processing method, and storage medium
Ikawati et al. VidyaMedic: Smart Alert System For Continous Vital Sign Monitoring
Dawoudi 3. FUTURES OF HEALTH AND WELL-BEING

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant