CN112099629B - Method and system for providing working operation guide - Google Patents

Method and system for providing working operation guide Download PDF

Info

Publication number
CN112099629B
CN112099629B CN202010954387.XA CN202010954387A CN112099629B CN 112099629 B CN112099629 B CN 112099629B CN 202010954387 A CN202010954387 A CN 202010954387A CN 112099629 B CN112099629 B CN 112099629B
Authority
CN
China
Prior art keywords
user
mode
motion
actions
deviation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010954387.XA
Other languages
Chinese (zh)
Other versions
CN112099629A (en
Inventor
吴晓军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hebei Jilian Human Resources Service Group Co ltd
Original Assignee
Hebei Jilian Human Resources Service Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hebei Jilian Human Resources Service Group Co ltd filed Critical Hebei Jilian Human Resources Service Group Co ltd
Priority to CN202010954387.XA priority Critical patent/CN112099629B/en
Publication of CN112099629A publication Critical patent/CN112099629A/en
Application granted granted Critical
Publication of CN112099629B publication Critical patent/CN112099629B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Tourism & Hospitality (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Human Resources & Organizations (AREA)
  • Psychiatry (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Economics (AREA)
  • Social Psychology (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • User Interface Of Digital Computer (AREA)
  • Medical Treatment And Welfare Office Work (AREA)

Abstract

The present disclosure provides a method and system for providing a work operation guideline, wherein the method comprises: sensing and identifying an ongoing mode of operation of the user; analyzing the working mode to obtain standard actions and environment scenes in the working mode; inputting the actual actions of the user and the standard actions obtained by analysis into a machine learning model, and calculating the deviation between the actual actions of the user and the standard actions; if the deviation exceeds the threshold value, the environment scene is combined, a prompt is sent to a user, and correct operation guidance is displayed; continuously monitoring whether the action of the user is corrected, and if the user cannot normally operate, sending an early warning to a preset object; and storing the data of the deviation.

Description

Method and system for providing working operation guide
Technical Field
The present disclosure relates to the field of human resource management and sensor technology, and in particular, to a method, a system, an electronic device, and a computer-readable storage medium for providing a work operation guide.
Background
The household service is a comprehensive labor, for example, the work of the nurse may include one or more of cooking, mopping, wiping windows, looking at children, looking at the elderly, etc., and the work of decoration or maintenance may include water works, electricians, tile works, clay works, painters, carpenters, etc. The skills required for each mode of operation are different, as are the payouts per unit time. These works, although very different, are short-term or zero-term works, and problems are difficult to follow.
In the prior art, participants of short workers and zero workers are good and uneven, real-time supervision is difficult to achieve, evidence is difficult to obtain after errors occur, so that a recognition device capable of recognizing, professionally prompting, avoiding danger and fixing evidence is urgently needed, a user is managed, prompting is carried out when actions of the user are not standard, warning is carried out when the user is continuously not standard, evidence is fixed, and an employer is notified.
Disclosure of Invention
Accordingly, it is an object of embodiments of the present disclosure to provide a method and system for providing work instruction, which provides work instruction to a user by identifying a correct working mode, improves the working level thereof, and plays roles of supervising, fixing evidence, and reminding employers.
According to a first aspect of the present disclosure, there is provided a method of providing a work operation guideline, comprising:
sensing and identifying an ongoing mode of operation of the user;
analyzing the working mode to obtain standard actions and environment scenes in the working mode;
inputting the actual actions of the user and the standard actions obtained by analysis into a machine learning model, and calculating the deviation between the actual actions of the user and the standard actions;
if the deviation exceeds the threshold value, the environment scene is combined, a prompt is sent to a user, and correct operation guidance is displayed;
continuously monitoring whether the action of the user is corrected, and if the user cannot normally operate, giving an early warning to preset personnel;
and storing the data of the deviation.
In one possible embodiment, the operation mode includes: a cleaning working mode, a decoration working mode, a worker protection working mode and a nurse protection working mode.
In a possible embodiment, in the cleaning operation mode, the method for analyzing the operation mode further includes: the analysis of the action of the leg area is enhanced, and the analysis of the cleaning effect of the cleaning object is enhanced.
In one possible embodiment, in the finishing operation mode, the method for resolving the operation mode further includes: the analysis of the actions of the hand area is enhanced, and the user is actively reminded of the operation specification to be noticed.
In one possible embodiment, in the working mode of the worker, the method for resolving the working mode further includes: the analysis of the medicine type and the medicine amount is enhanced.
In one possible embodiment, in the guarse operation mode, the method for resolving the operation mode further includes: enhancing the interpretation of the cared body's appearance and using the BERT-based machine learning model to interpret the user's dialog with the cared person.
In one possible embodiment, the method further comprises: when the dialog is abuse-related, the preset object is notified and data is stored.
According to a second aspect of the present disclosure, there is provided a system for providing a work operation guide, comprising:
the sensing unit is used for sensing and identifying the working mode in progress of a user;
the analysis unit is used for analyzing the working mode and acquiring standard actions and environment scenes in the working mode;
the deviation calculation unit is used for inputting the actual actions of the user and the standard actions obtained by analysis into the machine learning model and calculating the deviation between the actual actions of the user and the standard actions;
the prompting unit is used for combining the environment scenes, sending a prompt to a user if the deviation exceeds a threshold value, and displaying correct operation guidance;
the early warning unit is used for continuously monitoring whether the action of the user is corrected or not, and if the user cannot normally operate, early warning is carried out on preset people;
and the storage unit is used for storing the data of the deviation.
According to a third aspect of the present disclosure, there is provided an electronic device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to the first aspect when executing the program.
According to a fourth aspect of the present disclosure there is provided a computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to the first aspect.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art. The above and other objects, features and advantages of the present application will become more apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the several views of the drawings. The drawings are not intended to be drawn to scale, with emphasis instead being placed upon illustrating the principles of the present application.
Fig. 1 shows a schematic diagram of an exemplary home mode of operation identification device according to an embodiment of the present disclosure.
FIG. 2 illustrates a schematic diagram of a method of exemplary work operation guidelines according to embodiments of the disclosure.
Fig. 3 illustrates a schematic diagram of acceleration sensor values for a typical hand motion when wiping a window in accordance with an embodiment of the present disclosure.
FIG. 4 illustrates a schematic diagram of an exemplary visual alert according to an embodiment of the present disclosure.
Fig. 5 shows a schematic diagram of a system of typical work operations guidelines according to embodiments of the disclosure.
Fig. 6 shows a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The words "a", "an", and "the" as used herein are also intended to include the meaning of "a plurality", etc., unless the context clearly indicates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
The household service is a comprehensive labor, for example, the work of the nurse may include one or more of cooking, mopping, wiping windows, looking at children, looking at the elderly, etc., and the work of decoration or maintenance may include water works, electricians, tile works, clay works, painters, carpenters, etc. The skills required for each mode of operation are different, as are the payouts per unit time. These works, although very different, are short-term or zero-term works, and problems are difficult to follow.
In the prior art, participants of short workers and zero workers are good and uneven, real-time supervision is difficult to achieve, evidence is difficult to obtain after errors occur, so that a recognition device capable of recognizing, professionally prompting, avoiding danger and fixing evidence is urgently needed, a user is managed, prompting is carried out when actions of the user are not standard, warning is carried out when the user is continuously not standard, evidence is fixed, and an employer is notified.
Accordingly, it is an object of embodiments of the present disclosure to provide a method and system for providing work instruction, which provides work instruction to a user by identifying a correct working mode, improves the working level thereof, and plays roles of supervising, fixing evidence, and reminding employers.
The present disclosure is described in detail below with reference to the accompanying drawings.
Fig. 1 shows a schematic diagram of an exemplary home mode of operation identification device according to an embodiment of the present disclosure.
FIG. 2 illustrates a schematic diagram of a method of exemplary work operation guidelines according to embodiments of the disclosure.
Step 201, sensing and identifying the operating mode being performed by the user.
The user wears the recognition device, for example, the recognition device 100 of the household operation mode shown in fig. 1, senses the user's motion and recognizes the household operation mode. Employers or users may also set the current mode of home operation themselves. The camera 110 is used for collecting first view video data of a user. The camera 110 may be worn on the head, such as a hat, helmet, glasses, etc., wearable device that may have a wireless communication interface, such as WiFi, bluetooth, etc., to upload the acquired video data to a server (not shown) for processing. A computer program is deployed on the server for extracting spatial features 112 and temporal features 113 from the video data. A plurality of motion sensors including a wristband motion sensor 120-1 and a head motion sensor 120-2 are worn on different body parts of the user to detect corresponding motion sensing data. Specifically, the bracelet motion sensor 120-1 may be worn on the wrist of the user for detecting acceleration, angular acceleration and geomagnetic data of the wrist when the user acts; the head motion sensor 120-2 may be worn on the head of a user, such as a wearable device, e.g., a hat, helmet, glasses, etc., for detecting acceleration, angular acceleration, and geomagnetic data of the head while the user is acting. The acceleration includes translational acceleration in the X, Y, Z axis direction of the three-dimensional space coordinate system, and the angular acceleration includes acceleration around three coordinate axes of the three-dimensional space coordinate system, including angular acceleration of pitch, roll, and rotation. The geomagnetic data includes detected data about the geomagnetic direction, i.e., the azimuthal orientation of the motion sensor.
Motion sensing data of the motion sensors 120-1 and 120-2 may be input to a support vector machine 121, through which support vector machine 121 motion characteristics 122 are generated. The support vector machine 121 may be pre-trained to be suitable for generating the motion features 122-1 and 122-2 for the preset motion pattern. The motion features 122 may be motion pattern vectors, wherein each component represents a probability and intensity that the motion sensing data belongs to a respective motion class. The motion category comprises large motion translation, large motion rotation, large motion vibration, fine motion translation, fine motion rotation and fine motion vibration, and the intensity comprises displacement distance, amplitude and frequency.
And calculating the motion intensity according to the acceleration and the angular acceleration in the motion sensing data. For example, the displacement distance, amplitude and frequency of the motion sensor can be calculated by inertial navigation. The above-mentioned class components are combined with displacement distances, amplitudes, frequencies to form the motion profile 122.
The following data were obtained: spatial 112 and temporal 113, motion pattern 122 and the spatial relationship 130 features based on the image data.
The spatial features of the image data comprise 8 times downsampling features, 16 times downsampling features and 32 times downsampling features of video frames in the image data are extracted by using a convolutional neural network and combined to form multi-scale features. The temporal characteristics of the image data include randomly selecting a portion of the video frames from a plurality of video frames that precede the current frame by a period of time, combining spatial characteristics of the selected portion of the video frames to form the temporal characteristics. The motion modes comprise categories and intensities, the categories comprise large motion translation, large motion rotation, large motion vibration, fine motion translation, fine motion rotation and fine motion vibration, and the intensities comprise distance, amplitude and frequency. The spatial relationship features include: based on the sensing data of the head motion sensor and the sensing data of the bracelet motion sensor, calculating a spatial relation vector of the bracelet motion sensor relative to the head motion sensor in an inertial navigation mode, and obtaining a time sequence of the spatial relation vector as the spatial relation feature. The spatial relationship features are embodied as the spatial relationship of the limbs relative to the head with which the position of the hand in the vicinity of the body can be perceived, which helps determine the household pattern.
The spatial relationship feature 130 is generated from motion sensing data of the wristband motion sensor 120-1 and the head motion sensor 120-2. The spatial feature 112 and temporal feature 113 of the video data obtained above, the motion features 122-1 and 122-2 of the motion sensor, and the spatial relationship feature 130 may be input together into the neural network 140 by stitching. The neural network 140 may output a vector regarding the behavior pattern and the operation intensity, and take the behavior pattern with the highest probability as the operation pattern 150 of the user.
Step 202, analyzing the working mode, and obtaining standard actions and environment scenes in the working mode;
for four main household working modes, namely a cleaning working mode, a decoration working mode, a worker protection working mode and a nurse protection working mode, standard actions, also called template actions, are set for each mode. The motion is a standard motion obtained by wearing a motion catcher on a human body, inviting a subject to repeatedly perform the motion according to a standard flow and a gesture, and collecting the motion. In the identified or preset household mode of operation, the standard actions associated therewith are stored.
Fig. 3 illustrates a schematic diagram of acceleration sensor values for a typical hand motion when wiping a window in accordance with an embodiment of the present disclosure.
The acceleration value of the standard motion is expressed in the form of a three-dimensional coordinate system, and the acceleration of one axis is almost 0 because the hand moves almost on one plane when the window is wiped. Similarly, angular acceleration includes acceleration about three coordinate axes of a three-dimensional space coordinate system, including angular acceleration of pitch, roll, and rotation.
In addition to the acceleration values and angular acceleration values, the standard motion may be represented by the frame difference vector, displacement values, and gyroscope values of the video, and the above spatial features and temporal features based on image data, motion pattern features, and other dimensions such as the spatial relationship features may also represent the standard motion, which is not limited in this disclosure.
The environmental scenario in each of the housekeeping modes of operation is also pre-established and stored in order to reduce errors in calculating the motion deviation. For example, for caregivers, sanitation and decoration modes of operation, the environmental scene is divided into indoor and outdoor. Indoor refers to a daily living environment, and outdoor refers to a daily outdoor environment. For the worker-protecting operation mode, the daytime and night environments are increased outside the indoor and outdoor modes. The recognition device 100 may automatically recognize the current environment scene, and the employer or the user may set the current environment scene by himself.
In step 203, the actual motion of the user and the standard motion obtained by analysis are input into the machine learning model, and the deviation between the actual motion and the standard motion of the user is calculated.
The machine learning model used in the present disclosure is a deep learning neural network model based on, and includes a convolution layer, a pooling layer, a nonlinear transformation layer and a weight nonlinear layer which are sequentially connected.
When calculating the deviation of each action category, taking data in a specific sliding window size range of the actual action and the standard action as input; each data in the input window may be mapped to an N-dimensional vector; the convolution layer then generates a global feature corresponding to the hidden node; these features are fed to the pooling layer and then through a nonlinear variation layer and a weighting nonlinear layer. Finally, the features including local features and global features are sent into a standard radiation network together, and the hidden function value extracted by the features of the last layer is multiplied by a certain weight w i Reverse outputTo the linear neural unit, i.e., the pooling layer, to enable reuse of valuable information, increasing the weight in the overall information. The training is performed using a back propagation algorithm to a level that is suitably stable throughout the network.
The effect of the environmental scenarios is that for each environmental scenario, a weight w is set that reinforces the corresponding part of the machine learning model i Making it a concern for more notable matters. For example, for the cleaning operation mode in the indoor scene, the cleaning effect on the cleaning object in the image is more concerned. The recognition of the hand motion can help judge that the user is in different motions such as window wiping, floor sweeping, table wiping and the like. If the current action of the user is determined to be sweeping, the weight of the characteristics of the object on the floor in the image is increased so as to be beneficial to determining the cleaning effect.
Further, step 203 may preset values, such as setting floor area, window area, etc. to be cleaned for cleaning. Thus, in the cleaning operation, the cleaned area and the uncleaned area can be identified, and the time spent, by step 203.
Step 204, in combination with the environmental scenario, if the deviation exceeds the threshold, a prompt is sent to the user, and the correct operation guidance is displayed.
Since the accuracy of the recognition of the different actions is different, there is a corresponding threshold in the different scenarios in each operating mode. For example, in an indoor environment, the threshold is smaller because of less interference and higher recognition accuracy of actions. In an outdoor environment, the number of interfering objects is large, the light is strong or weak, the error of identification through video is large, so that the error of deviation is also large, and the corresponding threshold value is also large.
For the case that the threshold value is exceeded, the user is prompted and the correct operation guidance is presented, which is hoped to complete the work in the correct working method. For example, for the action of mopping the floor of an indoor scene in the cleaning working mode, the identified action is to move the mop once every second, which is obviously lower than the level of a normal person, and a user may be lazy. Assuming that the deviation of the threshold value is 10% at this time and the actual calculated deviation is 50%, the threshold value is exceeded, a prompt can be given to the user at this time.
Presenting proper operation guidelines may include: voice prompts and visual prompts. Wherein the visual alert is displayed through AR glasses 160 of device 100.
FIG. 4 illustrates a schematic diagram of an exemplary visual alert according to an embodiment of the present disclosure.
The visual presentation of fig. 4 is exemplified by a window wiping action, in which the cleaned area is shown as well as the uncleaned area. In the cleaned area, the degree of cleaning, the cleaning efficiency, the cleaning material are displayed. The cleaning degree is obtained in step 203, and the cleaning efficiency, which is the ratio of the time actually spent to the average time under the same area, can be calculated by the preset area to be cleaned. Glass washing agent: the use of the cleaning material normally reflects the use condition of the cleaning material, and prompts the condition of waste or use shortage. The cleaned area is identified by step 203. The cleaning path 410 is a cleaning path of the cleaned area, recorded by the apparatus 100.
In the uncleaned area, the remaining area and the expected time are displayed. The remaining area may be identified in step 203, or a difference between a predetermined area to be cleaned and the cleaned area may be calculated. The predicted time is estimated by the cleaning efficiency. The recommended cleaning route 411 is a recommended cleaning route preset in an unclean area, as needed or according to certain criteria. If no cleaning route is provided, the recommended cleaning route is not displayed, or the cleaning route of the apparatus 100 is displayed.
The visual presentation may also include the task type and in what order and time requirements the task is completed. Tasks that the user needs to clean properly (e.g., floors, furniture, windows, etc.) are marked red. The AR glasses of the user can display the information of the required cleaning tool, the instruction of the working steps and other instructions which are helpful for working, so that some cleaning tasks with special requirements can be smoothly carried out, errors in cleaning are reduced, the cleaner can receive the instruction while working, the time for training the cleaner is reduced, and the cleaning effect can be ensured.
For example, some users use the same wipe for any area in order to save trouble, such as: such a serious error can be immediately found at this time by cleaning the living room table using a cleaning cloth for cleaning a bathroom or a kitchen. For convenient operation, the colors of different areas can be set by oneself. For example, the uncleaned area is set to red, the cleaned area is set to green, and the uncleaned area is not friendly to the red-green color blindness, and can be adjusted to other colors.
Step 205, continuously monitoring whether the action of the user is corrected, and if the user cannot operate normally, giving an early warning to the preset person.
The action of the user is continuously detected, the deviation is calculated, if the user can not normally operate for a certain time, an early warning prompt can be sent to an employer or a preset object, and common communication means such as short messages, session, APP notification and the like can be used.
Step 206, storing the data of the deviation.
In one possible embodiment, in the cleaning operation mode, the method for resolving the operation mode further includes: the analysis of the action of the leg area is enhanced, and the analysis of the cleaning effect of the cleaning object is enhanced.
In the cleaning work, the body has more large displacement, such as sweeping, mopping, wiping the table top, and the like, and under the indoor environment, the displacement obtained by integrating the acceleration sensor can be influenced due to poor GPS signals, so that the error is larger. Therefore, the weight of the leg motion in the image data is enhanced, and errors generated when calculating the deviation are reduced as much as possible.
In one possible embodiment, in the finishing operation mode, the method for resolving the operation mode further includes: the analysis of the actions of the hand area is enhanced, and the user is actively reminded of the operation specification to be noticed.
The finishing work is a delicate hand work in which special attention is paid to the movements of the hands and the kinds of articles held by the hands. The purpose is to distinguish the specific working types of users, such as electricians, hydraulic engineering and the like.
The water engineering and electrician in the decoration have safe and standard operation. When the user is identified to be in the working mode, the operation standard which the user should pay attention to is actively reminded.
In one possible embodiment, in the worker protection operation mode, the method for resolving the operation mode further includes: the analysis of the medicine type and the medicine amount is enhanced.
Caregivers may be involved in feeding medications to caregivers and are very important and not subject to error. Therefore, the analysis of medicines and medicines is enhanced, and the weight of small articles such as bottles and tablets in the image data is increased.
In one possible embodiment, in the caregiver operating mode, the method for resolving the operating mode further includes: enhancing the interpretation of the cared body's appearance and using the BERT-based machine learning model to interpret the user's dialog with the cared person.
When the nurse works, the cared person is greatly different from the infant and the old, the infant may not need to have too much conversation, and the old may have more conversations. Abuse is also a abuse if the dialog is abuse-related. In this case, it is necessary to analyze the physical characteristics of the cared person to determine the age, and to analyze the dialogue between the user and the cared person to determine whether the abuse is involved.
In one possible embodiment, the preset object is notified and data is stored when the dialog is abuse-related. If the caretaker is abusive or abusive, the employer or the preset subject is notified while the data collected by the camera device and the voice device is stored.
One way to parse conversations and determine whether abuse is involved is to use a BERT-based machine learning model.
The best-effort classification models accepted in the field of Natural Language Processing (NLP) at present are BERT, ELECTRA and the like, so that the scheme adopted by the present disclosure is a machine learning model obtained by directly adding words related to abuse to train on the basis of the BERT pre-training model.
Fig. 5 shows a schematic diagram of a system of typical work operations guidelines according to embodiments of the disclosure. The system 500 includes:
a sensing unit 501 for sensing and identifying an ongoing operation mode of the user;
the parsing unit 502 is configured to parse the working mode, and obtain a standard action and an environmental scene in the working mode;
a deviation calculating unit 503, configured to input the actual motion of the user and the standard motion obtained by analysis into the machine learning model, and calculate a deviation between the actual motion of the user and the standard motion;
the prompting unit 504 is configured to combine the environmental scenario, and if the deviation exceeds a threshold value, send a prompt to a user and display a correct operation guide;
the early warning unit 505 is used for continuously monitoring whether the action of the user is corrected, and if the user cannot normally operate, early warning is carried out on preset personnel;
and a storage unit 506, configured to store the data of the deviation.
Fig. 6 shows a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure. As shown in fig. 6, the electronic device 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic apparatus 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, may be downloaded and installed from a network via the communication portion 609 and/or installed from the removable medium 611. When executed by a Central Processing Unit (CPU) 601, performs the various method steps described in this disclosure.
Although example embodiments have been described, it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the disclosed concept. Accordingly, it should be understood that the above-described example embodiments are not limiting, but rather illustrative.

Claims (10)

1. A method of providing a work operation guideline, comprising:
sensing and identifying an ongoing mode of operation of the user;
the sensing and identifying the operating mode in progress by the user comprises the following steps:
the method comprises the steps that a camera collects first visual angle video data of a user, and a convolutional neural network is used for extracting multi-scale space features and time features from the video data;
the motion sensor is worn on different body parts of a user to detect corresponding motion sensing data, the motion sensing data of the motion sensor are input into the support vector machine, and motion characteristics are generated through the support vector machine;
obtaining a time sequence of the spatial relation vector according to motion sensing data of the motion sensor, and obtaining spatial relation characteristics;
the method comprises the steps of splicing and combining the spatial characteristics and the temporal characteristics extracted from video data, the motion characteristics and the spatial relationship characteristics of a motion sensor, inputting the spatial characteristics and the spatial relationship characteristics into a neural network together, outputting a behavior mode and a vector of working intensity by the neural network, and taking the behavior mode with the highest probability as a working mode of a user;
analyzing the working mode to obtain standard actions and environment scenes in the working mode;
inputting the actual actions of the user and the standard actions obtained by analysis into a machine learning model, and calculating the deviation between the actual actions of the user and the standard actions;
the machine learning model comprises a convolution layer, a pooling layer, a nonlinear transformation layer and a weight nonlinear layer which are sequentially connected, wherein the characteristics of the weight nonlinear layer are input into a radiation network, the hidden function value of the last layer of the radiation network after characteristic extraction is multiplied by the weight, and the hidden function value is reversely output to the pooling layer;
if the deviation exceeds the threshold value, the environment scene is combined, a prompt is sent to a user, and correct operation guidance is displayed;
continuously monitoring whether the action of the user is corrected, and if the user cannot normally operate, sending an early warning to a preset object;
and storing the data of the deviation.
2. The method of claim 1, the operating mode comprising: a cleaning working mode, a decoration working mode, a worker protection working mode and a nurse protection working mode.
3. The method of claim 2, wherein in the cleaning mode of operation, the method of resolving the mode of operation further comprises: the analysis of the action of the leg area is enhanced, and the analysis of the cleaning effect of the cleaning object is enhanced.
4. The method of claim 2, wherein in the finishing operation mode, the method of resolving the operation mode further comprises: the analysis of the actions of the hand area is enhanced, and the user is actively reminded of the operation specification to be noticed.
5. The method of claim 2, wherein in the worker-protecting mode of operation, the method of resolving the mode of operation further comprises: the analysis of the medicine type and the medicine amount is enhanced.
6. The method of claim 2, wherein in the caregivers' mode of operation, the method of resolving the mode of operation further comprises: enhancing the interpretation of the cared body's appearance and using the BERT-based machine learning model to interpret the user's dialog with the cared person.
7. The method of claim 6, further comprising: when the dialog is abuse-related, the preset object is notified and data is stored.
8. A system for providing directions for work operations, comprising:
the sensing unit is used for sensing and identifying the working mode in progress of a user;
the sensing and identifying the operating mode in progress by the user comprises the following steps:
the method comprises the steps that a camera collects first visual angle video data of a user, and a convolutional neural network is used for extracting multi-scale space features and time features from the video data;
the motion sensor is worn on different body parts of a user to detect corresponding motion sensing data, the motion sensing data of the motion sensor are input into the support vector machine, and motion characteristics are generated through the support vector machine;
obtaining a time sequence of the spatial relation vector according to motion sensing data of the motion sensor, and obtaining spatial relation characteristics;
the method comprises the steps of splicing and combining the spatial characteristics and the temporal characteristics extracted from video data, the motion characteristics and the spatial relationship characteristics of a motion sensor, inputting the spatial characteristics and the spatial relationship characteristics into a neural network together, outputting a behavior mode and a vector of working intensity by the neural network, and taking the behavior mode with the highest probability as a working mode of a user;
the analysis unit is used for analyzing the working mode and acquiring standard actions and environment scenes in the working mode;
the deviation calculation unit is used for inputting the actual actions of the user and the standard actions obtained by analysis into the machine learning model and calculating the deviation between the actual actions of the user and the standard actions; the machine learning model comprises a convolution layer, a pooling layer, a nonlinear transformation layer and a weight nonlinear layer which are sequentially connected, wherein the characteristics of the weight nonlinear layer are input into a radiation network, the hidden function value of the last layer of the radiation network after characteristic extraction is multiplied by the weight, and the hidden function value is reversely output to the pooling layer;
the prompting unit is used for combining the environment scenes, sending a prompt to a user if the deviation exceeds a threshold value, and displaying correct operation guidance;
the early warning unit is used for continuously monitoring whether the action of the user is corrected or not, and if the user cannot normally operate, the early warning unit sends early warning to a preset object;
and the storage unit is used for storing the data of the deviation.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.
10. A computer readable storage medium having stored thereon executable instructions which when executed by a processor cause the processor to perform the method of any of claims 1 to 7.
CN202010954387.XA 2020-09-11 2020-09-11 Method and system for providing working operation guide Active CN112099629B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010954387.XA CN112099629B (en) 2020-09-11 2020-09-11 Method and system for providing working operation guide

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010954387.XA CN112099629B (en) 2020-09-11 2020-09-11 Method and system for providing working operation guide

Publications (2)

Publication Number Publication Date
CN112099629A CN112099629A (en) 2020-12-18
CN112099629B true CN112099629B (en) 2024-04-16

Family

ID=73751531

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010954387.XA Active CN112099629B (en) 2020-09-11 2020-09-11 Method and system for providing working operation guide

Country Status (1)

Country Link
CN (1) CN112099629B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI766491B (en) * 2020-12-22 2022-06-01 國立清華大學 Human negligence warning method based on augmented reality
CN113204306A (en) * 2021-05-12 2021-08-03 同济大学 Object interaction information prompting method and system based on augmented reality environment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010146223A (en) * 2008-12-18 2010-07-01 Hitachi Ltd Behavior extraction system, behavior extraction method, and server
CN102483618A (en) * 2010-07-28 2012-05-30 费希尔-罗斯蒙德系统公司 Intrinsically-safe handheld field maintenance tool with image and/or sound capture
CN107292247A (en) * 2017-06-05 2017-10-24 浙江理工大学 A kind of Human bodys' response method and device based on residual error network
CN110301012A (en) * 2017-02-24 2019-10-01 通用电气公司 The auxiliary information about health care program and system performance is provided using augmented reality
CN111274881A (en) * 2020-01-10 2020-06-12 中国平安财产保险股份有限公司 Driving safety monitoring method and device, computer equipment and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170188895A1 (en) * 2014-03-12 2017-07-06 Smart Monitor Corp System and method of body motion analytics recognition and alerting
US9805255B2 (en) * 2016-01-29 2017-10-31 Conduent Business Services, Llc Temporal fusion of multimodal data from multiple data acquisition systems to automatically recognize and classify an action
US10663298B2 (en) * 2017-06-25 2020-05-26 Invensense, Inc. Method and apparatus for characterizing platform motion

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010146223A (en) * 2008-12-18 2010-07-01 Hitachi Ltd Behavior extraction system, behavior extraction method, and server
CN102483618A (en) * 2010-07-28 2012-05-30 费希尔-罗斯蒙德系统公司 Intrinsically-safe handheld field maintenance tool with image and/or sound capture
CN110301012A (en) * 2017-02-24 2019-10-01 通用电气公司 The auxiliary information about health care program and system performance is provided using augmented reality
CN107292247A (en) * 2017-06-05 2017-10-24 浙江理工大学 A kind of Human bodys' response method and device based on residual error network
CN111274881A (en) * 2020-01-10 2020-06-12 中国平安财产保险股份有限公司 Driving safety monitoring method and device, computer equipment and storage medium

Also Published As

Publication number Publication date
CN112099629A (en) 2020-12-18

Similar Documents

Publication Publication Date Title
Deep et al. A survey on anomalous behavior detection for elderly care using dense-sensing networks
CN110605724B (en) Intelligence endowment robot that accompanies
US9262068B2 (en) Interactive surface
CN112099629B (en) Method and system for providing working operation guide
Zouba et al. Monitoring activities of daily living (ADLs) of elderly based on 3D key human postures
WO2020248458A1 (en) Information processing method and apparatus, and storage medium
Mazurek et al. Use of kinematic and mel-cepstrum-related features for fall detection based on data from infrared depth sensors
US11747443B2 (en) Non-contact identification of multi-person presence for elderly care
Dimitrievski et al. Towards application of non-invasive environmental sensors for risks and activity detection
JP7266984B2 (en) Server equipment
CN111166340A (en) Human body posture real-time identification method based on self-adaptive acceleration signal segmentation
Hwang et al. A rapport and gait monitoring system using a single head-worn IMU during walk and talk
JP2020187389A (en) Mobile body locus analysis apparatus, mobile body locus analysis program, and mobile body locus analysis method
JP2021033359A (en) Emotion estimation device, emotion estimation method, program, information presentation device, information presentation method and emotion estimation system
KR20200117772A (en) Electronic apparatus and method for controlling the electronic apparatus
Fernando et al. Computer vision based privacy protected fall detection and behavior monitoring system for the care of the elderly
KR102544147B1 (en) Image Analysis based One Person Fall Detection System and Method
JP2018079069A (en) Behavior recognition apparatus and method, and care information output device and method
US20210304420A1 (en) Apparatus and method for protecting against environmental hazards
US20230316891A1 (en) System and technique for controlling cleaning behavior and managing prohibited actions interfering with cleanliness in a cleanroom environment
Ballester et al. Vision-Based Toilet Assistant for People with Dementia in Real-Life Situations
Grewe et al. A cyber-physical system for senior collapse detection
US20240016455A1 (en) Systems and methods to protect against infectious disease
Frank et al. Monitoring technologies for buildings equipped with ambient assisted living: Current status and where next
JP7205540B2 (en) Computer Executed Programs, Information Processing Devices, and Computer Executed Methods

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant