CN115730236B - Medicine identification acquisition method, equipment and storage medium based on man-machine interaction - Google Patents
Medicine identification acquisition method, equipment and storage medium based on man-machine interaction Download PDFInfo
- Publication number
- CN115730236B CN115730236B CN202211486821.1A CN202211486821A CN115730236B CN 115730236 B CN115730236 B CN 115730236B CN 202211486821 A CN202211486821 A CN 202211486821A CN 115730236 B CN115730236 B CN 115730236B
- Authority
- CN
- China
- Prior art keywords
- medicine
- information
- image
- identification
- robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Manipulator (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a medicine identification and acquisition method based on man-machine interaction, which comprises the following steps: determining the type of drug required; the robot moves to a fixed medicine storage position through SLAM technology, and the field information is acquired and stored by adopting a visual servo method; processing the image obtained by the depth camera by adopting a histogram equalization method, and enhancing the overall contrast of the image; identifying a target object by adopting a multi-feature fusion object identification method, and determining the position information of a target medicine; and (3) utilizing the multi-degree-of-freedom soft claw mechanical arm to grab the first medicine with the matching degree for the first time, taking the first medicine into the field of vision of the old, inquiring whether the first medicine is the required medicine, and moving the first medicine to the front of the old if the requirement is met. The method provides a shielding object identification and global optimal path planning scheme, greatly reduces the influence of a complex environment on the robot work efficiency, and further improves the identification and grabbing accuracy of the appointed medicine.
Description
Technical Field
The application belongs to the technical field of robot control, and relates to a medicine identification and acquisition method, equipment and a storage medium based on man-machine interaction.
Background
The nursing robot is a semi-autonomous or fully autonomous robot, which can provide necessary life assistance for disabled people, and thus requires the robot to have good man-machine interaction capability and high processing efficiency in the face of abnormal conditions. The current nursing robot still has the defects of lower target recognition precision and larger influence by complex environment,
the working space of a nursing robot in a home environment, a hospital, a nursing home and the like usually belongs to an unstructured or semi-structured environment, and target medicines are often placed out of order, so that the operation of the targets by a mechanical arm is challenging. The mechanical arm is usually carried on a mobile platform, and the degree of freedom and the disturbance of self-positioning introduced by the mobile platform further improve the difficulty of identifying and operating the target. Therefore, the method has important significance in realizing accurate positioning and grabbing of the target through multi-sensor fusion in a complex indoor environment. The application provides a medicine identification and acquisition method of a nursing robot based on man-machine interaction.
Disclosure of Invention
Aiming at the defects of the prior art, the application provides a drug identification and acquisition method, equipment and storage medium based on human-computer interaction, and provides a shielding object identification and global optimal path planning scheme, thereby greatly reducing the influence of complex environment on the efficiency of robot work and further improving the identification and grabbing accuracy of specified drugs.
The application provides a medicine identification and acquisition method based on man-machine interaction, which comprises an intelligent robot and a mechanical arm carrying a mobile platform, wherein the medicine identification and acquisition method of an aging-assisting nursing robot comprises the following steps:
step (1), obtaining characteristic parameters of voice by adopting an MFCC parameter extraction method, performing DTW matching with a voice template manufactured in advance, mapping voice information to a drug image template, and determining the type of a required drug;
step (2), the robot moves to a fixed medicine storage position through SLAM technology, the field information is acquired and stored by adopting a visual servo method, and meanwhile, information reminding is sent to families through a network module, wherein the information comprises an identification result and the pose of the robot;
step (3), processing the image obtained by the depth camera by adopting a histogram equalization method, and enhancing the overall contrast of the image;
step (4), identifying a target object by adopting a multi-feature fusion object identification method, and determining the position information of a target medicine;
step (5), the multi-degree-of-freedom soft claw mechanical arm is utilized to grasp the first medicine with the matching degree sequence for the first time, the medicine is brought into the field of vision of the old, whether the medicine is required is inquired, and if the medicine meets the requirement, the medicine moves to the front of the old;
step (6), if the selected medicine does not meet the requirement, repeating the steps (2) - (5), and sequentially selecting the medicine with the highest residual matching degree until the medicine meets the requirement or a command for stopping searching is received;
and (7) after the old people send out the using instruction, grabbing the medicine in the old people and returning to the original placement position, and finally moving to the initial position to wait for further instructions.
The beneficial effects of the application are as follows:
the application fully considers man-machine interaction with handicapped people, increases convenience and intelligence in the use process of the nursing robot, and provides more comprehensive and comprehensive service;
the application provides a shielding object identification and global optimal path planning scheme, which greatly reduces the influence of complex environments on the robot work efficiency and further improves the identification and grabbing accuracy of the appointed medicines;
the application can upload the pose and the environmental information of the robot in real time, help families to know the situation in time and give further instructions, and reduce the probability of occurrence of accidents.
Drawings
The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate the application and together with the embodiments of the application, serve to explain the application. In the drawings:
fig. 1 is a flowchart of a task implementation method according to an embodiment of the present application.
Description of the embodiments
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In addition, numerous specific details are set forth in the following description in order to provide a better understanding of the present application. It will be understood by those skilled in the art that the present application may be practiced without some of these specific details. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present application.
Examples
As shown in fig. 1, the medicine identification and acquisition method for an old-fashioned care robot is based on an old-fashioned care robot provided with a movable grabbing object, wherein a mobile platform is mounted at the bottom of the old-fashioned care robot, a mechanical arm and a depth camera are mounted at the top of the old-fashioned care robot, and at least a laser radar is arranged on the old-fashioned care robot;
the medicine identification and acquisition method for the nursing robot for the elderly comprises the following steps:
step (1), obtaining characteristic parameters of voice by adopting an MFCC parameter extraction method, performing DTW matching with a voice template manufactured in advance, mapping voice information to a drug image template, and determining the type of a required drug, wherein the specific steps are as follows:
1-1 pre-emphasis: the voice signal is passed through a high-pass filter to raise the high-frequency part to obtain a signal function；
(1)
Wherein the method comprises the steps ofRepresenting the original speech signal,/-, and>representing pre-emphasis parameters;
1-2, F sampling points are collected into frames, M is added between two adjacent frames m Overlapping regions of the sampling points, in which；
1-3 adding Hamming window to the signal after framing, using window function：
(2)
Wherein the method comprises the steps ofA Hamming parameter, N represents the size of the frame; applying a window function to each frame, obtaining a new signal function +.>：
(3)
Performing fast Fourier transform on each frame signal to obtain each frame frequency:
(4)
Where N' represents the number of points of the Fourier transform;
1-4 passing the energy spectrum through a set of triangular filter banks of Mel scale, calculating the logarithmic energy output by each filter bank:
(5)
Obtaining MFCC coefficients via Discrete Cosine Transform (DCT):
(6)
Wherein the method comprises the steps ofThe number of triangular filters is represented, and L represents the MFCC coefficient order;
1-5 comparing the obtained MFCC coefficients with the drug template one by one, and calculating the matching degree：
(7)
Wherein the method comprises the steps ofRepresenting the probability of color matching, +.>Representing a contour matching probability;
then arranging the matching results in the order from high to low in matching degree;
step (2), the robot moves to a fixed medicine storage position through SLAM technology, the field information is acquired and stored by adopting a visual servo method, and meanwhile, information reminding is sent to families through a network module, wherein the information comprises an identification result and the pose of the robot, and the specific steps are as follows:
2-1, constructing a grid map by using a GMapping method and combining information acquired by a laser radar, an inertial measurement unit and an odometer in a coordinate conversion mode;
2-2, acquiring semantic information of the environment through the identification and positioning functions of the depth camera, loading the semantic information into a storage space, and establishing a semantic map model;
2-3, constructing a semantic information updating model, so that the robot can update corresponding semantic information according to the change of the environment;
2-4, moving to a designated position by adopting a path planning method, and uploading the current pose of the robot in real time;
and (3) processing an image obtained by the depth camera by adopting a histogram equalization method, and enhancing the overall contrast of the image, wherein the method comprises the following specific steps of:
3-1 normalizing the gray level, calculating a Cumulative Distribution Function (CDF) of the gray values of the original image;
3-2 determination of mapping transformation function：
(8)
Where r represents the original image gray after normalization processing,a probability density function representing the gray level of the original image;
for digital images with discrete gray levels, the transformation functionCan be expressed as:
(9)
And (4) identifying a target object by adopting a multi-feature fusion object identification method, and determining the position information of the target medicine, wherein the method specifically comprises the following steps:
4-1 target object feature extraction scheme:
4-1-1 is based on color characteristics: the object and the surrounding environment have obvious color fall, the object and the surrounding environment can be subjected to threshold segmentation, and the key part and the surrounding environment are segmented in a color histogram statistics or color moment mode, so that an image of the target object is obtained;
4-1-2 are based on profile features: taking the edge pixels of the target object into a complete contour, extracting contour information of the object, analyzing a quick communication area of the image, carrying out polygon approximation on the complete contour, calculating the area in the contour by utilizing a rectangular bounding box, or matching the contour by utilizing Hu moment after fitting;
4-2 matching recognition algorithm:
4-2-1 uses an article fusion recognition algorithm based on DSmT (Dezert-Smarandache) reasoning to apply a data fusion idea to perform fusion processing on recognition information provided by different deep learning models, such as a pre-training model AlexNet, caffeNet, googleNet under a Caffe framework;
4-2-2, performing specific fine adjustment according to a classification and identification task by using an existing pre-training deep learning model, and performing evidence source reliability assignment on discrimination output of an image by using a deep learning network aiming at the problem of difficulty in reliability assignment construction in a DSmT theory;
4-2-3, applying DSmT combination theory to assign and fuse the credibility in a decision level layer, and matching with a model corresponding to a training set so as to realize accurate identification of the object;
step (5), firstly taking and placing the first medicine with the matching degree ordering by using the multi-degree-of-freedom soft claw mechanical arm, taking the first medicine into the field of vision of the aged, inquiring whether the first medicine is the required medicine, and if the first medicine meets the requirement, moving the first medicine to the front of the aged, specifically comprising the following steps:
5-1, constructing a visual servo closed-loop control system;
5-2, generating a feasible capturing track based on the rapid search random tree RRT, carrying out prior probability evaluation on the feasible track by utilizing probability theory in combination with Kalman filtering and modern control theory, and taking the track with the maximum capturing target probability as a capturing track of the mechanical arm;
step (6), if the selected medicine does not meet the requirement, repeating the steps (2) - (5), and sequentially selecting the medicine with the highest residual matching degree until the medicine meets the requirement or a command for stopping searching is received;
step (7), after the old people use, returning the medicine to the original placement position, moving to the initial position, and waiting for further instructions, wherein the method specifically comprises the following steps:
7-1, after the old man sends out a use completion instruction, the robot holds the medicine by the old man through visual recognition, and drives the mechanical arm to accurately grasp;
7-2, the robot moves to a fixed medicine storage point, and a return place is determined by using medicine position information stored during grabbing;
7-3 driving the mechanical arm to return the medicine to the original position;
7-4 the robot moves to the initial position and enters a standby state.
Claims (4)
1. A medicine identification acquisition method based on man-machine interaction is characterized by comprising the following steps:
step (1), determining required medicines according to voice signals, wherein the specific steps are as follows:
1-1 pre-emphasis: acquiring a voice signal, and passing the voice signal through a high-pass filter to boost a high-frequency part to obtain a signal function y (t);
y (t) =x (t) - μx (t-1) formula (1)
Where x (t) represents the original speech signal and μ represents the pre-emphasis parameter;
1-2, F sampling points are collected into frames, M is added between two adjacent frames m Overlapping regions of the sampling points, in which
1-3 adds Hamming window to the framed signal, using window function W (n):
wherein a is 0 A Hamming parameter, N represents the size of the frame;
a window function is applied to each frame, obtaining a new signal function S' (n):
s' (n) =s (n) ×w (n) formula (3)
Performing fast Fourier transform on each frame signal to obtain frequency spectrum X of each frame a (k):
Where N' represents the number of points of the Fourier transform;
1-4 passing the energy spectrum through a set of triangular filter banks of Mel scale, calculating the logarithmic energy s (m) output by each filter bank:
obtaining MFCC coefficient C (l) through discrete cosine transform:
wherein M is t Showing the number of triangular filters, L representing the MFCC coefficient order;
1-5 mapping voice information to a drug image template, comparing the obtained MFCC coefficients with the information of the drug image template one by one, and arranging according to the sequence of the matching degree from high to low, wherein the matching degree M d The operation method of (1) is as follows:
M d ＝P C +P O (7)
Wherein P is C Representing the probability of color matching, P O Representing a contour matching probability;
step (2), the robot moves to a fixed medicine storage position through SLAM technology, and site information is acquired and stored by adopting a visual servo method, wherein the site information comprises positioning information and image information, and meanwhile, information reminding is sent through a network module, wherein the information comprises an identification result and the pose of the robot;
and (3) processing the obtained image by adopting a histogram equalization method to enhance the overall contrast of the image, wherein the method comprises the following specific steps of:
3-1, normalizing the gray level, and calculating a cumulative distribution function of the gray value of the original image;
3-2 determines the mapping transformation function T (r):
wherein r represents the original image gray level after normalization processing, and p r A probability density function representing the gray level of the original image;
for digital images with discrete gray levels, the transform function T (r v ) Can be expressed as:
step (4), identifying a target object by adopting a multi-feature fusion object identification method, and determining the position information of a target medicine;
step (5), firstly taking and placing the first medicine with the matching degree ordering by using the multi-degree-of-freedom soft claw mechanical arm, taking the first medicine into the field of view of a patient, inquiring whether the first medicine is the required medicine, and if the first medicine meets the requirement, moving the first medicine to the front of the patient;
step (6), if the selected medicine does not meet the requirement, repeating the steps (2) - (5), and sequentially selecting the medicine with the highest residual matching degree until the medicine meets the requirement or a command for stopping searching is received;
step (7), after the patient finishes using, grabbing the medicine in the hand of the patient, returning to the original placement position, moving to the initial position, and waiting for further instructions:
7-1, after the old man sends out a use completion instruction, the old man holds the medicine by visual recognition, and the mechanical arm is driven to accurately grasp;
7-2, the robot moves to a fixed medicine storage point, and a return place is determined by using medicine position information stored during grabbing;
7-3 driving the mechanical arm to return the medicine to the original position;
7-4 the robot moves to the initial position and enters a standby state.
2. The method for identifying and acquiring the medicine based on human-computer interaction according to claim 1, wherein the specific steps of the step (2) are as follows:
2-1, constructing a grid map by using a GMapping method and combining information acquired by a laser radar, an inertial measurement unit and an odometer in a coordinate conversion mode;
2-2, acquiring semantic information of the environment through the identification and positioning functions of the depth camera, loading the semantic information into a storage space, and establishing a semantic map model;
2-3, constructing a semantic information updating model, so that the robot can update corresponding semantic information according to the change of the environment;
2-4, moving to a designated position by adopting a path planning method, and uploading the current pose of the robot in real time.
3. The method for identifying and acquiring the medicine based on the human-computer interaction according to claim 2, wherein the specific content of the step (4) is as follows:
4-1 target object feature extraction:
4-1-1, based on color characteristics, the object has obvious color fall with the surrounding environment, the object can be subjected to threshold segmentation, and a key part is segmented from the surrounding environment in a color histogram statistics or color moment mode, so that an image of the target object is obtained;
4-1-2 determining image edge pixels of a target object as a complete contour based on contour features, extracting contour information of the object, analyzing a quick communication area of the image, combining adjacent pixels with the same pixel value into a set to obtain edge features of different objects, performing polygon approximation on the complete contour, calculating an area in the contour by using a rectangular bounding box, fitting the contour, and then performing matching by using Hu moment;
4-2 matching recognition algorithm:
4-2-1 using an article fusion recognition algorithm based on DSmT reasoning to perform fusion processing on recognition information provided by a plurality of depth model optimization algorithms;
4-2-2, performing fine adjustment according to the classification recognition task by using an existing pre-training deep learning model, and performing evidence source credibility assignment on the discrimination output of the image by using a deep learning network;
4-2-3, applying DSmT combination theory to assign and fuse the credibility in a decision level layer, and matching with a model corresponding to a training set so as to realize accurate identification of the object;
4-3 storing target position information by using the depth camera for comparing information when returning the medicine.
4. The method for identifying and acquiring the medicine based on human-computer interaction according to claim 3, wherein the specific content in the step (5) is as follows:
5-1, constructing a visual servo closed-loop control system;
5-2 searching the target with the first matching degree determined in the step (1) by using a depth camera, and calculating a specific capturing position;
5-3, generating a feasible capturing track based on the rapid search random tree RRT, carrying out prior probability evaluation on the feasible capturing track by utilizing probability theory and Kalman filtering and modern control theory, and taking the track with the maximum capturing target probability as a capturing track of the mechanical arm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211486821.1A CN115730236B (en) | 2022-11-25 | 2022-11-25 | Medicine identification acquisition method, equipment and storage medium based on man-machine interaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211486821.1A CN115730236B (en) | 2022-11-25 | 2022-11-25 | Medicine identification acquisition method, equipment and storage medium based on man-machine interaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115730236A CN115730236A (en) | 2023-03-03 |
| CN115730236B true CN115730236B (en) | 2023-09-22 |
Family
ID=85298233
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211486821.1A Active CN115730236B (en) | 2022-11-25 | 2022-11-25 | Medicine identification acquisition method, equipment and storage medium based on man-machine interaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115730236B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118155176B (en) * | 2024-05-09 | 2024-09-06 | 江苏智搬机器人科技有限公司 | Automatic control method and system for transfer robot based on machine vision |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102323817A (en) * | 2011-06-07 | 2012-01-18 | 上海大学 | Service robot control platform system and multimode intelligent interaction and intelligent behavior realizing method thereof |
| CN109102547A (en) * | 2018-07-20 | 2018-12-28 | 上海节卡机器人科技有限公司 | Robot based on object identification deep learning model grabs position and orientation estimation method |
| CN112223288A (en) * | 2020-10-09 | 2021-01-15 | 南开大学 | Visual fusion service robot control method |
| CN114029963A (en) * | 2022-01-12 | 2022-02-11 | 北京具身智能科技有限公司 | Robot operation method based on visual and auditory fusion |
| CN115081567A (en) * | 2022-06-15 | 2022-09-20 | 东南大学 | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3739417A4 (en) * | 2018-06-08 | 2021-02-24 | Ankobot (Shenzhen) Smart Technologies Co., Ltd. | Navigation method, navigation system, mobile control system, and mobile robot |
| CN112711249B (en) * | 2019-10-24 | 2023-01-03 | 科沃斯商用机器人有限公司 | Robot positioning method and device, intelligent robot and storage medium |
-
2022
- 2022-11-25 CN CN202211486821.1A patent/CN115730236B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102323817A (en) * | 2011-06-07 | 2012-01-18 | 上海大学 | Service robot control platform system and multimode intelligent interaction and intelligent behavior realizing method thereof |
| CN109102547A (en) * | 2018-07-20 | 2018-12-28 | 上海节卡机器人科技有限公司 | Robot based on object identification deep learning model grabs position and orientation estimation method |
| CN112223288A (en) * | 2020-10-09 | 2021-01-15 | 南开大学 | Visual fusion service robot control method |
| CN114029963A (en) * | 2022-01-12 | 2022-02-11 | 北京具身智能科技有限公司 | Robot operation method based on visual and auditory fusion |
| CN115081567A (en) * | 2022-06-15 | 2022-09-20 | 东南大学 | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion |
Non-Patent Citations (4)
| Title |
|---|
| Jointly Learning of Visual and Auditory: A New Approach for RS Image and Audio Cross-Modal Retrieval;Mao Guo等;《IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing》;第4644-4654页 * |
| 张弘等.《图像处理与分析 第3版》.机械工业出版社,2019,第65-71页. * |
| 杨光红等.《无人系统基础》.机械工业出版社,2021,第108-113页. * |
| 郎坦•约瑟夫等.《智能系统与技术丛书 机器人系统设计与制作 Python语言实现 原书第2版》.机械工业出版社,2021,第152-155页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115730236A (en) | 2023-03-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110059558B (en) | Orchard obstacle real-time detection method based on improved SSD network | |
| CN109685066B (en) | Mine target detection and identification method based on deep convolutional neural network | |
| CN109785337B (en) | In-column mammal counting method based on example segmentation algorithm | |
| US9403278B1 (en) | Systems and methods for detecting and picking up a waste receptacle | |
| CN108268850B (en) | Big data processing method based on image | |
| JP2020038660A (en) | Learning method and learning device for detecting lane by using cnn, and test method and test device using the same | |
| CN106407993A (en) | Intelligent voice robot system based on image recognition technology and method thereof | |
| CN115730236B (en) | Medicine identification acquisition method, equipment and storage medium based on man-machine interaction | |
| CN112287896A (en) | Unmanned aerial vehicle aerial image target detection method and system based on deep learning | |
| CN115439458A (en) | Industrial image defect target detection algorithm based on depth map attention | |
| CN114677323A (en) | Semantic vision SLAM positioning method based on target detection in indoor dynamic scene | |
| CN111091101A (en) | High-precision pedestrian detection method, system and device based on one-step method | |
| CN113139437A (en) | Helmet wearing inspection method based on YOLOv3 algorithm | |
| CN117237902B (en) | Robot character recognition system based on deep learning | |
| CN113936210A (en) | Anti-collision method for tower crane | |
| CN112329615A (en) | Environment situation evaluation method for autonomous underwater visual target grabbing | |
| CN114359702A (en) | Method and system for identifying building violation of remote sensing image of homestead based on Transformer | |
| CN113034497A (en) | Vision-based thermos cup weld positioning detection method and system | |
| CN116229385A (en) | Vehicle 3D multi-target tracking method based on convolutional neural network and greedy algorithm | |
| CN114399515A (en) | Language description-based class-level target object 6D pose acquisition method and storage medium | |
| CN112069997B (en) | Unmanned aerial vehicle autonomous landing target extraction method and device based on DenseHR-Net | |
| CN107030690A (en) | A kind of mechanical arm barrier-avoiding method of view-based access control model | |
| CN115797397B (en) | Method and system for all-weather autonomous following of robot by target personnel | |
| CN114359493B (en) | Method and system for generating three-dimensional semantic map for unmanned ship | |
| CN115661694A (en) | Intelligent detection method, system, storage medium and electronic equipment for light-weight main transformer focusing on key characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |