CN115081567A - Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion - Google Patents
Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion Download PDFInfo
- Publication number
- CN115081567A CN115081567A CN202210675263.7A CN202210675263A CN115081567A CN 115081567 A CN115081567 A CN 115081567A CN 202210675263 A CN202210675263 A CN 202210675263A CN 115081567 A CN115081567 A CN 115081567A
- Authority
- CN
- China
- Prior art keywords
- medicine
- robot
- voice
- information
- rfid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000003814 drug Substances 0.000 title claims abstract description 217
- 238000000034 method Methods 0.000 title claims abstract description 74
- 229940079593 drug Drugs 0.000 title claims abstract description 36
- 230000004927 fusion Effects 0.000 title claims abstract description 19
- 238000000605 extraction Methods 0.000 claims abstract description 8
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 7
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 8
- 230000009467 reduction Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 239000000463 material Substances 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 2
- 230000001915 proofreading effect Effects 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 claims description 2
- 238000012377 drug delivery Methods 0.000 abstract 1
- 230000006872 improvement Effects 0.000 description 9
- 238000013459 approach Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K17/00—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations
- G06K17/0022—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations arrangements or provisious for transferring data to distant stations, e.g. from a sensing device
- G06K17/0029—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations arrangements or provisious for transferring data to distant stations, e.g. from a sensing device the arrangement being specially adapted for wireless interrogation of grouped or bundled articles tagged with wireless record carriers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
- G10L15/063—Training
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a medicine high-reliability identification method based on image, RFID and voice multi-element data fusion, which mainly comprises the following steps: firstly, a user provides a voice instruction for a robot, the robot acquires medicine name information through voice recognition, and an image recognition link is entered; in the image recognition link, image enhancement and RGB extraction are carried out on a picture in front of the robot, position information of a corresponding medicine is obtained, and then the position information is sent to the robot and enters an RFID recognition link; in the RFID identification link, the robot grabs the medicine according to the position information of the target medicine and then places the medicine on the RFID identification for identification, so that the medicine delivery link is entered after the accuracy is ensured; in the drug delivery link, the robot delivers drugs to the user and informs the user of the drug name, usage amount and properties in a voice synthesis mode to serve as a third guarantee. The invention can realize high-reliability automatic identification of the medicine, thereby helping the old and the disabled with inconvenient actions to take the medicine on time.
Description
Technical Field
The invention belongs to the field of service robots, and particularly relates to a medicine high-reliability identification method based on image, RFID and voice multi-data fusion.
Background
With the development of smart home concepts, the service robot receives more and more attention. Many elderly and disabled people have the problem of inconvenience in their legs and feet, and therefore it is desirable that the service robot be able to automatically recognize medicines and help users to take medicines. Medicine identification is an indispensable technology in the process of taking and placing medicines by a service robot, whether the robot can realize high-accuracy identification on the medicines or not is judged, and whether the service robot has reliable home service capability or not is marked. The aim of the medicine identification task is that the robot automatically confirms whether a certain medicine is a medicine which needs to be eaten by a user currently, and specifically, the robot can automatically distinguish the medicine by using various information, accurately take the correct medicine for the user and remind the user of using the medicine. An early solution in the field of drug identification of service robots is to use an image recognition algorithm based on deep learning to train a large number of drug pictures for identification. The accuracy of the neural network in the method for identifying the articles basically cannot reach 99%, and if a user takes the medicine three times a day, the probability of completely not making a mistake in one month is about 40.5%, obviously, the accuracy does not meet the requirement of the user on the accuracy of the service robot identification, and the risk of eating wrong medicines for the user cannot be accepted. In recent years, the development of deep learning technology is also continuously improving the recognition accuracy, but the recognition accuracy of the existing model still cannot meet the requirements of the service robot.
Disclosure of Invention
Aiming at the problems in the prior art, the invention aims to provide a high-reliability medicine identification method based on image, RFID and voice multi-data fusion.
In order to realize the purpose of the invention, the technical scheme adopted by the invention is as follows:
a medicine high-reliability identification method based on image, RFID and voice multi-element data fusion comprises the following steps:
step 1: the method comprises the steps that voice recognition is carried out to obtain a medicine name, and if the voice recognition fails, the voice broadcast requests a user to send an instruction again;
step 2: acquiring position information of a target medicine through image identification, and entering an alternative step 1 if the identification fails;
and step 3: placing the medicine in an RFID identification area for identification, entering an alternative step 2 if no identification result exists, and entering an alternative step 1 if identification fails;
and 4, step 4: the robot sends the medicine to the user;
and 5: the robot broadcasts the medicine related information by voice, requests the user to judge whether the recognition is successful or not and provides a voice command, if the user judges that the recognition is failed, the robot puts the medicine back to the original place and returns to the step 2, otherwise, the whole medicine taking process is finished, and the robot enters a standby state.
Alternative step 1: the active light source is started, all medicine positions are obtained through the feature labels by using vision, the robot grabs the medicine bottles one by one according to the medicine positions until the RFID identification results are matched, the active light source is closed, the step 4 is carried out, if no result is matched, it is determined that no target medicine exists in the current environment, and the standby state is carried out after voice broadcasting is carried out;
alternative step 2: if the voice broadcast fails to acquire the RFID information, the user is asked to identify the RFID information in detail, and the step 4 is entered.
As an improvement of the present invention, in the step 1, the method for speech recognition includes:
in order to save computing resources and achieve the maximum utilization of the computing resources, the invention uses an off-line speech recognition method to work, and adopts a pre-trained model to perform speech feature matching to achieve the aim of speech recognition.
In the training stage, a CPU is used for collecting voice information of related medicine names and instructions in advance, and acoustic features corresponding to the voice information are stored in a Flash space after software noise reduction processing and feature extraction; and after voice information is collected in real time in a voice recognition stage, noise reduction processing and feature extraction are carried out, corresponding comparison is carried out on the real-time voice related features and the pre-training features to obtain feature error coefficients, and the voice recognition is considered to have the related instruction or the medicine name under the condition that the feature error coefficients are smaller than 254 for each word.
Because the voice recognition part needs to be matched with the image recognition link, in order to save storage space, the voice information is directly bound with the medicine name and the image characteristics in the pre-training part so that the link is directly communicated with the image recognition link.
As an improvement of the present invention, in step 1, the method for voice synthesis and broadcast includes:
and initially performing voice synthesis calculation by using the cloud computing platform according to the medicine information and the related instruction information, and storing the voice synthesized by the cloud computing platform as a wav format file for standby.
The method comprises the steps that an independent sub-thread is used for working in a voice broadcasting part, the period of the sub-thread is 0.1s, related voice files are broadcasted according to instructions once the shared instruction variable is found to be changed in the sub-thread, in order to avoid conflicts among threads, timing waiting can be carried out in the voice broadcasting thread, a control quantity is set to be an empty instruction until the complete voice broadcasting is finished, and other tasks are allowed to write in new instructions.
Meanwhile, in other tasks, if the voice broadcasting function needs to be used, the currently shared instruction variable needs to be monitored, the instruction is written when the instruction variable is in a no-instruction state, and otherwise, the instruction is written after the instruction variable is in the no-instruction state.
As an improvement of the present invention, in the step 2, the method for image recognition includes:
firstly, distortion removal processing is carried out on an image, then an original image is subjected to image enhancement by using a Gray World algorithm, RGB values of all regions are extracted in the processed image by using a space-time scale equalization mode and are compared with RGB characteristic ranges provided by a voice part, and when more than 95% of pixel points in a certain rectangular range are in the corresponding RGB characteristic ranges, a target medicine is identified.
As an improvement of the present invention, in the step 2, the method for estimating the position includes:
in order to save hardware resources, the center of mass and the pixel number information of the target medicine in the undistorted image obtained by identification are used for confirming the direction information and the distance information of the target medicine compared with the robot according to the following formula:
sinθ aim ＝K θ (Col aim -Col 0 )
in the formula [ theta ] aim Is the horizontal angle information of the target drug compared with the direction right in front of the robot, d is the linear distance information between the target drug and the robot, Col aim The number of columns, Col, where the detected centroid of the target drug is located in the image 0 Column number of center pixels of camera, num pix Number of pixel points occupied by the medicine, K θ And K d The parameters corresponding to the angle information and the distance information are converted into constants, and the specific values of the constants are determined by the actual sizes of the camera and the target medicine.
And confirming the position information of the target medicine compared with the robot according to the distance and direction information by the following formula:
Pos x ＝d sinθ aim
Pos y ＝d cosθ aim
pos in the formula x And Pos y The horizontal coordinate value of the medicine position relative to the robot position and the vertical coordinate value Pos z The relative height information of the robot and the medicine can be directly obtained.
Since the results of RFID identification and image identification may not be consistent, the CPU may use alternative step 2 to obtain the coordinates of all drugs that the robot may contact in case of an identification error. Using this approach can avoid huge computational effort in case the identification is accurate.
As an improvement of the present invention, in step 3, the method for the robot to perform RFID identification on the medicine includes:
firstly, resolving a target angle of each joint by a target medicine coordinate by the robot, grabbing the medicine and then placing the medicine in an RFID (radio frequency identification) identification area, structurally meeting Pieper criteria by the selected robot, and performing analytical inverse kinematics solution by using the following formula:
θ 1 ＝θ aim
Trans x ＝l 2 +l 3 sinθ 2 +l 4 sin(θ 2 +θ 3 )+l 5 sinθ tool
Trans y ＝l 1 +l 3 cosθ 2 +l 4 cos(θ 2 +θ 3 )+l 5 cosθ tool
θ 4 ＝θ tool -θ 2 -θ 3
in the formula [ theta ] 1 、θ 2 、θ 3 、θ 4 Is a target angle value, theta tool The required Z-axis angle of the end tool coordinate system to the world coordinate system, l 1 、l 2 、l 3 、l 4 、l 5 Length of the robot bar, Trans x And Trans y The horizontal linear distance and the vertical linear distance of the medicine compared with the robot are respectively obtained by using the following formula:
Trans y ＝Pos z
when the medicine is located in the RFID identification area, the independent RFID identifier is used for carrying out TTL serial port communication with the robot to obtain medicine related information (including name law usage amount and characters), the robot compares target medicine information received by the CPU and target medicine information identified by the RFID to confirm whether the currently-captured medicine is consistent with the target medicine or not, and an identification result is sent to the CPU for next processing.
As an improvement of the present invention, in step 4, the method for the robot to bring the medicine to the user includes:
firstly, a laser radar and a mapping algorithm are used for matching with a PC (personal computer) to build a map of an indoor environment, characteristic positions of conventional positions of several users such as a bedside and a chair side are marked in the map, and numbers of the characteristic positions are stored.
The robot grasps the medicine, then a user provides a voice instruction to the robot to enable the robot to confirm which characteristic position the medicine needs to be delivered to, then the robot confirms the self pose by using laser radar information and map information, then path planning is carried out by using an A-algorithm according to the identification position, the robot is controlled to move according to the path obtained by planning, the robot periodically judges the current robot position in the moving process to achieve the purpose of path closed-loop control, and after the robot moves to the target position, the robot lifts the medicine to the top of the head of the robot to ensure that the user can take the medicine.
The method for extracting the characteristic points of the map information acquired by the laser radar comprises the following steps:
P result ＝|P R -P r |
in the above formula if P result Greater than a set threshold value P t The point is judged to be one of the feature points and can be used for feature matching.
As an improvement of the present invention, in step 5, the robot voice proofreading method includes:
the robot judges whether the command variable of the voice broadcasting part is empty or not, waits if the command variable is not empty, modifies the command variable into broadcasting of the name, usage and property of the current medicine after the completion of the waiting, starts voice recognition to obtain the feedback of a user, judges whether the medicine is correct or not, returns to the step 2 if the medicine fed back by the user is incorrect, otherwise, judges that the task is completed, and enters a standby state.
As an improvement of the present invention, in the alternative step 1, the method for obtaining the medicine position through the feature tag includes:
the invention uses a holographic reflective material as the characteristic label of the medicine, the RGB value of the characteristic label under the camera viewing angle under the condition of active lighting of the robot is far larger than that of other objects, and the binaryzation segmentation is carried out by taking the RGB value as a threshold value, so that the robot can be ensured to quickly and accurately find the target medicine.
In the identification task, firstly, an active light source at the position of a camera is started for lighting, a vision part carries out image binarization processing according to the lower limit of a feature label RGB value, then noise points are removed by using a DBSACN method, different pixel point clusters are distinguished, each cluster corresponds to one medicine bottle, and then the position estimation is carried out according to the picture information of different medicine bottles by using the following formula:
position _ medium in the formula x For the estimated position of the medicine relative to the robotCoordinate system X-axis coordinate, Position _ coordinate y For estimated Y-axis coordinates, K, of the position of the drug relative to the robot coordinate system distance For the distance estimation of the parameters determined by the camera and the size parameters of the holographic feature label, num pix_medicine For the number of pixels in the estimated cluster, K rot Estimating parameters for the angle, the parameters being determined by the camera, Col medicine Array coordinate values for the centroid positions of the estimated clusters, Col middle The coordinate values of the whole picture center row.
And after the estimation of the position of the medicine is finished, the estimated positions are sent to the robot one by one in a queue mode to be grabbed and identified by the RFID.
As an improvement of the present invention, in the alternative step 2, the method for obtaining the medicine position through the feature tag includes:
when the current medicine is judged not to have the RFID label or the label is damaged, a piece of information is lost in the identification process, the probability of identification error is relatively high, and a user needs to be informed of careful identification through voice.
Compared with the prior art, the technical scheme of the invention has the following beneficial technical effects: aiming at the problem of medicine identification, the invention provides a medicine high-reliability identification method based on image, RFID and voice multi-data fusion, and the image identification part uses a method for estimating the position of a target object based on size information and picture mass center information based on a monocular camera, so that the distance estimation method needs few computing resources and hardware resources. The RFID identification uses an independent RFID identifier to communicate with the robot in a TTL serial port communication mode, so that the accuracy of medicine identification by the method is remarkably improved compared with that of a pure image identification method. In the aspect of robot movement and positioning, a PC-assisted mapping method is used to ensure that map information acquired by the robot is very accurate, and meanwhile, a 2D point cloud feature point extraction method is used to save time required by robot pose perception and strengthen real-time performance and reliability of robot movement. In the voice information part, the medicine related property information is synthesized in a cloud computing mode to assist a user in autonomously distinguishing the medicines, and the third information is used for guaranteeing the medicine identification accuracy.
Drawings
FIG. 1 is a flow chart of a method for identifying drugs with high reliability based on image, RFID and voice multi-element data fusion according to the invention;
FIG. 2 is a flow chart of a speech recognition method employed by the present invention;
fig. 3 is a flow chart of a voice broadcasting method adopted by the present invention;
FIG. 4 is a flow chart of a method for a robot to bring a medication to a user in accordance with the present invention;
FIG. 5 is a flow chart of a method of voice proofing employed by the present invention;
FIG. 6 is a flow chart of a method for obtaining drug locations via a feature tag as utilized by the present invention;
Detailed Description
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
As shown in FIG. 1, the invention provides a highly reliable drug identification method based on image, RFID and voice multi-element data fusion, which comprises the following detailed steps:
(1) and (3) voice recognition:
as shown in fig. 2, in order to save the computing resources and maximize the utilization of the computing resources, the present invention uses an off-line speech recognition method to work, and uses a pre-trained model to perform speech feature matching to achieve the purpose of speech recognition.
In the training stage, a CPU is used for collecting voice information of related medicine names and instructions in advance, and acoustic features corresponding to the voice information are stored in a Flash space after software noise reduction processing and feature extraction; and after voice information is collected in real time in a voice recognition stage, noise reduction processing and feature extraction are carried out, corresponding comparison is carried out on the real-time voice related features and the pre-training features to obtain feature error coefficients, and the voice recognition is considered to have the related instruction or the medicine name under the condition that the feature error coefficients are smaller than 254 for each word.
Because the voice recognition part needs to be matched with the image recognition link, in order to save storage space, the voice information is directly bound with the medicine name and the image characteristics in the pre-training part so that the link is directly communicated with the image recognition link.
If the voice recognition is wrong, the high school user using the voice broadcasting method needs to resend the voice command and restart the voice recognition.
(2) Speech synthesis and voice broadcast:
as shown in fig. 3, initially, a cloud computing platform is used for voice synthesis calculation according to the medicine information and the related instruction information, and the voice synthesized by the cloud computing platform is stored as a file in the wav format for standby.
The method comprises the steps that independent sub-threads are used for working in a voice broadcasting part, the period of the sub-threads is 0.1s, related voice files are broadcasted according to instructions once shared command variables are found to change in the sub-threads, in order to avoid conflicts among the threads, timing waiting can be carried out in the voice broadcasting threads, control quantities are set to be null instructions until complete voice broadcasting is finished, and other threads are allowed to write instructions.
If the condition that needs to use the voice broadcast function appears in other steps can judge the current broadcast condition, just can write the instruction and call the voice broadcast function after broadcasting the state and becoming the idle command.
(3) Image recognition and distance estimation:
firstly, distortion removal processing is carried out on an image, then an original image is subjected to image enhancement by using a Gray World algorithm, RGB values of all regions are extracted in the processed image by using a space-time scale equalization mode and are compared with RGB characteristic ranges provided by a voice part, and when more than 95% of pixel points in a certain rectangular range are in the corresponding RGB characteristic ranges, a target medicine is identified.
In order to save hardware resources, the center of mass and the pixel number information of the target medicine in the undistorted image obtained by identification are used for confirming the direction information and the distance information of the target medicine compared with the robot according to the following formula:
sinθ aim ＝K θ (Col aim -Col 0 )
in the formula [ theta ] aim Is the horizontal angle information of the target drug compared with the direction right in front of the robot, d is the linear distance information between the target drug and the robot, Col aim The number of columns, Col, where the detected centroid of the target drug is located in the image 0 Column number of center pixels of camera, num pix Number of pixel points occupied by the medicine, K θ And K d The parameters corresponding to the angle information and the distance information are converted into constants respectively, and the specific numerical values of the constants are determined by the camera and the target medicine.
And confirming the position information of the target medicine compared with the robot according to the distance and direction information by the following formula:
Pos x ＝d sinθ aim
Pos y ＝d cosθ aim
pos in the formula x And Pos y The vertical coordinate value Pos is the coordinate value on the X-axis and Y-axis of the medicine position relative to the robot position z The relative height information of the robot and the medicine can be directly obtained.
Since the results of RFID identification and image identification may not be consistent, the CPU may use alternative step 2 to obtain the coordinates of all drugs that the robot may contact in case of an identification error. Using this approach can avoid huge computational effort in case the identification is accurate.
The situation of image recognition failure may occur under the influence of illumination conditions, and the problem of light pollution and the like can be caused by not only increasing the cost but also providing an independent light source for the medicine position due to cost limitation.
As shown in fig. 6, in order to solve the problem, the invention uses a holographic reflective material as the feature label of the medicine, and the RGB value of the feature label at the view angle of the camera is far greater than that of other objects under the condition of active lighting by the robot, so that the robot can quickly and accurately find the target medicine by performing binarization segmentation by using the RGB value as a threshold value.
In the identification task, firstly, an active light source at the position of a camera is started for lighting, a vision part carries out image binarization processing according to the lower limit of a feature label RGB value, then noise points are removed by using a DBSACN method, different pixel point clusters are distinguished, each cluster corresponds to one medicine bottle, and then the position estimation is carried out according to the picture information of different medicine bottles by using the following formula:
position _ medium in the formula x For the estimated Position of the drug relative to the X-axis coordinate of the robot coordinate system, Position _ diameter y For estimated positions of the drug with respect to the Y-axis coordinates, K, of the robot coordinate system distance For the distance estimation of the parameters, which are determined by the camera and the size parameters of the holographic signature, num pix_medicine For the number of pixels in the estimated cluster, K rot For angle estimation of the parameters, which are determined by the camera, Col medicine Array coordinate value of centroid position for the estimated cluster, Col middle The coordinate values of the whole picture center row.
And after the estimation of the position of the medicine is finished, the estimated positions are sent to the robot one by one in a queue mode to be grabbed and identified by the RFID.
(4) The robot carries out RFID identification on the medicine:
the method comprises the following steps of performing RFID identification under the condition that a target medicine is obtained through image identification, firstly resolving a target angle of each joint through target medicine coordinates by a robot, grabbing the medicine and then placing the medicine in an RFID identification area, meeting Pieper criteria on the selected robot structure, and performing inverse kinematics solution by an analytic method according to the following formula:
θ 1 ＝θ aim
Trans x ＝l 2 +l 3 sinθ 2 +l 4 sin(θ 2 +θ 3 )+l 5 sinθ tool
Trans y ＝l 1 +l 3 cosθ 2 +l 4 cos(θ 2 +θ 3 )+l 5 cosθ tool
θ 4 ＝θ tool -θ 2 -θ 3
in the formula [ theta ] 1 、θ 2 、θ 3 、θ 4 Is the target angle value, θ tool The required Z-axis angle of the end tool coordinate system to the world coordinate system, l 1 、l 2 、l 3 、l 4 、l 5 Length of the robot bar, Trans x And Trans y The horizontal linear distance and the vertical linear distance of the medicine compared with the robot are respectively obtained by using the following formula:
Trans y ＝Pos z
when the medicine is located in the RFID identification area, the independent RFID identifier is used for carrying out TTL serial port communication with the robot to obtain medicine related information (including name law usage amount and characters), the robot compares target medicine information received by the CPU and target medicine information identified by the RFID to confirm whether the currently-captured medicine is consistent with the target medicine or not, and an identification result is sent to the CPU for next processing.
If the RFID cannot identify the medicine, the damage or the loss of the medicine RFID is judged, a piece of information is lost in the identification process, the probability of identification error is relatively high, and a user needs to be informed of careful identification through voice.
If the RFID information obtained by identification is inconsistent with the target RFID identification information, it is determined that the result of the image processing part is incorrect, as shown in fig. 6, an active light source at the position of the camera is turned on to perform lighting, the visual part performs image binarization processing according to the lower limit of the RGB value of the feature label, then uses a dbscan method to remove noise points and distinguish different pixel point clusters, each cluster corresponds to one medicine bottle, and then uses the following formula to perform position estimation according to the picture information of different medicine bottles:
position _ medium in the formula x Position-coordinate for the estimated Position of the drug relative to the X-axis coordinate of the robot coordinate system y For estimated positions of the drug with respect to the Y-axis coordinates, K, of the robot coordinate system distance For the distance estimation of the parameters determined by the camera and the size parameters of the holographic feature label, num pix_medicine For the number of pixels in the estimated cluster, K rot Estimating parameters for the angle, the parameters being determined by the camera, Col medicine Array coordinate values for the centroid positions of the estimated clusters, Col middle The coordinate values of the whole picture center row.
And after the estimation of the position of the medicine is finished, the estimated positions are sent to the robot one by one in a queue mode to be grabbed and identified by the RFID.
(5) The robot takes the medicine to the user:
firstly, a laser radar and a mapping algorithm are used for matching with a PC (personal computer) to build a map of an indoor environment, characteristic positions of conventional positions of several users such as a bedside and a chair side are marked in the map, and numbers of the characteristic positions are stored.
The robot grasps the medicine, then a user provides a voice instruction to the robot to enable the robot to confirm which characteristic position the medicine needs to be delivered to, then the robot confirms the self pose by using laser radar information and map information, then path planning is carried out by using an A-algorithm according to the identification position, the robot is controlled to move according to the path obtained by planning, the robot periodically judges the current robot position in the moving process to achieve the purpose of path closed-loop control, and after the robot moves to the target position, the robot lifts the medicine to the top of the head of the robot to ensure that the user can take the medicine.
The method for extracting the characteristic points of the map information acquired by the laser radar comprises the following steps:
P result ＝|P R -P r |
in the above formula if P result Greater than a set threshold value P t The point is judged to be one of the feature points and can be used for feature matching.
(6) And (3) carrying out voice calibration:
as shown in fig. 5, the robot determines whether the command variable of the voice broadcast part is empty, waits if not, modifies the command variable to the broadcast of the current drug name, usage amount and property after waiting, starts voice recognition to obtain the feedback of the user, returns to step 2 if the user determines that the recognition is incorrect, otherwise, determines that the task is finished, and enters a standby state.
It should be noted that the above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention, and all equivalent substitutions or substitutions made on the above-mentioned technical solutions belong to the scope of the present invention.
Claims (10)
1. A medicine high-reliability identification method based on image, RFID and voice multi-element data fusion is characterized by comprising the following steps:
step 1: the method comprises the steps that voice recognition is carried out to obtain a medicine name, and if the voice recognition fails, the voice broadcast requests a user to send an instruction again;
step 2: acquiring position information of a target medicine through image identification, and entering an alternative step 1 if the identification fails;
and step 3: placing the medicine in an RFID identification area for identification, entering an alternative step 2 if no identification result exists, and entering an alternative step 1 if identification fails;
and 4, step 4: the robot sends the medicine to the user;
and 5: the robot broadcasts the medicine related information by voice, requests the user to judge whether the identification is successful and provides a voice instruction, if the user judges that the identification is failed, the robot puts the medicine back to the original place and returns to the step 2, otherwise, the whole medicine taking process is finished, and the robot enters a standby state;
alternative step 1: the active light source is started, all medicine positions are obtained through the feature labels by using vision, the robot grabs the medicine bottles one by one according to the medicine positions until the RFID identification results are matched, the active light source is closed, the step 4 is carried out, if no result is matched, it is determined that no target medicine exists in the current environment, and the standby state is carried out after voice broadcasting is carried out;
alternative step 2: and (4) if the RFID information cannot be acquired through voice broadcasting, asking the user to identify in detail, and entering step 4.
2. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in step 1, the speech recognition method includes:
in the training stage, a CPU is used for collecting voice information of related medicine names and instructions in advance, and acoustic features corresponding to the voice information are stored in a Flash space after software noise reduction processing and feature extraction; and after voice information is collected in real time in a voice recognition stage, noise reduction processing and feature extraction are carried out, corresponding comparison is carried out on the real-time voice related features and the pre-training features to obtain feature error coefficients, and the voice recognition is considered to have the related instruction or the medicine name under the condition that the feature error coefficients are smaller than 254 for each word.
3. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in step 1, the method for voice synthesis and broadcast includes:
firstly, performing voice synthesis calculation by using a cloud computing platform according to medicine information and related instruction information, and storing voice synthesized by the cloud computing platform as a wav format file for later use;
the method comprises the steps that an independent sub-thread is used for working in a voice broadcasting part, the period of the sub-thread is 0.1s, related voice files are broadcasted according to instructions once a shared instruction variable is found to change in the sub-thread, the instruction variable is set to be in an instruction-free state after the broadcasting of the corresponding voice files is finished, and other tasks are allowed to write new instructions;
meanwhile, in other tasks, if the voice broadcasting function needs to be used, the currently shared instruction variable needs to be monitored, when the instruction variable is in a no-instruction state, the instruction is written, and otherwise, the instruction enters a waiting state until the instruction variable becomes in the no-instruction state and then the instruction is written.
4. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the step 2, the image recognition method comprises the following steps:
firstly, distortion removal processing is carried out on an image, then an original image is subjected to image enhancement by using a Gray World algorithm, RGB values of all regions are extracted in the processed image in a space-time scale equalization mode and are compared with RGB characteristic ranges provided by a voice part, and when more than 95% of pixel points in a certain rectangular range are in the corresponding RGB characteristic ranges, a target medicine is identified.
5. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the step 2, the position estimation method includes:
confirming the direction information and the distance information of the target medicine compared with the robot according to the following formula by the centroid and the pixel number information of the identified target medicine in the undistorted image:
sinθ aim ＝K θ (Col aim -Col 0 )
in the formula theta aim Is the horizontal angle information of the target drug compared with the direction right in front of the robot, d is the linear distance information between the target drug and the robot, Col aim The number of columns, Col, where the detected centroid of the target drug is located in the image 0 Column number of center pixels of camera, num pix Number of pixel points occupied by the medicine, K θ And K d The parameters of the corresponding angle information and the distance information are converted into constants respectively, and the specific numerical values of the constants are determined by the actual sizes of the camera and the target medicine;
and confirming the position information of the target medicine compared with the robot according to the distance and direction information by the following formula:
Pos x ＝d sinθ aim
Pos y ＝d cosθ aim
pos in the formula x And Pos y The horizontal coordinate value of the medicine position relative to the robot position and the vertical coordinate value Pos z The relative height information of the robot and the medicine can be directly obtained.
6. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the step 3, the method for the robot to perform RFID identification on the medicine comprises the following steps:
firstly, resolving a target angle of each joint by a target medicine coordinate by the robot, grabbing the medicine and then placing the medicine in an RFID (radio frequency identification) identification area, structurally meeting Pieper criteria by the selected robot, and performing analytical inverse kinematics solution by using the following formula to obtain the target angle of each joint:
θ 1 ＝θ aim
Trans x ＝l 2 +l 3 sinθ 2 +l 4 sin(θ 2 +θ 3 )+l 5 sinθ tool
Trans y ＝l 1 +l 3 cosθ 2 +l 4 cos(θ 2 +θ 3 )+l 5 cosθ tool
θ 4 ＝θ tool -θ 2 -θ 3
in the formula [ theta ] 1 、θ 2 、θ 3 、θ 4 Is the target angle value, θ tool The required Z-axis angle of the end tool coordinate system to the world coordinate system, l 1 、l 2 、l 3 、l 4 、l 5 Length of the robot bar, Trans x And Trans y The horizontal linear distance and the vertical linear distance of the medicine compared with the robot are respectively obtained by using the following formula:
Trans y ＝Pos z
when the medicine is located in the RFID identification area, the independent RFID identifier is used for carrying out TTL serial port communication with the robot to obtain medicine related information (including name law usage amount and characters), the robot compares target medicine information received by the CPU and target medicine information identified by the RFID to confirm whether the currently-captured medicine is consistent with the target medicine or not, and an identification result is sent to the CPU for next processing.
7. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the step 4, the method for the robot to take the medicine to the user comprises the following steps:
firstly, establishing a map of an indoor environment by using a laser radar and a mapping algorithm in cooperation with a PC (personal computer), marking characteristic positions of a plurality of user conventional positions such as a bedside, a chair side and the like in the map, and simultaneously storing the serial numbers of the characteristic positions;
after the robot grasps the medicine, a user provides a voice instruction to the robot to enable the robot to confirm which characteristic position the medicine needs to be delivered to, then the robot confirms the self pose by using laser radar information and map information, then path planning is carried out by using an A-algorithm according to the identification position, the robot is controlled to move according to the path obtained by planning, the robot periodically judges the current robot position in the moving process to achieve the purpose of path closed-loop control, and after the robot moves to the target position, the robot lifts the medicine to the top of the head of the robot to ensure that the user takes the medicine;
the method for extracting the characteristic points of the map information acquired by the laser radar comprises the following steps:
P result ＝|P R -P r |
in the above formula if P result Greater than a set threshold value P t The point is judged to be one of the feature points and can be used for feature matching.
8. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the step 5, the robot voice proofreading method includes:
the robot judges whether the command variable of the voice broadcasting part is empty or not, waits if the command variable is not empty, modifies the command variable into the broadcasting of the current medicine name, usage amount and character after the waiting is finished, starts voice recognition to obtain the feedback of a user, judges whether the medicine is correct or not, returns to the step 2 if the medicine fed back by the user is incorrect, otherwise, judges that the task is finished, and enters a standby state.
9. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the alternative step 1, the method for acquiring the position of the medicine through the image recognition feature tag comprises the following steps:
the method has the advantages that a holographic reflective material is used as the characteristic label of the medicine, the RGB value of the characteristic label under the visual angle of a camera under the condition of active lighting of the robot is far larger than that of other objects, and the target medicine can be found quickly and accurately by using the RGB value as a threshold value to carry out binarization segmentation;
in the identification task, firstly, an active light source at the position of a camera is started for lighting, a vision part carries out image binarization processing according to the lower limit of a feature label RGB value, then noise points are removed by using a DBSACN method, different pixel point clusters are distinguished, each cluster corresponds to one medicine bottle, and then the position estimation is carried out according to the picture information of different medicine bottles by using the following formula:
position _ medium in the formula x Position-coordinate for the estimated Position of the drug relative to the X-axis coordinate of the robot coordinate system y For estimated Y-axis coordinates, K, of the position of the drug relative to the robot coordinate system distance Estimating parameters for the distance, the parameters being characterized by the camera and the hologramLabel size parameter determination, num pix_medicine For the number of pixels in the estimated cluster, K rot Estimating parameters for the angle, the parameters being determined by the camera, Col medicine Array coordinate values for the centroid positions of the estimated clusters, Col middle Coordinate values of the whole picture center array;
and after the estimation of the position of the medicine is finished, the estimated positions are sent to the robot one by one in a queue mode to be grabbed and identified by the RFID.
10. The method for identifying the medicine with high reliability based on the fusion of the image, the RFID and the voice multi-element data as the claim 1, is characterized in that: in the alternative step 2, the method for feeding back the information loss condition includes:
when the current medicine is judged not to have the RFID label or the label is damaged, a piece of information is lost in the identification process, the probability of identification error is relatively high, and a user needs to be informed of careful identification through voice.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210675263.7A CN115081567A (en) | 2022-06-15 | 2022-06-15 | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210675263.7A CN115081567A (en) | 2022-06-15 | 2022-06-15 | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115081567A true CN115081567A (en) | 2022-09-20 |
Family
ID=83251620
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210675263.7A Pending CN115081567A (en) | 2022-06-15 | 2022-06-15 | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115081567A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115730236A (en) * | 2022-11-25 | 2023-03-03 | 杭州电子科技大学 | Drug identification acquisition method, device and storage medium based on man-machine interaction |
-
2022
- 2022-06-15 CN CN202210675263.7A patent/CN115081567A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115730236A (en) * | 2022-11-25 | 2023-03-03 | 杭州电子科技大学 | Drug identification acquisition method, device and storage medium based on man-machine interaction |
| CN115730236B (en) * | 2022-11-25 | 2023-09-22 | 杭州电子科技大学 | Medicine identification acquisition method, equipment and storage medium based on man-machine interaction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108885459B (en) | Navigation method, navigation system, mobile control system and mobile robot | |
| CN114494892B (en) | Goods shelf commodity display information identification method, device, equipment and storage medium | |
| WO2022095440A1 (en) | Self-driving-oriented human-machine collaborative perception method and system | |
| CN105678322A (en) | Sample labeling method and apparatus | |
| CN109087261B (en) | Face correction method based on unlimited acquisition scene | |
| CN110929795B (en) | Method for quickly identifying and positioning welding spot of high-speed wire welding machine | |
| CN115081567A (en) | Medicine high-reliability identification method based on image, RFID and voice multi-element data fusion | |
| CN109034694B (en) | Production raw material intelligent storage method and system based on intelligent manufacturing | |
| CN109919157A (en) | A kind of vision positioning method and device | |
| CN109407709A (en) | A kind of meeting camera shooting automatic tracking system based on Kinect bone track algorithm | |
| CN113538574A (en) | Pose positioning method, device and equipment and computer readable storage medium | |
| CN110718227A (en) | Multi-mode interaction based distributed Internet of things equipment cooperation method and system | |
| CN112040198A (en) | Intelligent water meter reading identification system and method based on image processing | |
| CN113936340B (en) | AI model training method and device based on training data acquisition | |
| Begum et al. | A probabilistic model of overt visual attention for cognitive robots | |
| CN116052222A (en) | Cattle face recognition method for naturally collecting cattle face image | |
| CN113505629A (en) | Intelligent storage article recognition device based on light weight network | |
| CN115884480B (en) | Multi-optical axis cloud desk lamp control method, equipment and storage medium based on image processing | |
| CN108074264A (en) | A kind of classification multi-vision visual localization method, system and device | |
| CN113172636A (en) | Automatic hand-eye calibration method and device and storage medium | |
| CN111079535A (en) | Human skeleton action recognition method and device and terminal | |
| CN116206189A (en) | Curved surface graphic identification code and identification method thereof | |
| CN115439821A (en) | Positioning and tracking method and system based on multi-view vision AGV | |
| CN109977710B (en) | Method for accurately positioning bookshelf to which book belongs, electronic equipment and storage medium | |
| CN111432131B (en) | Photographing frame selection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |