CN115810203A - Obstacle avoidance identification method, system, electronic equipment and storage medium - Google Patents
Obstacle avoidance identification method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115810203A CN115810203A CN202211634663.XA CN202211634663A CN115810203A CN 115810203 A CN115810203 A CN 115810203A CN 202211634663 A CN202211634663 A CN 202211634663A CN 115810203 A CN115810203 A CN 115810203A
- Authority
- CN
- China
- Prior art keywords
- image set
- image
- human body
- processing
- obstacle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012545 processing Methods 0.000 claims abstract description 51
- 210000000746 body region Anatomy 0.000 claims abstract description 46
- 238000007781 pre-processing Methods 0.000 claims abstract description 25
- 210000004394 hip joint Anatomy 0.000 claims abstract description 16
- 210000000323 shoulder joint Anatomy 0.000 claims abstract description 14
- 238000002372 labelling Methods 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 54
- 238000010586 diagram Methods 0.000 claims description 10
- 238000009432 framing Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 7
- 230000007935 neutral effect Effects 0.000 claims description 7
- 230000004888 barrier function Effects 0.000 claims description 3
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 230000003993 interaction Effects 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000009471 action Effects 0.000 description 7
- 235000013601 eggs Nutrition 0.000 description 6
- 239000003550 marker Substances 0.000 description 5
- 210000003414 extremity Anatomy 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000001624 hip Anatomy 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 210000002414 leg Anatomy 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000010248 power generation Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method, a system, electronic equipment and a storage medium for avoiding and identifying obstacles, wherein the method comprises the following steps: performing frame processing on an input video to be identified to obtain an image set to be identified; preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set; inputting the preprocessed image set into a pre-trained key point recognition model to perform key point labeling processing to obtain a labeled image set, wherein the key points comprise nose tips, shoulder joint points and hip joint points of a human body; acquiring the appearance position and the appearance time of the obstacle; and carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result. The embodiment of the invention can reduce the key points needing to be identified, thereby improving the processing efficiency of avoiding identification, and can be widely applied to the technical field of artificial intelligence.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to an obstacle avoidance identification method, an obstacle avoidance identification system, electronic equipment and a storage medium.
Background
With the rapid development of artificial intelligence technology, the man-machine interface of man-machine interaction games gradually develops from ways of controlling the movement of game characters by a keyboard, controlling aiming and shooting actions by a mouse and the like to ways of controlling the movement or starting skills of game characters by a microphone, and controlling the movement of any character by acquiring user actions through a camera or a sensor and the like. In a camera-based human-computer interaction obstacle avoidance game, whether a user successfully avoids an obstacle needs to be judged. However, the identification avoiding method in the related technology has the disadvantages of multiple key points of human bones to be identified, large calculation amount, complex calculation, multiple steps and low calculation efficiency. In view of the above, there is a need to solve the technical problems in the related art.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, an electronic device, and a storage medium for obstacle avoidance identification, so as to improve processing efficiency of the obstacle avoidance identification.
In one aspect, the present invention provides an obstacle avoidance identification method, including:
performing framing processing on an input video to be recognized to obtain an image set to be recognized;
preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
inputting the preprocessed image set into a pre-trained key point recognition model to perform key point labeling processing to obtain a labeled image set, wherein the key points comprise nose tips, shoulder joint points and hip joint points of a human body;
acquiring the appearance position and the appearance time of the obstacle;
and carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result.
Optionally, the performing frame division processing on the input video to be recognized to obtain an image set to be recognized includes:
performing video division processing on an input video to be identified to obtain a plurality of video segments;
and extracting each video segment according to a preset frame rate to obtain an image set to be identified.
Optionally, before the pre-processing is performed on the image set to be recognized through the pre-trained human body region marker model to obtain a pre-processed image set, training is performed on the human body region marker model, and the method includes:
acquiring a human body image training set;
carrying out marking frame selection processing on each image in the human body image training set to obtain a marked image set;
inputting the marked image set into the human body region marking model to obtain a marking result;
determining a loss value of training according to the marking result and the marks of the marked image set;
and updating the parameters of the human body region marking model according to the loss value.
Optionally, the pre-processing the image set to be recognized through a pre-trained human body region marker model to obtain a pre-processed image set includes:
inputting each frame of image in the image set to be identified into the human body region marking model to obtain an image marking frame set;
cutting the image set to be identified according to the image marking frame set to obtain a cut image set;
performing thermodynamic diagram generation processing on the cutting image set to obtain a thermodynamic image set;
determining the cut image set and the thermal image set as a pre-processing image set.
Optionally, before the preprocessing image set is input into a pre-trained key point recognition model for key point annotation processing to obtain an annotated image set, training the key point recognition model is included, and the method includes:
acquiring a recognition image training set, wherein the recognition image training set comprises a cutting training image and a thermal training image;
inputting the recognition image training set into the key point recognition model, and recognizing human nose tips, shoulder joint points and hip joint points in the recognition image training set to obtain a key point recognition result;
determining a loss value of training according to the key point identification result and a label of an identification image training set;
and updating the parameters of the key point identification model according to the loss value.
Optionally, the performing, in accordance with the appearance position and the appearance time of the obstacle, an obstacle avoidance identification process on the labeled image set to obtain an obstacle avoidance identification result includes:
selecting an image of a human body at a neutral position from the annotation image set as a reference image;
acquiring a mark key point according to the identification result of the reference image, and determining a reference vertical line according to the mark key point;
acquiring an image to be judged from the marked image set according to the appearance time of the obstacle;
and carrying out avoidance judgment according to the appearance position of the obstacle, the reference vertical line and the key point identification result of the image to be judged to obtain an obstacle avoidance identification result.
On the other hand, the embodiment of the invention also provides an obstacle avoidance identification system, which comprises: the first module is used for performing framing processing on an input video to be identified to obtain an image set to be identified;
the second module is used for preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
the third module is used for inputting the preprocessed image set into a pre-trained key point recognition model to perform key point annotation processing to obtain an annotated image set;
the fourth module is used for acquiring the appearance position and the appearance time of the barrier;
and the fifth module is used for carrying out the obstacle avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result.
Optionally, the first module is configured to perform framing processing on an input video to be identified to obtain an image set to be identified, and includes:
the device comprises a first unit, a second unit and a third unit, wherein the first unit is used for carrying out video division processing on an input video to be identified to obtain a plurality of video segments;
and the second unit is used for extracting each video segment according to the preset frame rate to obtain an image set to be identified.
On the other hand, the embodiment of the invention also discloses an electronic device, which comprises a processor and a memory;
the memory is used for storing programs;
the processor executes the program to implement the method as described above.
On the other hand, the embodiment of the invention also discloses a computer readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to realize the method.
In another aspect, an embodiment of the present invention further discloses a computer program product or a computer program, where the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and the computer instructions executed by the processor cause the computer device to perform the foregoing method.
Compared with the prior art, the invention adopting the technical scheme has the following technical effects: performing frame processing on an input video to be identified to obtain an image set to be identified; preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set; inputting the preprocessed image set into a pre-trained key point recognition model to perform key point labeling processing to obtain a labeled image set, wherein the key points comprise nose tips, shoulder joint points and hip joint points of a human body; acquiring the appearance position and the appearance time of the obstacle; and carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result. According to the embodiment of the invention, the evading and identifying treatment is carried out through the human body region marking model and the key point identifying model, so that the key points needing to be identified are reduced, and the processing efficiency of evading and identifying is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of an obstacle avoidance identification method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a neutral-position key point location identification provided in an embodiment of the present application;
fig. 3 is a schematic diagram of a key point identification avoiding towards the right according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In the related art, for a camera-based game for human-computer interaction and obstacle avoidance, multiple skeletal joint points of a human body, for example, coordinates of different joints such as a head, a neck, a caudal vertebra and a hip, need to be identified, and a standard human body skeletal key point model is preset for identification, so as to judge whether a user successfully avoids an obstacle, but the method has the disadvantages of multiple human body skeletal key points needing to be identified, large calculation amount, complex calculation, multiple steps and low calculation efficiency.
In view of this, referring to fig. 1, an embodiment of the present invention provides an obstacle avoidance identification method, including:
s101, performing framing processing on an input video to be identified to obtain an image set to be identified;
s102, preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
s103, inputting the preprocessed image set into a pre-trained key point recognition model to perform key point labeling processing to obtain a labeled image set, wherein the key points comprise nose tips, shoulder joint points and hip joint points of a human body;
s104, acquiring the appearance position and the appearance time of the obstacle;
and S105, carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result.
In the embodiment of the invention, the input video to be recognized is subjected to frame division to obtain the image set to be recognized, and then the image set to be recognized is subjected to frame-by-frame analysis, so that a real-time obstacle avoidance recognition result can be obtained. The embodiment of the invention preprocesses the image set to be recognized through a pre-trained human body region marking model to obtain a preprocessed image set, wherein the preprocessing mainly filters the background through the human body region marking model, marks and cuts out the human body region so as to reduce the influence of background noise; and generating a color thermodynamic diagram according to the cut human body region image, inputting the human body region image and the color thermodynamic diagram as a preprocessing image set into a key point identification model for key point identification, wherein the key points comprise human nose tips, shoulder joint points and hip joint points, and identifying and labeling to obtain a labeled image set. And then acquiring the appearance position and the appearance time of the obstacle, and carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result. According to the embodiment of the invention, a small number of key points are identified through the human body region marking model and the key point identification model, so that the identification processing time is reduced, and the identification avoiding efficiency is improved.
Further as a preferred embodiment, in the step S101, the performing frame division processing on the input video to be recognized to obtain an image set to be recognized includes:
performing video division processing on an input video to be identified to obtain a plurality of video segments;
and extracting each video segment according to a preset frame rate to obtain an image set to be identified.
In the embodiment of the invention, in the camera-based human-computer interaction obstacle avoidance game, a camera for acquiring the limb actions of a user is arranged on intelligent terminal equipment in advance, the intelligent terminal equipment can comprise portable computers, intelligent televisions, tablet computers and other intelligent equipment capable of performing human-computer interaction, the camera is arranged on the intelligent terminal equipment for acquiring the limb actions of the user, and the camera is used for acquiring a video to be identified according to the starting time of the human-computer interaction obstacle avoidance game. And then, performing video division on the input video to be identified, wherein the video can be divided according to the time period or the video length. In the embodiment of the invention, the video is divided once every minute to obtain a plurality of video segments, and each video segment is extracted according to the preset frame rate, wherein the preset frame rate can be designed autonomously according to actual conditions, and each frame of image is extracted to obtain the image set to be identified.
As a further preferred embodiment, before the pre-processing the image set to be recognized by the pre-trained body region labeling model to obtain a pre-processed image set, the method includes training the body region labeling model, and includes:
acquiring a human body image training set;
carrying out marking frame selection processing on each image in the human body image training set to obtain a marked image set;
inputting the marked image set into the human body region marking model to obtain a marking result;
determining a loss value of training according to the marking result and the marks of the marked image set;
and updating the parameters of the human body region marking model according to the loss value.
In the embodiment of the invention, the image set to be recognized is preprocessed through the pre-trained human body region mark model, and before the preprocessing, the human body region mark model needs to be trained. The embodiment of the invention trains the human body region marker model by acquiring the human body image training set, wherein the human body image training set comprises a large number of human body images, and the human body region marker model can be constructed by using a convolutional neural network. According to the embodiment of the invention, a large number of human body images are manually marked to select a pair of rectangular coordinate points, the marked image set is input into the human body region marking model, and four vertex coordinates of the human body marking frame are output to obtain a marking result. And determining a training loss value according to the marking result and a rectangular coordinate point which marks the marking image set in advance, updating parameters of the human body region marking model according to the loss value, and stopping training the human body region marking model when the loss value, namely a training error, is smaller than a preset value to obtain the trained human body region marking model. According to the embodiment of the invention, the human body marking frame selection is carried out on the image through the human body area marking model, so that the background noise of the image is reduced, and the accuracy of obstacle avoidance and identification is improved.
Further as a preferred embodiment, the preprocessing the image set to be recognized by the pre-trained human body region labeling model to obtain a preprocessed image set includes:
inputting each frame of image in the image set to be identified into the human body region marking model to obtain an image marking frame set;
cutting the image set to be identified according to the image marking frame set to obtain a cut image set;
generating a thermodynamic diagram for the cutting image set to obtain a thermodynamic image set;
determining the cut image set and the thermal image set as a pre-processing image set.
In the embodiment of the invention, a pre-trained human body region mark model is used for preprocessing an image set to be recognized, each frame of image in the image set to be recognized is firstly input into the human body region mark model to obtain a marked image mark frame set, and then the image set to be recognized is cut according to the image mark frame set, so that redundant background images are removed, and a cut image set is obtained, wherein the cut image set is an image set only containing a human body region. And then, performing thermodynamic diagram generation processing on each cutting image in the cutting image set to obtain a thermodynamic image set. The segmentation image generation thermodynamic diagram can be processed by using a thermodynamic generation formula as follows:
in the formula, Y xy Represents the value of the filter coefficient, p, at the image coordinate point (x, y) x Is the midpoint value of the horizontal direction of the image, p y Is the midpoint value in the vertical direction of the image,
the standard deviation is indicated.
And adding the value calculated by the thermal power generation formula to the original pixel value (Rxy, gxy, bxy) of the original image coordinate (x, y) to obtain a new image pixel value (YRxy, YGxy, YBxy), namely the thermal power image. And finally, determining the cutting image set and the thermodynamic image set as a preprocessing image set, and inputting the preprocessing image set into the key point recognition model for training, so that the recognition precision of the key point recognition model is improved, and the accuracy of obstacle avoidance recognition is improved.
As a further preferred embodiment, before the preprocessing image set is input to a pre-trained keypoint identification model for keypoint annotation processing to obtain an annotated image set, training the keypoint identification model is included, and the method includes:
acquiring a recognition image training set, wherein the recognition image training set comprises a cutting training image and a thermal training image;
inputting the recognition image training set into the key point recognition model, and recognizing human nose tips, shoulder joint points and hip joint points in the recognition image training set to obtain a key point recognition result;
determining a loss value of training according to the key point identification result and the label of the identification image training set;
and updating the parameters of the key point identification model according to the loss value.
In the embodiment of the invention, the key point identification model can be built by adopting a convolutional neural network, and is trained by an identification image training set, wherein the identification image training set comprises a large number of cutting training images and thermal training images. The embodiment of the invention also adopts a supervised learning method to train the key point recognition model, carries out key point marking on the cut training image, inputs the recognition image training set into the key point recognition model, recognizes the nose tip, shoulder joint points and hip joint points of the human body in the recognition image training set through the key point recognition model, and outputs to obtain a key point recognition result. And determining a training loss value according to a key point recognition result output by the model and a label of a pre-marked recognition image training set, and updating parameters of the key point recognition model according to the loss value. According to the embodiment of the invention, the obstacle avoidance identification can be carried out by identifying a small number of key points, so that the complex calculation process is reduced, and the identification efficiency is improved.
As a further preferred embodiment, the performing, as a result of the obstacle avoidance recognition, the avoidance recognition processing on the labeled image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance recognition result includes:
selecting an image of a human body at a neutral position from the marked image set as a reference image;
acquiring a mark key point according to the identification result of the reference image, and determining a reference vertical line according to the mark key point;
acquiring an image to be judged from the marked image set according to the appearance time of the obstacle;
and carrying out avoidance judgment according to the appearance position of the obstacle, the reference vertical line and the key point identification result of the image to be judged to obtain an obstacle avoidance identification result.
In the embodiment of the invention, the image of the human body in the neutral position is selected from the annotation image set as the reference image, in the camera-based human-computer interaction obstacle avoidance game, a user can be reminded to stand at the neutral position through prompting on the display interface of the intelligent terminal equipment, and then the image of the human body in the neutral position is obtained through the camera and is used as the reference image. Referring to fig. 2, the mark key points including a nose tip a, a left shoulder joint point B, a right shoulder joint point C, a left hip joint point D, and a right hip joint point E are obtained according to the recognition result of the reference image, and a reference vertical line is determined according to the mark key points. In the embodiment of the invention, a vertical line is made in the reference image through the middle point of the point D and the point E and is marked as a reference vertical line L, the image to be judged is obtained from the marked image set according to the appearance time of the obstacle, and each marked image in the marked image set is provided with a time label, so that the image to be judged can be obtained according to the appearance time of the obstacle to avoid and identify the obstacle, and the complexity of calculation is reduced. And finally, carrying out avoidance judgment according to the appearance position of the obstacle, the reference vertical line and the key point identification result of the image to be judged to obtain an obstacle avoidance identification result.
In a feasible embodiment, a camera for acquiring the limb actions of the user is arranged on the intelligent terminal device in advance, and a human body region marking model and a key point identification model for identifying the tip of the nose, the shoulder joint points and the hip joint points of the human body are trained in advance. The intelligent terminal device referred to herein may be: mobile phones with cameras, iPads, computers, televisions, intelligent interactive large screens and the like. The camera on the intelligent terminal equipment is used for monitoring user information, and when the body image above the hip joint of the user is obtained, the user is identified as an operation user; when the limb images of a plurality of users are acquired simultaneously (namely when a plurality of users are in the same frame), the user with the largest occupied area is taken as the operation user. When the operation user determines to start the game, the operation user firstly prompts the user to stand straight through the game interfaceIn a neutral position. And extracting images above hip joints of the operating user through a pre-trained human body region marking model and a key point identification model, and identifying and marking key points to serve as reference images. Referring to fig. 3, when an obstacle appears on the left side of the screen, it is assumed that the operating user avoids to the right, where VD AC Is the horizontal distance projected from the point A to the point C in the vertical direction, VD AB (not shown) is the horizontal distance projected in the vertical direction from point a to point B. Then when the point B moves to the right of the reference vertical line L and VD AC /VD AB And when the value is less than the threshold value, judging that the user avoids rightward. Similarly, when the point C moves to the left of the reference vertical line L, and VD AB /VD AC When the value is less than the threshold value, the user can be judged to hide leftwards; and (4) combining and comparing the positions and time periods of the obstacles in the game to judge whether the user successfully completes one avoiding operation. The threshold in the embodiment of the present invention may be set to 0.5, and may be adjusted according to actual situations. The embodiment of the invention can be applied to a human-computer interaction real-time game, and the human-computer interaction game is one of applications in the intelligent television on the intelligent television with the camera. The game rule is that the user keeps his legs still and avoids the free falling eggs by moving his shoulders left and right. After the game is started, the camera identifies the game user in real time. When eggs in free falling bodies appear in the game interface and before the eggs fall to 1/2 of the height of the game interface, if the game user is identified to complete one avoiding action according to the obstacle avoiding and identifying method of the embodiment of the invention, the eggs are successfully avoided, and if the user is identified to not perform the avoiding action according to the obstacle avoiding and identifying method of the embodiment of the invention, the eggs are not successfully avoided, the eggs are smashed, and the game fails.
On the other hand, the embodiment of the invention also provides an obstacle avoidance identification system, which comprises: the first module is used for performing framing processing on an input video to be identified to obtain an image set to be identified;
the second module is used for preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
the third module is used for inputting the preprocessed image set into a pre-trained key point recognition model to perform key point annotation processing to obtain an annotated image set;
the fourth module is used for acquiring the appearance position and the appearance time of the barrier;
and the fifth module is used for carrying out the obstacle avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result.
Further as a preferred embodiment, the first module is configured to perform framing processing on an input video to be recognized to obtain an image set to be recognized, and includes:
the device comprises a first unit, a second unit and a third unit, wherein the first unit is used for carrying out video division processing on an input video to be identified to obtain a plurality of video segments;
and the second unit is used for extracting each video segment according to the preset frame rate to obtain an image set to be identified.
It can be understood that the contents in the above-mentioned embodiment of the obstacle avoidance identification method are all applicable to this embodiment of the system, the functions implemented in this embodiment of the system are the same as those in the above-mentioned embodiment of the obstacle avoidance identification method, and the beneficial effects achieved by this embodiment of the system are also the same as those achieved by the above-mentioned embodiment of the obstacle avoidance identification method.
Corresponding to the method of fig. 1, an embodiment of the present invention further provides an electronic device, which includes a processor and a memory; the memory is used for storing programs; the processor executes the program to implement the method as described above.
Corresponding to the method of fig. 1, the embodiment of the present invention also provides a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.
Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and executed by the processor, causing the computer device to perform the method illustrated in fig. 1.
In summary, the embodiments of the present invention have the following advantages: according to the embodiment of the invention, only three joint points of the nose tip, the shoulder joint and the hip joint are identified through the human body area marking model and the key point identification model, and the vertical distance and the ratio thereof between the two points are further identified and judged, so that whether a user can successfully avoid an obstacle or not can be judged, the complexity of avoiding identification is reduced, and the processing efficiency of avoiding identification is improved.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer given the nature, function, and interrelationships of the modules. Accordingly, those of ordinary skill in the art will be able to practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is to be determined from the appended claims along with their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description of the specification, reference to the description of "one embodiment," "some embodiments," "an example," "a specific example," or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. An obstacle avoidance identification method, the method comprising:
performing frame processing on an input video to be identified to obtain an image set to be identified;
preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
inputting the preprocessed image set into a pre-trained key point recognition model to perform key point labeling processing to obtain a labeled image set, wherein the key points comprise nose tips, shoulder joint points and hip joint points of a human body;
acquiring the appearance position and the appearance time of the obstacle;
and carrying out avoidance identification processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance identification result.
2. The method according to claim 1, wherein the framing the input video to be recognized to obtain the image set to be recognized comprises:
performing video division processing on an input video to be identified to obtain a plurality of video segments;
and extracting each video segment according to a preset frame rate to obtain an image set to be identified.
3. The method according to claim 1, wherein before the pre-processing the image set to be recognized by the pre-trained body region labeling model to obtain a pre-processed image set, training the body region labeling model comprises:
acquiring a human body image training set;
carrying out marking frame selection processing on each image in the human body image training set to obtain a marked image set;
inputting the marked image set into the human body region marking model to obtain a marking result;
determining a loss value of training according to the marking result and the marking of the marked image set;
and updating the parameters of the human body region marking model according to the loss value.
4. The method according to claim 1, wherein the preprocessing the image set to be recognized through a pre-trained human body region labeling model to obtain a preprocessed image set comprises:
inputting each frame of image in the image set to be identified into the human body region marking model to obtain an image marking frame set;
cutting the image set to be identified according to the image marking frame set to obtain a cut image set;
performing thermodynamic diagram generation processing on the cutting image set to obtain a thermodynamic image set;
determining the cut image set and the thermal image set as a pre-processing image set.
5. The method according to claim 1, wherein before inputting the preprocessed image set into a pre-trained keypoint recognition model for keypoint annotation processing to obtain an annotated image set, the method comprises training the keypoint recognition model, and the steps comprise:
acquiring a recognition image training set, wherein the recognition image training set comprises a cutting training image and a thermal training image;
inputting the recognition image training set into the key point recognition model, and recognizing human nose tips, shoulder joint points and hip joint points in the recognition image training set to obtain a key point recognition result;
determining a loss value of training according to the key point identification result and the label of the identification image training set;
and updating the parameters of the key point identification model according to the loss value.
6. The method according to claim 1, wherein the avoiding and recognizing the labeled image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoiding and recognizing result comprises:
selecting an image of a human body at a neutral position from the marked image set as a reference image;
acquiring a mark key point according to the identification result of the reference image, and determining a reference vertical line according to the mark key point;
acquiring an image to be judged from the marked image set according to the appearance time of the obstacle;
and carrying out avoidance judgment according to the appearance position of the obstacle, the reference vertical line and the key point identification result of the image to be judged to obtain an obstacle avoidance identification result.
7. An obstacle avoidance identification system, the system comprising:
the first module is used for performing framing processing on an input video to be identified to obtain an image set to be identified;
the second module is used for preprocessing the image set to be recognized through a pre-trained human body region mark model to obtain a preprocessed image set;
the third module is used for inputting the preprocessed image set into a pre-trained key point recognition model to perform key point annotation processing to obtain an annotated image set;
the fourth module is used for acquiring the appearance position and the appearance time of the barrier;
and the fifth module is used for carrying out avoidance recognition processing on the marked image set according to the appearance position and the appearance time of the obstacle to obtain an obstacle avoidance recognition result.
8. The system according to claim 7, wherein the first module is configured to perform framing processing on an input video to be recognized to obtain a set of images to be recognized, and includes:
the device comprises a first unit, a second unit and a third unit, wherein the first unit is used for carrying out video division processing on an input video to be identified to obtain a plurality of video segments;
and the second unit is used for extracting each video segment according to the preset frame rate to obtain an image set to be identified.
9. An electronic device, comprising a memory and a processor;
the memory is used for storing programs;
the processor executing the program implements the method of any one of claims 1 to 6.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211634663.XA CN115810203B (en) | 2022-12-19 | 2022-12-19 | Obstacle avoidance recognition method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211634663.XA CN115810203B (en) | 2022-12-19 | 2022-12-19 | Obstacle avoidance recognition method, system, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115810203A true CN115810203A (en) | 2023-03-17 |
| CN115810203B CN115810203B (en) | 2024-05-10 |
Family
ID=85486148
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211634663.XA Active CN115810203B (en) | 2022-12-19 | 2022-12-19 | Obstacle avoidance recognition method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115810203B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106598237A (en) * | 2016-11-30 | 2017-04-26 | 宇龙计算机通信科技(深圳)有限公司 | Game interaction method and device based on virtual reality |
| CN107894773A (en) * | 2017-12-15 | 2018-04-10 | 广东工业大学 | A kind of air navigation aid of mobile robot, system and relevant apparatus |
| CN108985259A (en) * | 2018-08-03 | 2018-12-11 | 百度在线网络技术（北京）有限公司 | Human motion recognition method and device |
| US20190374857A1 (en) * | 2018-06-08 | 2019-12-12 | Brian Deller | System and method for creation, presentation and interaction within multiple reality and virtual reality environments |
| KR102151494B1 (en) * | 2019-11-12 | 2020-09-03 | 가천대학교 산학협력단 | Active feedback virtual reality system for brain and physical health through user's activity |
| CN112052786A (en) * | 2020-09-03 | 2020-12-08 | 上海工程技术大学 | Behavior prediction method based on grid division skeleton |
| CN112473121A (en) * | 2020-11-13 | 2021-03-12 | 海信视像科技股份有限公司 | Display device and method for displaying dodging ball based on limb recognition |
| US20210089040A1 (en) * | 2016-02-29 | 2021-03-25 | AI Incorporated | Obstacle recognition method for autonomous robots |
| CN112990057A (en) * | 2021-03-26 | 2021-06-18 | 北京易华录信息技术股份有限公司 | Human body posture recognition method and device and electronic equipment |
| KR20210110064A (en) * | 2020-02-28 | 2021-09-07 | 엘지전자 주식회사 | Moving Robot and controlling method |
| CN113709411A (en) * | 2020-05-21 | 2021-11-26 | 陈涛 | Sports auxiliary training system of MR intelligent glasses based on eye movement tracking technology |
| CN114582030A (en) * | 2022-05-06 | 2022-06-03 | 湖北工业大学 | Behavior recognition method based on service robot |
| CN115213903A (en) * | 2022-07-19 | 2022-10-21 | 深圳航天科技创新研究院 | Mobile robot path planning method and device based on obstacle avoidance |
| CN115424236A (en) * | 2022-08-15 | 2022-12-02 | 南京航空航天大学 | Pedestrian crossing trajectory prediction method integrating pedestrian intention and social force models |
-
2022
- 2022-12-19 CN CN202211634663.XA patent/CN115810203B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210089040A1 (en) * | 2016-02-29 | 2021-03-25 | AI Incorporated | Obstacle recognition method for autonomous robots |
| CN106598237A (en) * | 2016-11-30 | 2017-04-26 | 宇龙计算机通信科技(深圳)有限公司 | Game interaction method and device based on virtual reality |
| CN107894773A (en) * | 2017-12-15 | 2018-04-10 | 广东工业大学 | A kind of air navigation aid of mobile robot, system and relevant apparatus |
| US20190374857A1 (en) * | 2018-06-08 | 2019-12-12 | Brian Deller | System and method for creation, presentation and interaction within multiple reality and virtual reality environments |
| CN108985259A (en) * | 2018-08-03 | 2018-12-11 | 百度在线网络技术（北京）有限公司 | Human motion recognition method and device |
| KR102151494B1 (en) * | 2019-11-12 | 2020-09-03 | 가천대학교 산학협력단 | Active feedback virtual reality system for brain and physical health through user's activity |
| KR20210110064A (en) * | 2020-02-28 | 2021-09-07 | 엘지전자 주식회사 | Moving Robot and controlling method |
| CN113709411A (en) * | 2020-05-21 | 2021-11-26 | 陈涛 | Sports auxiliary training system of MR intelligent glasses based on eye movement tracking technology |
| CN112052786A (en) * | 2020-09-03 | 2020-12-08 | 上海工程技术大学 | Behavior prediction method based on grid division skeleton |
| CN112473121A (en) * | 2020-11-13 | 2021-03-12 | 海信视像科技股份有限公司 | Display device and method for displaying dodging ball based on limb recognition |
| CN112990057A (en) * | 2021-03-26 | 2021-06-18 | 北京易华录信息技术股份有限公司 | Human body posture recognition method and device and electronic equipment |
| CN114582030A (en) * | 2022-05-06 | 2022-06-03 | 湖北工业大学 | Behavior recognition method based on service robot |
| CN115213903A (en) * | 2022-07-19 | 2022-10-21 | 深圳航天科技创新研究院 | Mobile robot path planning method and device based on obstacle avoidance |
| CN115424236A (en) * | 2022-08-15 | 2022-12-02 | 南京航空航天大学 | Pedestrian crossing trajectory prediction method integrating pedestrian intention and social force models |
Non-Patent Citations (2)
| Title |
|---|
| ZHAO, JIANGBO 等: "Path Planning and Evaluation for Obstacle Avoidance of Manipulator Based on Improved Artificial Potential Field and Danger Field", 《PROCEEDINGS OF THE 33RD CHINESE CONTROL AND DECISION CONFERENCE (CCDC 2021)》, 31 December 2021 (2021-12-31) * |
| 仇婷: "人机合作多模式兵乓球机器人竞赛系统", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》, no. 8, 15 August 2015 (2015-08-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115810203B (en) | 2024-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111126272B (en) | Posture acquisition method, and training method and device of key point coordinate positioning model | |
| WO2020207281A1 (en) | Method for training posture recognition model, and image recognition method and apparatus | |
| CN112184705B (en) | Human body acupuncture point identification, positioning and application system based on computer vision technology | |
| CN103941866B (en) | Three-dimensional gesture recognizing method based on Kinect depth image | |
| KR100886407B1 (en) | Information processing apparatus and control method therefor | |
| CN111222486B (en) | Training method, device and equipment for hand gesture recognition model and storage medium | |
| Aran et al. | Signtutor: An interactive system for sign language tutoring | |
| CN111240476B (en) | Interaction method and device based on augmented reality, storage medium and computer equipment | |
| CN109145802B (en) | Kinect-based multi-person gesture man-machine interaction method and device | |
| CN110531853B (en) | Electronic book reader control method and system based on human eye fixation point detection | |
| CN111178170B (en) | Gesture recognition method and electronic equipment | |
| CN110561399A (en) | Auxiliary shooting device for dyskinesia condition analysis, control method and device | |
| CN114549557A (en) | Portrait segmentation network training method, device, equipment and medium | |
| CN113781271A (en) | Makeup teaching method and device, electronic equipment and storage medium | |
| CN114120170A (en) | Video picture analysis method, apparatus, device, medium, and program product | |
| Chen et al. | Real-time human-computer interaction using eye gazes | |
| CN115994944A (en) | Three-dimensional key point prediction method, training method and related equipment | |
| CN111078008B (en) | Control method of early education robot | |
| CN104933278A (en) | Multi-channel interactive method and system used for speech disorder rehabilitation training | |
| CN115810203A (en) | Obstacle avoidance identification method, system, electronic equipment and storage medium | |
| CN114779942B (en) | Virtual reality immersive interaction system, device and method | |
| CN116704603A (en) | Action evaluation correction method and system based on limb key point analysis | |
| Patel et al. | Gesture Recognition Using MediaPipe for Online Realtime Gameplay | |
| CN114779925A (en) | Sight line interaction method and device based on single target | |
| Vančo et al. | Gesture identification for system navigation in 3D scene |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |