CN115423865A - Obstacle detection method, obstacle detection device, mowing robot, and storage medium - Google Patents
Obstacle detection method, obstacle detection device, mowing robot, and storage medium Download PDFInfo
- Publication number
- CN115423865A CN115423865A CN202210910652.3A CN202210910652A CN115423865A CN 115423865 A CN115423865 A CN 115423865A CN 202210910652 A CN202210910652 A CN 202210910652A CN 115423865 A CN115423865 A CN 115423865A
- Authority
- CN
- China
- Prior art keywords
- information
- obstacle
- semantic segmentation
- parallax
- parallax information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims description 40
- 230000011218 segmentation Effects 0.000 claims abstract description 81
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000012549 training Methods 0.000 claims description 27
- 238000003709 image segmentation Methods 0.000 claims description 9
- 230000004888 barrier function Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims 1
- 230000000007 visual effect Effects 0.000 abstract description 11
- 230000000875 corresponding effect Effects 0.000 description 20
- 230000006870 function Effects 0.000 description 14
- 230000008569 process Effects 0.000 description 10
- 238000013527 convolutional neural network Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 239000000284 extract Substances 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 244000025254 Cannabis sativa Species 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000009194 climbing Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/12—Acquisition of 3D measurements of objects
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application discloses a method and a device for detecting obstacles, a mowing robot and a storage medium, wherein the method comprises the following steps: the method comprises the steps of obtaining at least one initial image aiming at a current scene, extracting parallax information and semantic segmentation information in the initial image, matching the semantic segmentation information with the parallax information to determine parallax information of an obstacle in the parallax information, and calculating the position and the size of the obstacle according to the parallax information of the obstacle. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the accuracy is high.
Description
Technical Field
The application relates to the technical field of computers, in particular to a method and a device for detecting obstacles, a mowing robot and a storage medium.
Background
Along with the continuous improvement of living standard of people, people have higher and higher requirements on leisure environment, places such as private gardens, parks and playgrounds become the best places for people to enjoy leisure and entertainment, but lawns such as private gardens, parks and playgrounds need to be irregularly trimmed so as to ensure the attractiveness. At present, a mowing robot is generally adopted to replace manual trimming.
However, lawn mowing robots often encounter various obstacles during operation, and existing lawn mowing robots can detect obstacles through recognition devices arranged on the robots, such as methods based on laser radar, millimeter wave radar and multi-sensor information fusion.
Disclosure of Invention
The embodiment of the application provides an obstacle detection method and device, a mowing robot and a storage medium, which can detect the position and the size of an obstacle only by a vision sensor and have higher accuracy.
In a first aspect, an embodiment of the present application provides an obstacle detection method, including:
acquiring at least one initial image for a current scene;
extracting parallax information and semantic segmentation information in the initial image;
matching the semantic segmentation information with the parallax information to determine obstacle parallax information in the parallax information;
and calculating the position and the size of the obstacle according to the obstacle parallax information.
In an embodiment, before matching the semantic segmentation information with the disparity information, the method further comprises:
training an image segmentation model through a parallax information sample and label information corresponding to the sample;
and segmenting the parallax information in the initial image into foreground parallax information and background parallax information through the trained image segmentation model.
In an embodiment, the matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information includes:
matching the semantic segmentation information with the foreground parallax information, and judging whether an obstacle exists in the foreground;
if so, acquiring obstacle parallax information in the foreground parallax information;
and if the semantic segmentation information does not exist, matching the semantic segmentation information with the background parallax information to acquire barrier parallax information in the background parallax information.
In an embodiment, the extracting semantic segmentation information in the initial image includes:
extracting edge information and depth information in the initial image;
semantic segmentation information in the initial image is determined based on the edge information and depth information.
In an embodiment, matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information comprises:
respectively judging whether the pixel point corresponding to each pixel point in the parallax image in the semantic segmentation image belongs to the semantic of the barrier category;
if yes, the pixel points are reserved, and barrier parallax information is determined according to all the finally reserved pixel points.
In one embodiment, calculating the position of the obstacle according to the obstacle parallax information includes:
determining a contour of the obstacle in the initial image;
establishing a space rectangular coordinate system based on the initial image, and acquiring an x value and a y value of the contour central point;
and calculating the depth value of the contour central point according to the obstacle parallax information and using the depth value as a z value.
In an embodiment, the method further comprises: constructing a training set through an image sample and a depth label corresponding to the image sample to train a preset convolutional neural network;
and inputting the initial image to a depth estimation network after training is finished, and outputting a second depth value of the contour central point.
In an embodiment, after outputting the second depth value of the contour center point, the method further comprises:
calculating a difference value between the first depth value and the second depth value, and judging whether the difference value is smaller than a preset value;
if the first depth value is smaller than the first depth value, the first depth value is determined to be successfully verified, and the position of the obstacle is output according to the x value, the y value and the z value of the contour center point.
In one embodiment, calculating the size of the obstacle according to the obstacle disparity information comprises:
calculating at least one edge point coordinate of the obstacle in the space rectangular coordinate system;
and calculating the size of the obstacle according to the outline of the obstacle and the at least one edge point coordinate.
In a second aspect, an embodiment of the present application provides an obstacle detection apparatus, including:
the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring at least one initial image aiming at a current scene;
the extraction module is used for extracting parallax information and semantic segmentation information in the initial image;
a matching module, configured to match the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information;
and the calculating module is used for calculating the position and the size of the obstacle according to the obstacle parallax information.
In a third aspect, embodiments of the present application provide a lawn mowing robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the obstacle detection method described above when executing the program.
In a fourth aspect, embodiments of the present application provide a storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps of the obstacle detection method as described above.
The obstacle detection method provided by the embodiment of the application can acquire at least one initial image of a current scene, extract parallax information and semantic segmentation information in the initial image, match the semantic segmentation information with the parallax information to determine obstacle parallax information in the parallax information, and calculate the position and size of an obstacle according to the obstacle parallax information. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the accuracy is high.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of a scene of an obstacle detection method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an obstacle detection method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a route design provided by an embodiment of the present application;
FIG. 4 is a schematic diagram of semantic segmentation provided by an embodiment of the present application;
fig. 5 is a schematic flow chart of an obstacle detection method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present application;
fig. 7 is another schematic structural diagram of an obstacle detection device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element. In addition, the connection may be for either a fixing function or a circuit connection function.
It is to be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in an orientation or positional relationship indicated in the drawings to facilitate the description of the embodiments of the invention and to simplify the description, and are not intended to indicate or imply that the device or element so referred to must have a particular orientation, be constructed in a particular orientation, and be constructed in operation as a limitation of the invention.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present application, "a plurality" means two or more unless specifically defined otherwise.
The embodiment of the application provides an obstacle detection method and device, a mowing robot and a storage medium.
The obstacle detection device may be integrated in a Micro Controller Unit (MCU) of the mowing robot, or may be integrated in an intelligent terminal or a server, the MCU is also called a Single Chip Microcomputer (Single Chip Microcomputer) or a Single Chip Microcomputer (Single Chip Microcomputer), and is configured to appropriately reduce the frequency and specification of a Central Processing Unit (CPU), and form a Chip-level computer by using peripheral interfaces such as a memory, a counter (Timer), a USB, an analog-to-digital conversion/digital-to-analog conversion, a UART, a PLC, and a DMA, so as to perform different combination controls for different application occasions. The robot that mows can walk voluntarily, and the collision prevention is automatic within the scope returns to charge, possesses safety inspection and battery power and detects, possesses certain climbing ability, especially is fit for places such as family courtyard, public greenery patches and carries out lawn mowing maintenance, and its characteristics are: automatic mowing, cleaning grass scraps, automatic rain sheltering, automatic charging, automatic obstacle sheltering, small and exquisite appearance, electronic virtual fence, network control and the like.
The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through a wired or wireless communication manner, the server may be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, and a big data and artificial intelligence platform, which is not limited herein.
For example, referring to fig. 1, the present application provides a mowing system, which includes a mowing robot 10, a server 20 and a user device 30, which are communicatively connected to each other. The user can control the robot lawnmower 10 to perform obstacle detection while traveling through the user device 30. Specifically, in the embodiment of the present application, the robot lawnmower 10 may incorporate a vision sensor, and detect an obstacle by using the vision sensor. In other embodiments, the mowing robot 10 may further integrate other sensors, for example, a rain sensor may be disposed on the top of the mowing robot 10 to detect whether it is raining currently, a lateral ultrasonic sensor and an infrared sensor may be disposed on the side of the mowing robot 10, and a collision sensor may be disposed around the mowing robot 10, so that the obstacle may be detected comprehensively by various sensors. When an obstacle is detected, the robot lawnmower 10 can detour accordingly.
During the operation, the user may control and adjust the moving path, the moving speed, the mowing range, and the like of the mowing robot 10 in real time through the user device 30. After the operation is completed, the data corresponding to the mowing operation can be synchronized to the server 20, so that the user can conveniently check the data.
For example, the robot lawnmower 10 starts work in response to a user instruction, which may be generated and transmitted by the user device 30, and during the work, the robot lawnmower 10 detects an obstacle in the current path, and generates an obstacle avoidance path according to the detection result to perform work so as to avoid the obstacle.
The following are detailed below. It should be noted that the description sequence of the following embodiments is not intended to limit the priority sequence of the embodiments.
An obstacle detection method comprising: the method comprises the steps of obtaining at least one initial image aiming at a current scene, extracting parallax information and semantic segmentation information in the initial image, matching the semantic segmentation information with the parallax information to determine obstacle parallax information in the parallax information, and calculating the position and the size of an obstacle according to the obstacle parallax information.
Referring to fig. 2, fig. 2 is a schematic flow chart of an obstacle detection method according to an embodiment of the present application. The specific flow of the obstacle detection method may be as follows:
101. at least one initial image for a current scene is acquired.
In one embodiment, the mowing robot can mow a work area with a boundary defined in advance and plan a corresponding action path according to the work area. As shown in fig. 3, after the mowing robot starts to operate, a movement path is generated in all the current area to determine that the movement path can cover all the current area, for example, a turning point is set according to the boundary of the current area, an arcuate movement path is generated according to the turning point to serve as the movement path, and mowing operation is performed according to the movement path.
In an embodiment, during the process of moving the mowing robot according to the initial action path to mow, the mowing robot can also detect obstacles in the current path and detour the obstacles. The detection may be specifically performed by a visual sensor integrated on the mowing robot, for example, a current scene is photographed by a plurality of cameras, so as to obtain at least one initial image. In other embodiments, other types of sensors, such as a collision sensor, a depth sensor, an ultrasonic sensor, an infrared sensor, and the like, may be integrated into the lawn mowing robot.
Specifically, after a vision sensor arranged on the mowing robot is turned on, an initial image can be acquired in real time or at a fixed working frequency at preset intervals, so that at least one initial image for a current scene is continuously acquired.
102. And extracting parallax information and semantic segmentation information in the initial image.
In an embodiment, the vision sensor that sets up on the robot that mows can be binocular camera, including left camera and right camera, shoots to current scene simultaneously through binocular camera and can obtain two images to current scene based on different visual angles, and then can calculate the parallax information that obtains in the image. The disparity information is the pixel coordinate difference of points in the same world coordinate system in different images, and is usually represented in the form of an image, that is, a disparity map.
Further, in the embodiment of the present application, semantic segmentation may be performed on the initial image to classify each pixel point in the initial image, that is, the initial image is divided into a plurality of blocks according to the difference of the categories, as shown in fig. 4. Specifically, when performing semantic segmentation of an image, the semantic segmentation may be performed by using a Normalized-cut (Normalized cut), a structured random forest, an SVM (Support Vector Machine), and the like. In this embodiment, semantic segmentation may also be performed by means of deep learning.
Specifically, the semantic segmentation process generally includes three steps, i.e., training, verifying, and testing, where training may read training samples and labels in the data set into the convolutional neural network according to the size of batch _ size (the number of parameters used for training). The training pictures and the labels are preprocessed according to actual needs, such as cutting and data enhancement, which is beneficial to training of deep networks, the convergence process is accelerated, meanwhile, the overfitting problem is avoided, and the generalization capability of the model is enhanced. And after training an epoch, reading the verification sample and the label in the data set into the convolutional neural network, and loading the training weight. And verifying according to the compiled semantic segmentation indexes to obtain index scores in the current training process, and storing corresponding weights. A method of training once and verifying once is commonly used to better supervise model performance. After all training is finished, the test samples and the labels in the data set are read into the convolutional neural network, and the stored best weight values are loaded into the model for testing. The test results are divided into two types, one is to measure the network performance according to the common index score, and the other is to store the prediction result of the network in the form of pictures, so that the accuracy degree of the segmentation is intuitively felt.
103. And matching the semantic segmentation information with the parallax information to determine obstacle parallax information in the parallax information.
In an embodiment, after obtaining the disparity information and the semantic segmentation information corresponding to the initial image, an obstacle in the initial image and disparity information corresponding to the obstacle may be further determined according to a semantic segmentation result. Specifically, since the semantic segmentation information includes the category information of each object in the initial image, it is possible to determine that those objects are obstacles based on the category information, and then further determine the parallax information of the obstacles by combining the parallax information.
Specifically, each pixel point in the disparity map can be matched with the corresponding semantic segmentation map to determine whether the pixel point belongs to the category of the obstacle, if so, the pixel point is reserved, if not, the pixel point can be removed, finally, an image formed by all the reserved pixel points is an image of the obstacle, and disparity information corresponding to the obstacle is acquired in the disparity map. That is, the step of matching the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information may include: and respectively judging whether the corresponding pixel point of each pixel point in the parallax image in the semantic segmentation image belongs to the semantic of the barrier category, if so, retaining the pixel point, and determining barrier parallax information according to all the finally retained pixel points.
104. And calculating the position and the size of the obstacle according to the parallax information of the obstacle.
In one embodiment, after obtaining the parallax information of the obstacle, the position and size information of the obstacle can be further calculated, and the mowing robot designs an obstacle avoiding route for the obstacle, so that mowing operation is completed. Specifically, the three-dimensional coordinates of each vertex of the obstacle in the current spatial coordinate system can be calculated through the parallax information, and then the volume of the obstacle can be calculated.
For example, the MCU in the mowing robot may generate an obstacle avoidance route according to the position and size of the obstacle, and then control the mowing robot to perform mowing operation based on the obstacle avoidance route; for another example, the server or the user device may control the mowing robot to travel according to the obstacle avoidance route, so as to perform mowing operation. Namely, the mowing robot executes mowing operation according to the obstacle avoidance route, and after the mowing robot walks the obstacle avoidance route, the mowing robot is controlled to continue to walk the initial action path and continue obstacle detection.
As can be seen from the above, the obstacle detection method provided in the embodiment of the present application may acquire at least one initial image of a current scene, extract disparity information and semantic segmentation information in the initial image, match the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information, and calculate a position and a size of an obstacle according to the obstacle disparity information. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the method and the device have high accuracy.
Referring to fig. 5, fig. 5 is another schematic flow chart of the obstacle detection method according to the embodiment of the present application. The specific flow of the obstacle detection method may be as follows:
201. at least one initial image for a current scene is acquired.
In an embodiment, the visual sensor that sets up on the robot that mows can be binocular camera, including left camera and right camera, can obtain two images to current scene based on different visual angles through binocular camera shooting to current scene simultaneously.
202. And extracting parallax information and semantic segmentation information in the initial image.
In an embodiment, a coordinate system may be established in the current scene, and a coordinate difference between the two images captured by the binocular camera for the same point in the coordinate system is parallax information, and the parallax information of the initial image may be represented by a parallax map.
Further, edge information of the object in the initial image and depth information of different objects can be obtained, so that different types of objects in the image are classified, and finally semantic segmentation information is obtained. Specifically, the edge information may be extracted from the initial image by a Canny edge detection algorithm, and the depth information may be extracted from a feature extraction layer of a VGG (visual geometry group, computer vision group) network, where the VGG network may include 16 layers, each layer is provided with a 3 × 3 convolution kernel, and the number of channels of 4 layers of convolution computation layers is 64, 128, 256, and 512, respectively. After the edge information and the depth information of the initial image are obtained, feature fusion can be performed to obtain a semantic segmentation map. That is, the step of extracting semantic segmentation information in the initial image may include: extracting edge information and depth information in the initial image, and determining semantic segmentation information in the initial image based on the edge information and the depth information.
203. And training the image segmentation model through the parallax information sample and the label information corresponding to the sample.
In an embodiment, the image segmentation model may be a U-Net model, and it should be noted that the U-Net model does not directly pool and directly up-sample the U-Net model to an output image having a size consistent with that of an original image, but maps a high-dimensional feature to a low-dimensional feature again by deconvolution, and in order to enhance the segmentation accuracy in the mapping process, an image having the same dimension in a shrinkage network in the same dimension is fused, and since the dimension becomes 2 times of the original dimension in the fusing process, the U-Net model needs to be convolved again, so that the processed dimension is the same as the dimension before the fusion operation, so that the U-Net model can be secondarily fused with the image in the same dimension after the deconvolution again, until the U-Net model can finally output the image having the same dimension as that of the original image.
When the U-net model is trained in advance, a disparity map obtained by an upstream algorithm can be made into a training data set, a supervision label is a self-built corresponding foreground and background image, then the U-net model is trained with supervision by adopting the made data set and a real label, and foreground and background information is output.
204. And segmenting parallax information in the initial image into foreground parallax information and background parallax information through the trained image segmentation model.
In actual use, considering that some objects in the background of the scene are also recognized as obstacles, and the obstacles do not actually interfere with the lawn mower person, the current scene needs to be distinguished into a foreground scene and a background scene. In an embodiment, after the U-net model training is completed, foreground and background separation may be further performed on the disparity map, so as to obtain disparity information of the foreground and disparity information of the background.
205. And matching the semantic segmentation information with the foreground parallax information, judging whether an obstacle exists in the foreground, if so, executing step 206, and if not, executing step 207.
206. And acquiring obstacle parallax information in the foreground parallax information.
And 207, matching the semantic segmentation information with the background parallax information to acquire barrier parallax information in the background parallax information.
In an embodiment, after obtaining the disparity information and the semantic segmentation information corresponding to the initial image, an obstacle in the initial image and disparity information corresponding to the obstacle may be further determined according to a semantic segmentation result. In particular, the embodiment preferentially acquires parallax information of an obstacle in the foreground, and if no obstacle exists in the foreground, the parallax information of the obstacle in the background can be further acquired.
And 208, calculating the position and the size of the obstacle according to the obstacle parallax information.
In an embodiment, the position of the obstacle may be a center point of the obstacle contour, for example, a spatial rectangular coordinate system may be established based on the initial image, a pixel position of the center point in the image is an x and y value of the obstacle in a camera coordinate system, a z value of the x and y value may be represented by a depth value, and specifically, a disparity value corresponding to the point may be converted into the z value. Wherein, the x value, the y value and the z value respectively represent a horizontal axis coordinate, a vertical axis coordinate and a vertical axis coordinate in the space rectangular coordinate system.
In the embodiment of the present application, the depth value of the center point may be calculated based on the parallax information of the binocular image. Specifically, the depth value may be calculated by dividing the product of the focal length and the baseline by the parallax, where the focal length is the focal length of the initial image captured by the binocular camera, that is, the distance from the optical center of the lens to the imaging plane such as a film, a CCD or a CMOS, and the baseline is the distance between the optical centers of the two cameras of the binocular camera. The step of calculating the position of the obstacle from the obstacle disparity information may therefore comprise: determining the outline of the obstacle in the initial image, establishing a space rectangular coordinate system based on the initial image, acquiring an x value and a y value of the outline central point, and calculating the depth value of the outline central point according to the obstacle parallax information and using the depth value as a z value.
In another embodiment, the depth value of the center point may be calculated based on one of the monocular image and the binocular image. The depth estimation network may be a convolutional neural network, for example, by inputting the monocular image to a trained depth estimation network to output depth values, wherein a large training set is also required to be prepared in advance for training network parameters of the convolutional neural network. Each set of training sets may include a common color picture taken of an outdoor scene at an angle and a depth label corresponding to the color picture, where the data depth label may be made by a binocular depth estimation algorithm. Alternatively, the convolutional neural network may be trained using the SYNTHIA data set as a training set, and the data set collects data by simulating a car driving in real traffic, and photographs are taken at regular intervals from a fixed position and angle on the car. There will be many groups of nearly identical pictures in the data set. Redundant data can be eliminated through the vehicle speed, and therefore the calculation amount is reduced.
In the process of training the convolutional neural network, a loss function can be adopted to iteratively update the network parameters of the convolutional neural network, so as to obtain a trained deep estimation network. In one embodiment, when the model is trained, a training set is input into the network model, a predicted value is obtained through a forward channel, then a loss function is used for calculating a difference value between the predicted value and a true value, the value of the loss function is fed back and transmitted to the training network model, various parameters in the model are optimized, the purpose of reducing the loss value is achieved, the predicted value generated by the model is continuously close to the true value, and iterative updating is completed.
Optionally, when the depth value of the central point is calculated based on the disparity information of the binocular image, the depth value may be further checked. Specifically, in this embodiment, after the first depth value is calculated by using the disparity information of the binocular image, a second depth value may be output by using a monocular image of the binocular image through the depth estimation network, and then the first depth value is checked based on the second depth value, for example, a difference between the first depth value and the second depth value is compared to determine whether the difference is greater than a preset difference, so as to determine whether the depth value of the central point has an error in calculation. When the error is large, the initial image can be acquired again, and parallax information and depth value calculation can be carried out.
In an embodiment, the step of calculating the size of the obstacle according to the obstacle parallax information may include: and calculating at least one edge point coordinate of the obstacle in the rectangular space coordinate system, and calculating the size of the obstacle according to the outline of the obstacle and the at least one edge point coordinate.
As can be seen from the above, the obstacle detection method provided in this embodiment of the application may obtain at least one initial image for a current scene, extract disparity information and semantic segmentation information in the initial image, train an image segmentation model through a disparity information sample and tag information corresponding to the sample, segment the disparity information in the initial image into foreground disparity information and background disparity information through the trained image segmentation model, match the semantic segmentation information with the foreground disparity information, determine whether an obstacle exists in a foreground, if so, obtain obstacle disparity information in the foreground disparity information, if not, match the semantic segmentation information with the background disparity information to obtain obstacle disparity information in the background disparity information, and calculate a position and a size of the obstacle according to the obstacle disparity information. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the accuracy is high.
In order to better implement the obstacle detection method of the embodiment of the present application, the embodiment of the present application further provides an obstacle detection device based on the above. The terms are the same as those in the above-described obstacle detection method, and details of implementation may be referred to the description in the method examples.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present application, where the obstacle detection device may include:
an obtaining module 301, configured to obtain at least one initial image of a current scene;
an extracting module 302, configured to extract disparity information and semantic segmentation information in the initial image;
a matching module 303, configured to match the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information;
a calculating module 304, configured to calculate a position and a size of the obstacle according to the obstacle parallax information.
In an embodiment, referring to fig. 7, the extracting module 302 specifically includes:
an extraction submodule 3021, configured to extract edge information and depth information in the initial image;
a first determining sub-module 3022 configured to determine semantic segmentation information in the initial image based on the edge information and the depth information.
In one embodiment, the matching module 303 may include:
the determining submodule 3031 is configured to respectively determine whether a pixel corresponding to each pixel in the disparity map in the semantic segmentation map belongs to a semantic of an obstacle category;
and a second determining submodule 3032, configured to, when the determining submodule 3031 determines that the pixel points are valid, reserve the pixel points, and determine parallax information of the obstacle according to all the pixel points that are finally reserved.
As can be seen from the above, in the embodiment of the present application, the obtaining module 301 obtains at least one initial image of a current scene, the extracting module 302 extracts disparity information and semantic segmentation information in the initial image, the matching module 303 matches the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information, and the calculating module 304 calculates a position and a size of an obstacle according to the obstacle disparity information. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the method and the device have high accuracy.
In addition, an embodiment of the present application also provides a robot mower, as shown in fig. 8, which shows a schematic structural diagram of the robot mower according to the embodiment of the present application, and specifically:
the mowing robot can include components such as a control module 501, a travel mechanism 502, a cutting module 503, and a power supply 504. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 8 does not constitute a limitation of the electronic device and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:
the control module 501 is a control center of the robot mower, and the control module 501 may specifically include a Central Processing Unit (CPU), a memory, an input/output port, a system bus, a timer/counter, a digital-to-analog converter, an analog-to-digital converter, and other components, where the CPU executes various functions and processes data of the robot mower by running or executing software programs and/or modules stored in the memory and calling data stored in the memory; preferably, the CPU may integrate an application processor, which mainly handles an operating system, application programs, and the like, and a modem processor, which mainly handles wireless communication. It will be appreciated that the modem processor described above may not be integrated into the CPU.
The memory may be used to store software programs and modules, and the CPU executes various functional applications and data processing by operating the software programs and modules stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the CPU access to the memory.
The traveling mechanism 502 is electrically connected to the control module 501, and is configured to adjust a traveling speed and a traveling direction of the mowing robot in response to a control signal transmitted by the control module 501, so as to implement a self-moving function of the mowing robot.
The cutting module 503 is electrically connected to the control module 501, and is configured to adjust the height and the rotation speed of the cutter disc in response to a control signal transmitted by the control module, so as to achieve mowing operation.
The power supply 504 may be logically connected to the control module 501 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The power supply 504 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
Although not shown, the mowing robot may further include a communication module, a sensor module, a prompt module, and the like, which are not described in detail herein.
The communication module is used for receiving and sending signals in the process of receiving and sending information, and realizes the signal receiving and sending with the user equipment, the base station or the server by establishing communication connection with the user equipment, the base station or the server.
The sensor module is used for collecting internal environment information or external environment information, and feeding collected environment data back to the control module for decision making, so that the accurate positioning and intelligent obstacle avoidance functions of the mowing robot are realized. Optionally, the sensor may comprise: without limitation, ultrasonic sensors, infrared sensors, collision sensors, rain sensors, lidar sensors, inertial measurement units, wheel speed gauges, image sensors, position sensors, and other sensors.
The prompting module is used for prompting the working state of the current mowing robot of a user. In this scheme, the prompt module includes but is not limited to pilot lamp, bee calling organ etc.. For example, the mowing robot can prompt a user of the current power state, the working state of the motor, the working state of the sensor and the like through the indicator lamp. For another example, when the mowing robot is detected to be out of order or stolen, the warning prompt can be realized through the buzzer.
Specifically, in this embodiment, the processor in the control module 501 loads the executable file corresponding to the process of one or more application programs into the memory according to the following instructions, and the processor runs the application programs stored in the memory, thereby implementing various functions as follows:
the method comprises the steps of obtaining at least one initial image aiming at a current scene, extracting parallax information and semantic segmentation information in the initial image, matching the semantic segmentation information with the parallax information to determine parallax information of an obstacle in the parallax information, and calculating the position and the size of the obstacle according to the parallax information of the obstacle.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
According to the method and the device, at least one initial image of the current scene can be acquired, the parallax information and the semantic segmentation information in the initial image are extracted, the semantic segmentation information is matched with the parallax information to determine the parallax information of the obstacle in the parallax information, and the position and the size of the obstacle are calculated according to the parallax information of the obstacle. According to the method and the device, the current scene can be detected only by means of the visual sensor, the position and the size of the obstacle in the current scene are calculated according to the parallax information and the semantic segmentation information, other types of sensors are not needed, and the accuracy is high.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, the present application provides a storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the obstacle detection methods provided by the embodiments of the present application. For example, the instructions may perform the steps of:
the method comprises the steps of obtaining at least one initial image aiming at a current scene, extracting parallax information and semantic segmentation information in the initial image, matching the semantic segmentation information with the parallax information to determine parallax information of an obstacle in the parallax information, and calculating the position and the size of the obstacle according to the parallax information of the obstacle.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the storage medium may execute the steps in any obstacle detection method provided in the embodiment of the present application, beneficial effects that can be achieved by any obstacle detection method provided in the embodiment of the present application may be achieved, for details, see the foregoing embodiments, and are not described herein again.
The method, the device, the mowing robot and the storage medium for detecting the obstacle provided by the embodiment of the application are described in detail, and the principle and the implementation mode of the application are explained by applying a specific example, and the description of the embodiment is only used for helping to understand the method and the core idea of the application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (12)
1. An obstacle detection method, comprising:
acquiring at least one initial image for a current scene;
extracting parallax information and semantic segmentation information in the initial image;
matching the semantic segmentation information with the parallax information to determine obstacle parallax information in the parallax information;
and calculating the position and the size of the obstacle according to the obstacle parallax information.
2. The method of claim 1, wherein prior to matching the semantic segmentation information with the disparity information, the method further comprises:
training an image segmentation model through a parallax information sample and label information corresponding to the sample;
and segmenting the parallax information in the initial image into foreground parallax information and background parallax information through the trained image segmentation model.
3. The method of claim 2, wherein matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information comprises:
matching the semantic segmentation information with the foreground parallax information, and judging whether an obstacle exists in the foreground;
if so, acquiring obstacle parallax information in the foreground parallax information;
and if the semantic segmentation information does not exist, matching the semantic segmentation information with the background parallax information to acquire obstacle parallax information in the background parallax information.
4. The method of claim 1, wherein the extracting semantic segmentation information in the initial image comprises:
extracting edge information and depth information in the initial image;
semantic segmentation information in the initial image is determined based on the edge information and depth information.
5. The method of claim 1, wherein matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information comprises:
respectively judging whether the pixel point corresponding to each pixel point in the parallax image in the semantic segmentation image belongs to the semantic of the barrier category;
if yes, the pixel points are reserved, and barrier parallax information is determined according to all the finally reserved pixel points.
6. The method of claim 1, wherein said calculating the location of the obstacle from the obstacle disparity information comprises:
determining a contour of the obstacle in the initial image;
establishing a space rectangular coordinate system based on the initial image, and acquiring an x value and a y value of the contour central point;
and calculating a first depth value of the contour central point according to the obstacle parallax information and using the first depth value as a z value.
7. The method of claim 6, further comprising:
constructing a training set through an image sample and a depth label corresponding to the image sample to train a preset convolution neural network;
and inputting the initial image to a depth estimation network after training is finished, and outputting a second depth value of the contour central point.
8. The method of claim 7, wherein after outputting the second depth value for the contour center point, the method further comprises:
calculating a difference value between the first depth value and the second depth value, and judging whether the difference value is smaller than a preset value or not;
if the first depth value is smaller than the first depth value, the first depth value is determined to be successfully verified, and the position of the obstacle is output according to the x value, the y value and the z value of the contour center point.
9. The method of claim 6, wherein calculating the size of the obstacle from the obstacle disparity information comprises:
calculating at least one edge point coordinate of the obstacle in the rectangular spatial coordinate system;
and calculating the size of the obstacle according to the outline of the obstacle and the at least one edge point coordinate.
10. An obstacle detection device, comprising:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring at least one initial image aiming at a current scene;
the extraction module is used for extracting parallax information and semantic segmentation information in the initial image;
a matching module, configured to match the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information;
and the calculating module is used for calculating the position and the size of the obstacle according to the obstacle parallax information.
11. A robot lawnmower comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program performing the steps of the method of detecting an obstacle according to any one of claims 1 to 9.
12. A storage medium, characterized in that a computer program is stored thereon which, when being executed by a processor, carries out the steps of the obstacle detection method according to any one of claims 1-9.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210910652.3A CN115423865A (en) | 2022-07-29 | 2022-07-29 | Obstacle detection method, obstacle detection device, mowing robot, and storage medium |
| PCT/CN2023/109120 WO2024022337A1 (en) | 2022-07-29 | 2023-07-25 | Obstacle detection method and apparatus, mowing robot, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210910652.3A CN115423865A (en) | 2022-07-29 | 2022-07-29 | Obstacle detection method, obstacle detection device, mowing robot, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115423865A true CN115423865A (en) | 2022-12-02 |
Family
ID=84196495
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210910652.3A Pending CN115423865A (en) | 2022-07-29 | 2022-07-29 | Obstacle detection method, obstacle detection device, mowing robot, and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115423865A (en) |
| WO (1) | WO2024022337A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115617053A (en) * | 2022-12-19 | 2023-01-17 | 松灵机器人(深圳)有限公司 | Obstacle traversing method and device, mowing robot and storage medium |
| CN116704473A (en) * | 2023-05-24 | 2023-09-05 | 禾多科技（北京）有限公司 | Obstacle information detection method, obstacle information detection device, electronic device, and computer-readable medium |
| WO2024022337A1 (en) * | 2022-07-29 | 2024-02-01 | 松灵机器人(深圳)有限公司 | Obstacle detection method and apparatus, mowing robot, and storage medium |
| CN117608283A (en) * | 2023-11-08 | 2024-02-27 | 浙江孚宝智能科技有限公司 | Autonomous navigation method and system for robot |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107909010B (en) * | 2017-10-27 | 2022-03-18 | 北京中科慧眼科技有限公司 | Road obstacle detection method and device |
| CN108230392A (en) * | 2018-01-23 | 2018-06-29 | 北京易智能科技有限公司 | A kind of dysopia analyte detection false-alarm elimination method based on IMU |
| CN109634286B (en) * | 2019-01-21 | 2021-06-25 | 傲基科技股份有限公司 | Visual obstacle avoidance method for mowing robot, mowing robot and readable storage medium |
| CN110287964B (en) * | 2019-06-13 | 2021-08-03 | 浙江大华技术股份有限公司 | Stereo matching method and device |
| CN114019956A (en) * | 2021-10-14 | 2022-02-08 | 科沃斯机器人股份有限公司 | Method and system for determining region boundary, autonomous traveling equipment and mowing robot |
| CN115423865A (en) * | 2022-07-29 | 2022-12-02 | 松灵机器人(深圳)有限公司 | Obstacle detection method, obstacle detection device, mowing robot, and storage medium |
-
2022
- 2022-07-29 CN CN202210910652.3A patent/CN115423865A/en active Pending
-
2023
- 2023-07-25 WO PCT/CN2023/109120 patent/WO2024022337A1/en unknown
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024022337A1 (en) * | 2022-07-29 | 2024-02-01 | 松灵机器人(深圳)有限公司 | Obstacle detection method and apparatus, mowing robot, and storage medium |
| CN115617053A (en) * | 2022-12-19 | 2023-01-17 | 松灵机器人(深圳)有限公司 | Obstacle traversing method and device, mowing robot and storage medium |
| CN116704473A (en) * | 2023-05-24 | 2023-09-05 | 禾多科技（北京）有限公司 | Obstacle information detection method, obstacle information detection device, electronic device, and computer-readable medium |
| CN116704473B (en) * | 2023-05-24 | 2024-03-08 | 禾多科技（北京）有限公司 | Obstacle information detection method, obstacle information detection device, electronic device, and computer-readable medium |
| CN117608283A (en) * | 2023-11-08 | 2024-02-27 | 浙江孚宝智能科技有限公司 | Autonomous navigation method and system for robot |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024022337A1 (en) | 2024-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115423865A (en) | Obstacle detection method, obstacle detection device, mowing robot, and storage medium | |
| US10580162B2 (en) | Method for determining the pose of a camera and for recognizing an object of a real environment | |
| CN110147706A (en) | The recognition methods of barrier and device, storage medium, electronic device | |
| US20230072730A1 (en) | Target detection method and apparatus | |
| WO2019157455A1 (en) | Aerial vehicle smart landing | |
| CN110988912A (en) | Road target and distance detection method, system and device for automatic driving vehicle | |
| CN111213155A (en) | Image processing method, device, movable platform, unmanned aerial vehicle and storage medium | |
| CN111679695B (en) | Unmanned aerial vehicle cruising and tracking system and method based on deep learning technology | |
| CN110667474B (en) | General obstacle detection method and device and automatic driving system | |
| Ji et al. | Obstacle detection and recognition in farmland based on fusion point cloud data | |
| CN115016502A (en) | Intelligent obstacle avoidance method, mowing robot and storage medium | |
| US20230047094A1 (en) | Image processing method, network training method, and related device | |
| CN114454875A (en) | Urban road automatic parking method and system based on reinforcement learning | |
| CN113191297A (en) | Pavement identification method and device and electronic equipment | |
| WO2022047744A1 (en) | Road surface extraction method and device for map | |
| CN113537047A (en) | Obstacle detection method, obstacle detection device, vehicle and storage medium | |
| CN115053689A (en) | Intelligent obstacle avoidance method and device, mowing robot and storage medium | |
| CN116416291A (en) | Electronic fence automatic generation method, real-time detection method and device | |
| CN112506204B (en) | Robot obstacle meeting processing method, device, equipment and computer readable storage medium | |
| WO2024008016A1 (en) | Operation map construction method and apparatus, mowing robot, and storage medium | |
| CN117152719A (en) | Weeding obstacle detection method, weeding obstacle detection equipment, weeding obstacle detection storage medium and weeding obstacle detection device | |
| CN115268438A (en) | Intelligent obstacle avoidance method and device, mowing robot and storage medium | |
| WO2023155903A1 (en) | Systems and methods for generating road surface semantic segmentation map from sequence of point clouds | |
| CN115493579A (en) | Positioning correction method, positioning correction device, mowing robot and storage medium | |
| CN115718304A (en) | Target object detection method, target object detection device, vehicle and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: 518000 9/F, Building A3, Nanshan Zhiyuan, No. 1001, Xueyuan Avenue, Changyuan Community, Taoyuan Street, Nanshan District, Shenzhen, Guangdong Province Applicant after: Shenzhen Kuma Technology Co.,Ltd. Applicant after: Songling Robot (Chengdu) Co.,Ltd. Address before: 518000 1201, Tianlong mobile headquarters building, Tongfa South Road, Xili community, Xili street, Nanshan District, Shenzhen, Guangdong Province Applicant before: Songling robot (Shenzhen) Co.,Ltd. Country or region before: China Applicant before: Songling Robot (Chengdu) Co.,Ltd. |