TWI690816B - Map constructing apparatus and map constructing method - Google Patents

Abstract

A map constructing method adopted by a map constructing apparatus and includes following steps of: continually obtaining colored image and depth-image information by an image sensor and a depth sensor of the map constructing apparatus while the map constructing apparatus is moving; identifying the colored image for determining whether a dynamic object exists; labeling an object area corresponding to the location of the dynamic object if the dynamic object exists in the colored image; mapping the object area to a depth-image coordinates of the depth-image information, searching the corresponding position of the object area upon the depth-image information and filtering the object area from the depth-image information for generating an adjusted depth-image information; and, inputting the adjusted depth-image information to a map constructing algorithm for creating map data.

Description

Map building equipment and map building method

The invention relates to a map building device and a map building method, in particular to a map building device equipped with an RGB-D device, and a map building method realized by such map building device.

In the current market, many mobile robots (such as automatic trucks) are directly embedded with map building equipment. Through the use of map building equipment, this type of mobile robot can build a map of its own area and move and perform related operations according to the built map.

Generally speaking, mobile robots currently on the market are mostly based on 2D radar simultaneous positioning and mapping technology (Lidar Simultaneous Localization and Mapping) or 2D visual image simultaneous positioning and mapping technology (Visual Simultaneous Localization and Mapping). Build means for the main map. However, this type of map building technology has many limitations and shortcomings, so the maps built by current mobile robots usually have insufficient accuracy.

Specifically, if you use 2D radar synchronous positioning and map building technology, because 2D optical radar can only obtain one-dimensional distance information at a fixed height, it is impossible to grasp the real environmental conditions, so it is difficult To build accurate maps. If the 2D visual image synchronization positioning and map building technology is used, because the current visual assistance technology only stays in the functions of obstacle avoidance and ranging, and cannot be applied to the map building stage, it is difficult to exclude temporary objects from the map.

As mentioned above, if the above technology is used, the map can only be built without any dynamic objects in the space. If there are dynamic objects (such as people, carts, etc.) in the space where the map building device is located, or any objects that are not in the space, then these objects will be built into the map. In this way, the built map will not be accurate enough, which will affect the subsequent positioning and use of the mobile robot.

In view of this, when building a map at this stage, in order to obtain the accuracy of the map, it is usually necessary to clear the entire space before the mobile robot can move in the space and build the map. However, this approach will actually increase the difficulty of building maps and increase the cost of building.

The main purpose of the present invention is to provide a map building device and a map building method, which can improve the shortcomings of the traditional map building device and method inadequate perception of the surrounding environment, and strengthen the device in object image detection and environmental image recognition The ability to effectively filter the dynamic objects belonging to the foreground part and retain the environmental images belonging to the background part during the construction of the map.

In order to achieve the above object, the map building method of the present invention is mainly applied to a map building device having an image sensor and a depth sensor, and includes the following steps: passing when the map building device starts to move The image sensor and the depth sensor continuously obtain color image and depth image information; perform image recognition on the color image to determine whether there is a dynamic object; when the dynamic object exists, mark an object in the color image Area; map the object area to depth image information A depth image coordinate; searching and filtering the object area in the depth image information and generating corrected depth image information; inputting the corrected depth image information into a map building algorithm to generate a map data.

The technical effects that the present invention can achieve relative to the related art include at least: (1) setting RGB-D devices (including image sensors and depth sensors) on the map building equipment to enhance the environment of the map building equipment to the environment The cognitive ability of the image; (2) By the enhanced cognitive ability, the dynamic objects and the objects that are infrequently present in the space can be detected and marked in the color image; (3) By detecting and marking the dynamic objects , The background part (ie, static spatial information) and the foreground part (ie, the dynamic objects and objects that are not in the space infrequently) can be divided and saved in real time when building the map; (4) Under construction Actively modify or delete the information of dynamic objects (such as depth information) and retain the background when placing the map, thereby enhancing the spatial information in the map data; and (5) the local map construction equipment moves according to the built map, and When it is found that the environment has changed (for example, the installation location of the machine is changed, the local field is changed, etc.), it can actively update the built map.

1: Map building equipment

10: processor

11: Image sensor

12: Depth sensor

13: Storage unit

131: Map data

132: Object characteristic data

14: moving member

15: Action widget

16: Human Machine Interface

2: space

3: color image

31: Background

32: Dynamic objects

33: Mark box

4: Depth image information

41: Object depth information

5: Depth image information after correction

6: Map image

7: Map image after filtering

S10~S32: Construction steps

S40~S56: Update procedure

FIG. 1 is a first specific embodiment of a map construction schematic diagram of the present invention.

FIG. 2 is a first specific embodiment of a block diagram of a map building device of the present invention.

FIG. 3A is a first specific embodiment of the first part of the map building flowchart of the present invention.

FIG. 3B is a first specific embodiment of the second part of the map building flowchart of the present invention.

FIG. 4 is a first specific embodiment of a schematic diagram of a marker frame of the present invention.

FIG. 5A is a first specific embodiment of a schematic diagram of depth image information of the present invention.

FIG. 5B is a second embodiment of the schematic diagram of the depth image information of the present invention.

FIG. 6 is a first specific embodiment of a schematic diagram of corrected depth image information of the present invention.

7A is a first specific embodiment of a schematic diagram of a map image of the present invention.

7B is a second specific embodiment of a schematic diagram of a map image of the present invention.

8A is a first specific embodiment of a schematic diagram of a filtered map image of the present invention.

8B is a second specific embodiment of the schematic diagram of the filtered map image of the present invention.

9 is a first specific embodiment of the map update flowchart of the present invention.

The following is a detailed description of a preferred embodiment of the present invention with reference to the drawings.

Please refer first to FIG. 1, which is a first specific embodiment of a map construction schematic diagram of the present invention. As shown in FIG. 1, the present invention discloses a map building device 1. The map building device 1 may be installed on a mobile robot (such as an automatic truck, not shown in the figure) to assist the mobile robot in performing maps Build operations.

When the mobile robot equipped with (can be embedded or externally connected) the map building device 1 is placed in an unknown space 2, the map of the space 2 can be built by the map building device 1. Specifically, when the mobile robot moves in the space 2, the image of the surrounding environment can be detected by the map building device 1, thereby automatically creating a map of the space 2 in which it is located. After one or more movements, the map building device 1 can create map data depicting the planar spatial information (such as depth information) of the space 2 in which it is located. As a result, when mobile robots move in space 2 and perform tasks in the future, they can rely on Move according to the pre-built map data to eliminate the problems that may occur when moving (such as collision with the machine, entering the dead end, etc.).

Please refer to FIG. 2, which is a first specific embodiment of the block diagram of the map building device of the present invention. As shown in FIG. 2, the map building device 1 includes at least a processor 10, an image sensor 11, a depth sensor 12 and a storage unit 13 electrically connected to the processor 10. In this embodiment, the processor 10 is electrically connected to the image sensor 11, the depth sensor 12, and the storage unit 13 through an internal bus, and is mainly used to execute an embedded or externally input algorithm to Realize the various requirements and functions of the map building device 1 (to be described in detail later).

The map building device 1 of the present invention mainly implements the building and updating of the map via built-in or external RGB-D (ie RGB-Depth Map) devices. In an embodiment, the RGB-D device may be two separate devices, including a first device for capturing a color image (ie, RGB image) of the surrounding environment and a depth information for sensing the surrounding environment (Ie, Depth Info) the second device. In another embodiment, the RGB-D device may be an integrated image capture device that can simultaneously obtain color images and depth information of the surrounding environment. In this embodiment, the map building device 1 does not need to be provided Separate image sensor 11 and depth sensor 12. In yet another embodiment, the RGB-D device may be an image sensor 11 and a depth sensor 12 that are separately provided, wherein the image sensor 11 may be, for example, a color camera, and the depth sensor may be, for example, optical radar.

The above description is only a specific implementation example of the present invention and is not limited thereto. For ease of understanding, the image sensor 11 and the depth sensor 12 which are provided separately will be described as an example in the following description.

In this embodiment, the user can directly start the map building device 1, or start the mobile robot, and the mobile robot actively starts the connected map building device 1. When the map After the building device 1 is started, the processor 10 can drive the image sensor 11 to continuously obtain color images of the surrounding environment, and drive the depth sensor 12 to continuously obtain depth image information of the surrounding environment.

In an embodiment, the processor 10 can identify the dynamic objects (such as people, carts, or animals, etc.) that need to be excluded from the map from the surrounding environment through the color image, and through the depth image Information to build a two-dimensional map that depicts the surrounding environment. In another embodiment, the processor 10 can simultaneously refer to the color image and the depth image information to construct a three-dimensional map for depicting the surrounding environment. For ease of understanding, the following will take the construction of a two-dimensional map as an example to explain.

After the map building device 1 is started, the processor 10 can continue to obtain the color image from the image sensor 11. In this embodiment, the processor 10 may execute an embedded or externally input detection algorithm to perform image recognition on the received color image. Specifically, the processor 10 uses the obtained color image as an input parameter of the detection algorithm, and the detection algorithm returns a recognition result to the processor 10 after the execution is completed, the recognition result indicating whether the color image There are identifiable dynamic objects (ie, whether there are dynamic objects in the surrounding environment of the map building device 1).

One of the technical features of the present invention is that the map building device 1 can detect the foreground part (ie, dynamic objects) and the background part (ie, static space) during the construction of the map, and the foreground part and the background part Perform segmentation so that the foreground part will not appear in the completed map. If the processor 10 determines that there is at least one dynamic object in the color image from the recognition result of the detection algorithm, the processor 10 will further mark the dynamic object and generate a corresponding marking box (Bounding box). In this way, when the processor 10 executes the map building process by other algorithms, the algorithm can search for and identify the marker frame, and exclude relevant information of the marker frame from the built map Outside (to be described in detail later).

As described above, the processor 10 mainly executes the construction operation of the two-dimensional image based on the depth image information. Therefore, when the marker frame is generated, the processor 10 regards the marker frame as a region of interest ( Region of Interest (ROI), and further mapping the marker frame to the depth image coordinates used by the depth image information. In one embodiment, the processor 10 may perform a conversion operation by a conversion algorithm to correspond the position of the marker frame in the color image to the depth image coordinate of the depth image information. It is worth mentioning that the color image and depth image information referred to in this embodiment are preferably the color image and depth image information obtained at the same time point (that is, the two have the same acquisition time, or the time difference is less than one Error tolerance value).

After the above conversion operation is completed, the processor 10 may further search for the marker frame in the depth image information (for example, search for the corresponding coordinates of the marker frame), and filter the relevant information of the marker frame in the depth image information To generate corrected depth image information. In one embodiment, the processor 10 may execute a built-in or externally input filtering algorithm to filter the relevant information of the marked frame from the depth image information and generate the corrected depth image information. In the present invention, the corrected depth image information only describes the static spatial information of the background part, so it can be regarded as the planar spatial depth information under the space 2 in which it is located.

Finally, the processor 10 can also execute a built-in or externally input map building algorithm to generate map data 131 capable of depicting the space 2 according to the corrected depth image information. Furthermore, the processor 10 can store the generated map data 131 in the storage unit 13. When the mobile robot officially starts and executes the task, it can move according to the pre-built map data 131.

As shown in FIG. 2, in a specific embodiment of the present invention, the map building device 1 can be integrated with the mobile robot into a single entity, and the processor 10 can be electrically connected to various components of the mobile robot at the same time , Such as moving member 14, operating member 15, man-machine interface 16, etc., not limited set. In an embodiment, the moving member 14 may be, for example, a gear, a tire, a transmission shaft, a conveyor belt, and the like that can drive the map-building device 1 to move, and the moving member 15 may be, for example, a motor, a robot arm, or the like to assist A component of the map building device 1 for performing robot-related tasks. The human-machine interface 16 may be, for example, a screen, a keyboard, a mouse, a touch screen, or a voice control unit, etc., which can issue commands to the map building device 1 and display the built Components of the map data 131. However, the above descriptions are only some specific examples of the present invention, and should not be used to limit the actual patent scope of the present invention.

In an embodiment of the present invention, the processor 10 may record one or more detection algorithms (not shown), such as a histogram of oriented gradient (HOG) feature extraction algorithm, support vector Support Vector Machine (SVM) algorithm, Convolutional Neural Network (CNN) algorithm, YOLO (You Only Look Once) algorithm, SSD (Single Shot multibox Detector) algorithm or other neural networks One or a combination of training object detection algorithms and the like is not limited here.

Before the map building device 1 is used, the processor 10 can perform machine learning by one or a combination of the above detection algorithms. Specifically, the manufacturer of the map building device 1 can import a large amount of identification materials (such as photos or videos recorded with dynamic objects) into the detection algorithm to train the detection algorithm. The detection algorithm can obtain object feature data 132 of various object types (such as people, carts, animals, etc.) through machine learning and store them in the storage unit 13.

During the construction of the map, the processor 10 may continuously compare the color image obtained by the image sensor 11 with the plurality of object characteristic data 132 stored in the storage unit 13, when the color image has a matching or When there are multiple images, the processor 10 treats these images as the dynamic objects and marks the kinetic energy objects. In this way, the processor 10 can filter the dynamic objects in the subsequent build process.

Please refer to FIG. 3A and FIG. 3B for the first specific embodiment of the first part of the map building flowchart and the second part of the map building flowchart of the present invention. First, the user activates the map building device 1 when building a map (step S10). Specifically, the user can directly start the map building device 1 or start the mobile robot equipped with the map building device 1, and the mobile robot actively starts the map building device 1.

The map building device 1 can start moving after being activated (step S12), for example, by the moving member 14, or the mobile building robot drives the map building device 1 to move. When moving, the map building device 1 detects the surrounding environment through the image sensor 11 and the depth sensor 12 (step S14). In an embodiment, the image sensor 11 may be, for example, a color camera. The map building device 1 continuously detects the surrounding environment through the image sensor 11 and obtains color images; the depth sensor 12 is an optical radar, map The installation device 1 continuously detects the surrounding environment through the depth sensor 12 and obtains depth image information (step S16).

Next, the map building device 1 executes a detection algorithm through the processor 10 to perform image recognition on the color image (step S18), and determines whether there is a dynamic object in the color image (step S20).

As mentioned above, the map building device 1 can use a histogram feature extraction algorithm such as a direction gradient, a support vector machine algorithm, a convolutional neural network algorithm, a YOLO algorithm, an SSD algorithm or other neural network training One or a combination of object detection algorithms, etc., is used for machine learning to generate and store a plurality of object feature data 132 belonging to different object types. In the above step S18, the processor 10 compares the obtained color image with the object characteristic data 132, and regards the one or more images matching the comparison as the dynamic object.

If the processor 10 determines in step S20 that no dynamic object exists, it can directly construct the map data 131 according to the currently acquired depth image information, that is, directly execute step S28 according to the current depth image information.

If the processor 10 determines that there is a dynamic object in step S20, the dynamic object is further marked, and a corresponding mark frame is generated (step S22). In an embodiment, the processor 10 treats the marked frame as a region of interest (ROI). In another embodiment, the processor 10 further converts the region of interest into an easily recognizable mask (MASK). The processor 10 can store the coordinates, pixel information, etc. of the position of the marker frame (ie, the region of interest or mask), thereby recording the position of the marker frame as the foreground part in the color image, The location is recorded as the background part of the color image. In this way, the processor 10 can effectively segment the foreground part and the background part in the color image, and filter out the foreground part in the generated map data 131 (to be described in detail later).

After step S22, the processor 10 further maps the marker frame to the depth image coordinates of the depth image information (step S24), whereby when the processor 10 constructs the map data 131 based on the depth image information When the action is set, the depth information of the corresponding position (that is, the region of interest or mask) of the marker frame can be filtered in the map data 131 by printing.

Please also refer to FIG. 4, which is a first specific embodiment of the schematic diagram of the marker frame of the present invention. FIG. 4 reveals a color image 3 captured by the image sensor 11. As shown in FIG. 4, the color image 3 mainly records the background 31 of the surrounding environment of the map building device 1, when the processor 10 When the detection algorithm recognizes a dynamic object 32 in the color image 3 (taking human as an example in FIG. 4 ), the processor 10 generates a marking frame 33 that can cover all or part of the dynamic object 32.

In one embodiment, after recognizing the dynamic object 32, the processor 10 obtains the X-axis start coordinate position (X _start ), X-axis end coordinate position (X _end ) of the dynamic object 32 in the color image 3, The Y-axis starting coordinate position (Y _start ) and the Y-axis ending coordinate position (Y _end ), and according to these coordinate positions (X _start , Y _start , X _end , Y _end ), the marker frame 33 (in other words, borrow These coordinate positions constitute a region of interest (ROI).

As mentioned above, when the marker frame 33 is mapped to the depth image coordinates of the depth image information (ie, when step S24 of FIG. 3B is executed), the processor 10 corresponds the position of the marker frame 33 to the depth image On the coordinates, and set the pixel information of all pixels in the mark frame 33 to a pixel extremum, and generate a custom pixel information (ie, convert the region of interest into a mask). In this way, the algorithm used by the processor 10 can identify the corresponding position of the marker frame 33 (ie, the mask) in the depth image information. In one embodiment, the pixel extremum can be set to zero. In another embodiment, the pixel extremum can be set to 255. In yet another embodiment, the pixel extremum can be set to a specific value that the algorithm can directly identify, but it is not limited.

For example, the processor 10 can generate a mask according to the code shown below: If Object is exist

For X=X _start to X _end

For Y=Y _start to Y _end

Image[X][Y]=Mask Value

End for X,Y

End if

However, the above is only one specific implementation example of the present invention, but should not be limited to this.

Return to Figure 3B. After step S24, the processor 10 can search the corresponding position of the marker frame 33 in the depth image information (for example, search for the position where the pixel extremum is located), and filter the marker in the depth image information The relevant information in box 33 and produce a corrected depth image Image information (step S26). In this embodiment, the corrected depth image information will not include the depth information of the location of the dynamic object 32.

Please refer to FIG. 5A and FIG. 5B at the same time, which are the first specific embodiment and the second specific embodiment of the schematic diagram of the depth image information of the present invention, respectively.

As shown in FIGS. 5A and 5B, if the processor 10 does not filter the depth information of the corresponding position of the dynamic object 32 from the depth image information, when the dynamic object 32 (such as a person) appears in the color image 3, the depth The corresponding object depth information 41 will appear at the corresponding position of the image information. If the processor 10 directly constructs the map data 131 according to the depth image information, the object depth information 41 will be regarded as a part of the planar space, which will cause the accuracy of the map to decrease.

Please also refer to FIG. 6, which is a first embodiment of the schematic diagram of the corrected depth image information of the present invention.

According to the technical solution of the present invention, the processor 10 may mark the dynamic object 32 after identifying the dynamic object 32 and generate a marking frame 33, correspond the position of the marking frame 33 to the depth image coordinates, and mark the The pixel information of is set to the pixel extremum. In this way, the processor 10 can search for the corresponding positions of the marker frame 33 in the depth image information according to the pixel extreme values, and filter the depth information at these positions in the depth image information.

As shown in FIG. 6, although the dynamic object 32 can be seen in the color image 3, the depth information at the corresponding position of the dynamic object 32 has been filtered in the depth image information (ie, the corrected depth image information 5 is generated ), that is to say, the above-mentioned object depth information 41 will not appear in the corrected depth image information 5. If the processor 10 builds the map data 131 according to the corrected depth image information 5, it will effectively exclude dynamic objects, thereby improving the accuracy of the map.

In one embodiment, when determining that the dynamic object 32 exists in the color image 3, the processor 10 obtains the scan range of the corresponding depth image information through a command (such as Get Depth Raw Data), and then obtains the dynamic object 32 (or mark Box 33) The object boundary start point (DX _start ) and object boundary end point (DX _end ) on the depth image information, and the depth information of the range formed by the object boundary start point and the object boundary end point is set to 0, thereby generating Depth image information after correction 5. Specifically, the processor 10 searches the depth image information for the range in which the custom pixel information is located, and sets the depth information in this range to 0 to generate the corrected depth image information 5.

Specifically, the above object boundary start point and object boundary end point are the results obtained by the processor 10 mapping the position of the marker frame 33 to the depth image coordinates, and the processor 10 may search for the pixel extremum (That is, the custom pixel information, or called a mask) to confirm the range formed by the start point of the object boundary and the end point of the object boundary.

In an embodiment, the processor 10 can search and filter the object depth information according to the following code: If Object is exist

Function Get Depth Daw Data input DY _layer return Range _environment

For Range=0 to Rnage _environment

If Range _environment subtraction DX _start >=Range

AND Range _environment subtract DX _end <=Range

Range _environment [Range]=Filter Value

End for Range

End if

In the above code, the DY _layer is the scanning range of the depth image information, and the Filter Value can be set to 0 (that is, a value that makes the object depth information 41 disappear in the depth image information), but it is not limited. In other embodiments, the processor 10 may also set the Filter Value to a preset special value. When the algorithm reads this special value, it will automatically filter the depth information at this location from the map data 131.

Return to Figure 3B. After step S26, the processor 10 can execute the map building algorithm based on the corrected depth image information 5 (step S28) to generate map data 131 and store the map data 131 in the storage unit 13 (step S30). In one embodiment, the processor 10 is a portable network graphics (Portable Network Graphics, png), a bitmap (BitMap, bmp), or a portable grayscale map (Portable Gray Map, pgm), etc. The file format stores the built map data 131 in the storage unit 13, but it is not limited.

Specifically, the processor 10 uses the corrected depth image information 5 as an input parameter of the map building algorithm, by which the processor 10 can create and correct the depth image information after executing the map building algorithm The map data 131 corresponding to the content of 5.

In an embodiment, the map building algorithm of the present invention may be a Visual Simultaneous Localization and Mapping (SLAM) algorithm, and the map building algorithm may use triangulation, Karl Kalman Filter, Particle Filter, Monte Carlo Localization (MCL), Hybrid Monte Carlo Localization (Mixture MCL) or Grid-based Markov Localization (Grid -Based on one or a combination of technologies such as Based Markov).

Please refer to FIG. 7A and FIG. 7B at the same time, which are the first specific embodiment and the second specific embodiment of the schematic diagram of the map image of the present invention, respectively. If the processor 10 does not perform searching and filtering of dynamic objects Process (ie, the corrected depth image information 5 is not generated), as shown in FIGS. 7A and 7B, when the processor 10 recognizes the dynamic object 32 in the color image 3, the map building algorithm is based on the depth image In the map image 6 generated by the information, the corresponding object depth information 41 is displayed at the position of the dynamic object 32. In this way, when the local map construction device 1 or the mobile robot moves according to this map image 6 (ie, the map data 131), it will avoid the position indicated by the object depth information 41 according to the instruction, which may cause inconvenient.

Please refer to FIG. 8A and FIG. 8B at the same time, which are the first specific embodiment and the second specific embodiment of the schematic diagram of the filtered map image of the present invention, respectively. If the processor 10 adopts the technical solution of the present invention, the object depth information 41 at the corresponding position of the dynamic object 32 is filtered in the depth image information and the corrected depth image information 5 is generated, as shown in FIGS. 8A and 8B, when When the processor 10 recognizes the dynamic object 32 in the color image 3, since the map building algorithm is based on the filtered depth map information 5 after the correction, the filtered map image 7 will not be displayed in the filtered map image 7.述 Object depth information 41. When the map building device 1 or the mobile robot moves according to the filtered map image 7 (ie, the map data 131), it will not be affected by the dynamic object 32.

Return to Figure 3B. After step S30, the processor 10 further determines whether the map building procedure is completed (step S32), and repeats steps S12 to S30 before the building procedure is completed. In this way, the map building device 1 can continue to move, and the map data 131 of the surrounding environment can be continuously created based on the detected complex color image and depth image information for future use when moving.

It is worth mentioning that, in addition to the above-mentioned dynamic objects, the position of static objects in the environment may also be changed (for example, the placement of the machine is changed, the decoration of the space is changed, etc.). Therefore, one of the technical features of the present invention is to make the map building device 1 (or the mobile type equipped with the map building device 1 (Robot) During normal use, the map data 131 can still be updated through the above technical solution, thereby overcoming the inconvenience that the conventional robot can only move according to a fixed map.

Referring to FIG. 9, it is a first specific embodiment of the map update flowchart of the present invention. FIG. 9 discloses a method for updating a map, which is applied to a map-building device 1 (for example, having a moving member 14 and an action member 15 shown in FIG. 2) with mobile capabilities, or is equipped with the map-building device 1 Mobile robot. For ease of understanding, the mobile robot equipped with the map building device 1 will be exemplified in the description below, but not limited to this.

The mobile robot stores map data 131 that has been built in advance. In this embodiment, the user can start the mobile robot first (step S40), and the mobile robot automatically loads the map data 131 after starting (step S42). In an embodiment, the mobile robot can load the map data 131 in the storage unit 13 shown in FIG. 2. In another embodiment, the mobile robot loads pre-built map data 131 from a built-in memory, an external hard disk, or a wirelessly connected database, which is not limited.

After the mobile robot loads the map data 131, it can start to move in space according to the instructions of the map data 131 (step S44). In this embodiment, the map data 131 records the planar spatial information of the space where the mobile robot is located.

During the movement, the mobile robot determines whether it is necessary to update the current map data 131 through the map building device 1 (step S46). Specifically, the map building device 1 can continuously obtain color images through the image sensor 11 and continuously obtain depth image information through the depth sensor 12 during the movement of the mobile robot, and based on the color image and the depth image information Content to determine whether the map data 131 needs to be updated.

In one embodiment, the map building device 1 may determine that the map data 131 needs to be updated when a dynamic object is identified in the color image. In another embodiment, the map building device 1 may determine that the map data 131 needs to be updated when the obtained depth image information does not match the content of the map data 131. In yet another embodiment, the map building device 1 may not execute the determination procedure of step S46, and automatically and continuously update the map data 131 during the movement of the mobile robot. However, the above descriptions are only specific implementation examples of the present invention and are not intended to limit the patent scope of the present invention.

If the map building device 1 needs to execute the determination procedure of step S46 and determines that the map data 131 does not need to be updated, the map building device 1 does not perform any action.

If the map building device 1 determines that the map data 131 needs to be updated in step S46, the map building device 1 executes the detection algorithm according to the technical means of the present invention to perform image recognition on the color image. The dynamic object is marked in the middle and a corresponding marked frame is generated (step S48). Next, the map building device 1 corresponds the marker frame to the depth image coordinates of the depth image information (step S50), and sets the pixel information of all pixels in the marker frame 33 to a pixel extremum to generate a custom pixel News.

After step S50, the map building device 1 searches the depth image information for the custom pixel information (ie, the marker frame), and filters the depth information at the corresponding position of the marker frame from the depth image information to generate a correction Rear depth image information (step S52). Furthermore, the map building device 1 executes the map building algorithm based on the corrected depth image information, thereby updating the currently used map data 131 (step S54).

It is worth mentioning that, in step S54, the map building algorithm can sequentially generate one or more partial map images according to the corrected depth image information, and update the map data 131 according to the one or more partial map images . In this way, the map building device 1 can be modified according to actual needs A part of the map data 131 does not need to update the entire map data 131, which makes the map update program of the present invention more flexible.

After step 54, the mobile robot determines whether its actuation procedure is completed (step S56), for example, whether the task assigned by the user via the human-machine interface 16 is completed, whether the mobile robot is turned off, whether the map-building device 1 is turned off, or moved Whether the robot will stop moving, etc. In addition, the mobile robot returns to step S44 before the operation procedure is completed, so as to continue the movement and the updating operation of the map data 131.

Through the technical solution of the present invention, it is possible to enhance the cognitive ability of the map building device 1 in the surrounding environment, detect objects that are not present frequently in the space, and filter the depth information of the objects in the built map, by This improves the accuracy of the built map. Furthermore, by continuing to execute the technical solution of the present invention after the map has been built, the map building device 1 can actively update the built map when it discovers that the surrounding environment has changed, thereby maintaining the map and the actual environment. consistency.

The above is only a preferred specific example of the present invention, and the patent scope of the present invention is not limited by this, so all equivalent changes in applying the content of the present invention are included in the scope of the present invention in the same way. Bright.

S20~S32: Construction steps

Claims (10)

A map building method is applied to a map building device, and the above map building method includes the following steps: a) After the map building device is started, a color image of the surrounding environment is continuously obtained through an image sensor, And continuously obtain a depth image information of the surrounding environment through a depth sensor; b) a processor executes a detection algorithm to perform image recognition on the color image and determine whether a dynamic object exists in the color image, Among them, the detection algorithm is a Histogram of Oriented Gradient (HOG) feature extraction algorithm, a Support Vector Machine (SVM) algorithm, and a Convolutional Neural Network (CNN) algorithm. Method, YOLO (You Only Look Once) algorithm, SSD (Single Shot multibox Detector) algorithm or other object detection algorithms using neural network training, one or a combination of these; c) when the dynamic object exists Dynamic objects are marked and a marked frame is generated; d) The marked frame is mapped to a depth image coordinate of the depth image information; e) The marked image frame is searched and filtered in the depth image information to generate a corrected depth image information , Where the corrected depth information filters out the depth information at the corresponding position of the marker frame; f) the processor executes a map building algorithm based on the corrected depth image information; and g) builds by the map The algorithm generates a map data and stores it in a storage unit. The map building method according to claim 1, wherein the image sensor is a color camera (RGB camera), and the depth sensor is an optical radar. The map building method according to claim 1, wherein the storage unit stores a plurality of object feature data corresponding to different object categories, and the step b) is the detection algorithm to use the color image and the plurality of object feature data The comparison is performed, and the image matching the comparison is regarded as the dynamic object. The map building method according to claim 3, wherein the step c) is to obtain an X-axis starting coordinate position (X _start ), an X-axis ending coordinate position (X _end ) of the dynamic object in the color image, A Y-axis starting coordinate position (Y _start ) and a Y-axis ending coordinate position (Y _end ), and according to these coordinate positions (X _start , Y _start , X _end , Y _end ) to create the marker frame, the step d) The coordinate position of the mark frame is mapped to the depth image coordinate, and one pixel information of all pixels in the mark frame is set as a pixel extremum to generate a custom pixel information. The map building method according to claim 4, wherein the pixel extreme value is 0 or 255. The map building method according to claim 4, wherein the step e) searches the depth image information for a range in which the custom pixel information is located, and sets a depth information in the range to 0 to generate the Corrected depth image information. The map building method according to claim 1, wherein the map building algorithm is a visual image synchronous positioning and map construction (Visual Simultaneous Localization and Mapping, SLAM) algorithm, and the triangulation method and the Kalman filter (Kalman Filter), Particle Filter (Particle Filter), Monte Carlo Localization (MCL), Mixed Monte Carlo Localization (Mixture MCL) or Grid-Based Markov )to fulfill. The map construction method according to claim 1, wherein the map data is a portable network graphics (Portable Network Graphics, png), a bitmap (BitMap, bmp), or a portable gray map (Portable Gray Map) , pgm) is stored in the storage unit. The map construction method according to claim 1, wherein the map construction equipment is mounted on a mobile robot, and the map construction method further includes the following steps: h) the mobile robot loads the map from the storage unit Data and start to move; i) continue to obtain the color image through the image sensor, and continue to obtain the depth image information through the depth sensor; j) determine whether the map data needs to be updated; k) determine the need to update the When mapping data, execute the detection algorithm to perform image recognition on the color image; l) mark a second dynamic object in the color image and generate a second mark frame; m) the second mark frame The coordinate of the depth image corresponding to the depth image information; n) searching and filtering the second marker frame in the depth image information to generate the corrected depth image information; o) performing the map building based on the corrected depth image information Placement algorithm; and p) generating a partial map image from the map creation algorithm, and updating the map data according to the partial map image. A map building device includes: an image sensor that continuously obtains a color image of the surrounding environment after the map building device is activated; and a depth sensor that continuously obtains the surrounding environment after the map building device is activated A depth image information; A processor, electrically connected to the image sensor and the depth sensor, the processor executes a detection algorithm to perform image recognition on the color image and determine whether a dynamic object exists in the color image, when the When the dynamic object exists, the dynamic object is marked in the color image and a mark frame is generated, wherein the processor corresponds the mark frame to a depth image coordinate of the depth image information, and searches in the depth image information And filtering the marker frame to generate a corrected depth image information, and the processor executes a map building algorithm based on the corrected depth image information to generate a map data, wherein the marker is filtered out of the corrected depth image information Depth information at the corresponding position of the frame, where the detection algorithm is a Histograml of Oriented Gradient (HOG) feature extraction algorithm, Support Vector Machine (SVM) algorithm, convolutional neural network One or a combination of Convolutional Neural Network (CNN) algorithm, YOLO (You Only Look Once) algorithm, SSD (Single Shot multibox Detector) algorithm or other object detection algorithms trained with neural networks; and A storage unit is electrically connected to the processor and stores the map data.

Images