KR102521714B1 - Moving device equipped with obstacle avoidance self-determination solution based artificial intelligence and point cloud, method thereof - Google Patents

Abstract

The present invention is a method for controlling obstacle avoidance driving in an autonomous driving space performed by a mobile device, comprising: collecting point cloud data in the autonomous driving space through a first sensor of the mobile device; acquiring an image of the self-driving space through a second sensor of the image, determining whether or not at least one obstacle included in the obtained image is a non-avoidable obstacle, and as a result of the determination, the obstacle is a non-avoidable obstacle. In this case, converting the coordinates of the position of the non-avoiding obstacle in the image into non-avoiding point cloud coordinates corresponding to the point cloud data, and non-avoiding point cloud coordinates of the non-avoiding obstacle in the point cloud data. Clustering them to create at least one cluster, and controlling the non-avoidable obstacle to move without avoiding it while moving the autonomous driving space based on point cloud data including the cluster.

Description

Mobile device equipped with artificial intelligence and point cloud-based obstacle avoidance self-determination solution and its method

The present invention relates to a mobile device equipped with an artificial intelligence and point cloud-based obstacle avoidance self-determination solution and a method therefor.

Recently, service robots can be easily seen in various places such as hotels, shopping malls, and public institutions. These service robots may mean all of professional service robots used in enterprises and public places, general home service robots, and personal service robots.

In 2019, the global service robot market size was $9.46 billion, up 14.1% from the previous year. Over the past five years, the service robot market has recorded an average annual growth rate of 21.9% and is expected to grow to $4 billion in 2021.

In particular, service robots are mostly self-driving robots and are widely used in indoor spaces as robots for housework, customer reception, and education.

In the indoor space, there are many obstacles that limit the movement of autonomous robots, such as walls, pillars, chairs, desks, bookshelves, and multifunction devices. On the other hand, there are non-avoidable objects that are not damaged even if an autonomous robot collides in an indoor space.

However, self-driving robots that mostly drive indoors recognize non-avoidable objects as obstacles and avoid them before driving, which reduces the efficiency of self-driving.

Therefore, there is a need for a method to increase the accuracy and efficiency of automatic driving by self-determining non-avoidable objects indoors.

Republic of Korea Patent Publication No. 10-2013-0045290

In order to solve the above problems, the present invention can provide a method of driving without avoiding a non-avoiding object by determining a non-avoiding object by itself.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In the method for controlling obstacle avoidance driving in an autonomous driving space performed by a mobile device according to the present invention for solving the above problems, point cloud data in the autonomous driving space through a first sensor of the mobile device collecting an image of the autonomous driving space through a second sensor of the mobile device, determining whether at least one obstacle included in the acquired image is a non-avoidable obstacle, the As a result of the determination, if the obstacle is a non-avoidance obstacle, converting coordinates of the position of the non-avoidance obstacle in the image into non-avoidance point cloud coordinates corresponding to the point cloud data; Generating at least one cluster by clustering non-avoidance point cloud coordinates for obstacles, and controlling the non-avoidance obstacles to move without avoiding them while moving the autonomous driving space based on point cloud data including the clusters. can include

Here, the non-avoidance obstacle determining step is performed based on a preset learning model, and the learning model is constructed by learning learning data based on a plurality of avoidance obstacles and images of a plurality of non-avoidance obstacles. it could be

Here, the learning data may be labeled with identification information indicating that each obstacle included in the image is an avoidance obstacle or a non-avoidance obstacle.

In addition, the first sensor includes a lidar sensor, the second sensor includes a camera sensor capable of sensing a depth with an obstacle in front, and the coordinates of the position of the non-avoidable obstacle are, Pixel coordinates of the non-avoiding obstacle may be included in the image, and the non-avoiding point cloud coordinates may be converted based on the pixel coordinates and depth coordinates of the non-avoiding obstacle.

Also, the non-avoidance point cloud coordinates may be converted by further reflecting internal and external parameters of the camera sensor.

In addition, the non-avoidance point cloud coordinates may be converted based on the following equation.

(Wherein u and v represent 2D (Dimensional) image pixel coordinates for the position of the non-avoidable obstacle, fx and fy represent focal lengths within camera internal parameters, and cx and cy represent principal points within camera internal parameters In addition, the r11 to r33 represent rotation matrix information within camera external parameters, the tx, ty, and tz represent translation information within the camera external parameters, and the Xw, Yw, and Zw represent u, Represents the non-avoiding point cloud coordinates mapped to v)

The method may further include determining at least some of the areas in the cluster as non-avoidance areas, and the moving step may include controlling the non-avoidance areas to be moved without avoiding them.

Also, the non-avoidance area may include a partial area of the non-avoidance obstacle.

In addition, a mobile device according to the present invention for solving the above problems is provided based on a memory having a plurality of processes for controlling obstacle avoidance driving of the mobile device in an autonomous driving space and the plurality of processes. and a processor controlling obstacle avoidance driving, wherein the plurality of processes include a first process for collecting point cloud data in the autonomous driving space through a first sensor of the mobile device and a second sensor of the mobile device. A second process for acquiring an image of the autonomous driving space through a second process, a third process for determining whether or not at least one obstacle included in the acquired image is a non-avoidable obstacle, and as a result of the determination, the obstacle is a non-avoidable obstacle. , a fourth process of converting coordinates of the position of the non-avoiding obstacle in the image into non-avoiding point cloud coordinates corresponding to the point cloud data, a non-avoiding point cloud for the non-avoiding obstacle in the point cloud data A fifth process of generating at least one cluster by clustering coordinates, a sixth process of determining at least some of the regions within the cluster as non-avoidance regions, and moving the autonomous driving space based on point cloud data including the cluster. and a seventh process of controlling the non-avoidance area of the non-avoidance obstacle to be moved without avoiding it.

In addition, the computer program according to the present invention for solving the above problems is a computer program for executing a method for controlling obstacle avoidance driving of the mobile device in an autonomous traveling space by being combined with a mobile device that is hardware. The computer program includes: collecting point cloud data in the autonomous driving space through a first sensor of the mobile device; obtaining an image of the autonomous driving space through a second sensor of the mobile device; Determining whether or not at least one obstacle included in the obtained image is a non-avoidable obstacle, and if the obstacle is a non-avoidable obstacle as a result of the determination, the coordinates of the position of the non-avoidable obstacle in the image are set to the point Converting to non-avoidance point cloud coordinates corresponding to cloud data, generating at least one cluster by clustering non-avoidance point cloud coordinates for the non-avoidance obstacle in the point cloud data, and generating at least one region within the cluster. Determining the area as a non-avoidance area and controlling the non-avoidance area of the non-avoidance obstacle to move without avoiding it while moving the autonomous driving space based on point cloud data including the cluster may be performed. .

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, by providing a method for driving without avoiding a non-avoiding object by determining a non-avoiding object by itself, the efficiency of driving is improved by driving without avoiding a passable space even though there is a non-avoiding object. There is an effect that can be increased.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram for explaining a system for controlling obstacle avoidance driving in an autonomous driving space according to the present invention.
2 is an exemplary diagram illustrating point cloud data acquired based on a first sensor for an autonomous driving space according to the present invention.
3 is an exemplary diagram illustrating an image obtained based on a second sensor for an autonomous driving space according to the present invention.
4 is an exemplary diagram illustrating clustering of non-avoidable obstacles when the autonomous driving space according to the present invention is an office.
5 is an exemplary view showing a non-avoidance area among areas in a cluster for non-avoidance obstacles according to the present invention.
6 is a flowchart illustrating an obstacle avoidance driving process in the second processor of the mobile device according to the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

1 is a diagram for explaining a system 1 for controlling obstacle avoidance driving in an autonomous driving space according to the present invention.

2 is an exemplary diagram illustrating point cloud data acquired based on a first sensor for an autonomous driving space according to the present invention.

3 is an exemplary diagram illustrating an image obtained based on a second sensor for an autonomous driving space according to the present invention.

4 is an exemplary diagram illustrating clustering of non-avoidable obstacles when the autonomous driving space according to the present invention is an office.

5 is an exemplary view showing a non-avoidance area among areas in a cluster for non-avoidance obstacles according to the present invention.

Hereinafter, with reference to FIGS. 1 to 5 , a system 1 for controlling obstacle avoidance driving within an autonomous driving space of a mobile device 20 according to the present invention will be described.

The system 1 determines the non-avoidance object by itself and provides a method of driving without avoiding the non-avoidance object, thereby increasing the efficiency of driving by driving without avoiding the passable space even though there is a non-avoidance object. can have an effect.

Referring to FIG. 1 , a system 1 according to the present invention generates a software or firmware program for controlling obstacle avoidance driving within an autonomous driving space of a mobile device 20 and provides the program to the mobile device 20 . (10), a mobile device 20 capable of autonomous driving, and a communication network 30 may be included. Here, the system 1 may include fewer or more components than those shown in FIG. 1 .

First, the server 10 generates and stores a program composed of a plurality of processes for controlling obstacle avoidance driving within an autonomous driving space of at least one mobile device 20 in the form of software or firmware, and stores the generated software or Firmware may be provided to the at least one mobile device 20 .

The server 10 may include a first communication unit 110 , a first memory 120 and a first processor 130 . Here, the server 10 may include fewer or more components than those shown in FIG. 1 .

The first communication unit 110 may be used between the server 10 and the wireless communication system, between the server 10 and the mobile device 20, between the server 10 and a user terminal (not shown), or between the server 10 and an external server. (not shown) may include one or more modules enabling wireless communication between them. Also, the first communication unit 110 may include one or more modules that connect the server 10 to one or more networks.

The first memory 120 may store data supporting various functions of the server 10 . The first memory 120 may store a plurality of application programs (application programs or applications) running in the server 10 , data for operation of the server 10 , and commands. At least some of these input receiving application programs may exist for basic functions of the server 10 .

More specifically, the first memory 120 may store an obstacle avoidance driving program composed of a plurality of processes for controlling obstacle avoidance driving in the autonomous driving space. Here, the plurality of processes will be described later in detail when the operation of the first processor 230 is described.

The first processor 130 may control general operations of the server 10 in addition to operations related to the application program. The first processor 130 processes signals, data, information, etc. input or output through the components described above or runs an application program stored in the first memory 120, so that the user terminal or mobile device 20 can may provide or process information or functions;

In addition, the first processor 130 may control at least some of the components discussed in conjunction with FIG. 1 in order to drive an application program stored in the first memory 120 . Furthermore, the first processor 130 may combine and operate at least two or more of the components included in the server 10 to drive the application program.

The first processor 130 may generate (or manufacture) an obstacle avoidance driving program composed of a plurality of processes for controlling obstacle avoidance driving within the autonomous driving space stored in the first memory 120 .

Thereafter, the first processor 130 transmits the automatic driving control program to the first communication unit so that the mobile device 20 can automatically drive through a plurality of processes provided by the generated obstacle avoidance driving program (or firmware). 110) may be provided to the mobile device 20.

Accordingly, the first processor 130 may control at least some of the components discussed in conjunction with FIG. 1 in order to generate the obstacle avoidance driving program stored in the first memory 120 . Furthermore, the first processor 130 may combine and operate at least two or more of the components included in the server 10 to generate the obstacle avoidance driving program.

Next, the mobile device 20 determines whether an obstacle is an avoidance obstacle or a non-avoidance obstacle when driving in an autonomous driving space, and controls obstacle avoidance driving based on the determination.

The mobile device 20 may include a second communication unit 210 , a sensor unit 220 , a second memory 230 , a second processor 240 and a driving unit 250 . Here, the mobile device 20 may be a robot capable of autonomous driving.

The second communication unit 210 is used to communicate between the mobile device 20 and the wireless communication system, between the mobile device 20 and the server 10, between the mobile device 20 and other mobile devices (not shown), and the mobile device 20. and one or more modules enabling wireless communication between the user terminal (not shown) or the mobile device 20 and an external server (not shown). Also, the second communication unit 210 may include one or more modules that connect the mobile device 20 to one or more networks.

The sensor unit 220 may include at least one of a first sensor 221 and a second sensor 222 . The sensor unit 220 may detect an obstacle or an unexpected situation with respect to the front of the mobile device 20 .

Here, the first sensor 221 may emit a laser pulse, receive the reflected light from the target object, measure the distance to the object, and image the shape of the object. For example, the first sensor 221 may be a LiDar sensor.

The second sensor 222 may obtain an image having depth data. The second sensor 222 may include a camera sensor capable of sensing a depth with an obstacle in front. For example, the second sensor 222 may be an RGB-D sensor, and the RGB-D sensor may obtain depth information of the target through the time it takes for the infrared beam to be reflected and returned after transmitting the infrared beam.

The second memory 230 may store data supporting various functions of the mobile device 20 . The second memory 230 may store a plurality of application programs (applications) that are driven in the mobile device 20 , data for operating the mobile device 20 , and commands. At least some of these application programs may be downloaded from the server 10 or an external server (not shown) through wireless communication. Also, at least some of these application programs may exist for basic functions of the mobile device 20 . Meanwhile, the application program may be stored in the second memory 230, installed on the mobile device 20, and driven by the second processor 240 to perform the operation (or function) of the device 20. there is.

In addition, the second memory 230 may be installed after receiving and storing an obstacle avoidance driving program composed of a plurality of processes for controlling automatic driving from the server 10 in the form of software or firmware.

The second processor 240 may control general operations of the mobile device 20 in addition to operations related to the application program. The second processor 240 processes signals, data, information, etc. input or output through the components described above or drives an application program stored in the second memory 230, thereby providing appropriate information or information to a user terminal (not shown). A function can be provided or processed.

In addition, the second processor 240 may control at least some of the components discussed in conjunction with FIG. 1 in order to drive an application program stored in the second memory 230 . Furthermore, the second processor 240 may combine and operate at least two or more of the components included in the mobile device 20 to drive the application program.

The second processor 240 may drive an obstacle avoidance driving program composed of a plurality of processes for controlling obstacle avoidance driving within the autonomous driving space installed in the second memory 230 .

In addition, the second processor 240 may control at least some of the components discussed in conjunction with FIG. 1 in order to drive the obstacle avoidance driving program stored in the second memory 230 . Furthermore, the second processor 240 may combine and operate at least two or more of the components included in the mobile device 20 to drive the obstacle avoidance driving program.

Hereinafter, the second processor 240 drives the obstacle avoidance driving program in a state in which an obstacle avoidance driving program composed of a plurality of processes for controlling obstacle avoidance driving within the autonomous traveling space of the mobile device 20 is installed. I would like to explain the process. Here, an operation according to the obstacle avoidance driving program of the second processor 240 may be the same as a plurality of processes of the obstacle avoidance driving program.

When the obstacle avoidance driving program is driven, the second processor 240 operates a point cloud within the autonomous driving space through the first sensor 221 of the mobile device 20 according to a first process provided by the driven program. Data can be collected (first process).

Referring to FIG. 2 , point cloud data may be 3D point cloud data for an autonomous driving space of the mobile device 20 obtained using LiDAR, which is the first sensor 221 . More specifically, the point cloud data may be 3D point cloud data representing the outline, shape, structure, and the like of the autonomous driving space.

The second processor 240 may obtain an image of the autonomous driving space through the second sensor 222 of the mobile device 20 (second process).

Referring to FIG. 3 , an image having depth data in which the autonomous driving space is photographed may be obtained. More specifically, an image is generated through the second sensor 222 in depth with respect to an obstacle (eg, a wall, a pillar, a chair, a desk, a bookshelf, a multifunction device, a chair, or clothes hanging on a hanger) located in the autonomous driving space. An image with data can be obtained.

The second processor 240 may determine whether at least one obstacle included in the obtained image is a non-avoidable obstacle (third process).

Specifically, the second processor 240 may determine whether at least one obstacle included in the acquired image is a non-avoidable obstacle based on a preset learning model.

The learning model may be built by learning learning data based on images of a plurality of avoidance obstacles and a plurality of non-avoidance obstacles. Also, the learning model may be yolo4-tiny, and yolo4-tiny may be a CNN-based deep learning algorithm called fast R-CNN.

The images of the plurality of avoidance obstacles may be images of avoidance obstacles that may be damaged when collided with, such as a wall, a pillar, a chair, a desk, a bookshelf, a multifunction device, a person (the front of the clothes when the person is wearing clothes), and the like.

Also, the images of the plurality of non-avoidable obstacles may be images of non-avoidable obstacles that are not damaged even when bumped into, such as a chair or the side of clothes hanging on a hanger.

Here, the learning data may be labeled with identification information representing avoidance obstacles or non-avoidance obstacles for each obstacle included in the image.

If the determination result is that the obstacle is a non-avoiding obstacle, the second processor 240 may convert the coordinates of the location of the non-avoiding obstacle in the image into non-avoiding point cloud coordinates corresponding to the point cloud data. (fourth process).

Here, the coordinates of the location of the non-avoidable obstacle may include pixel coordinates of the non-avoidable obstacle in the image.

Also, the non-avoidance point cloud coordinates may be converted based on the pixel coordinates and the depth coordinates of the non-avoidance obstacle.

Also, the non-avoidance point cloud coordinates may be converted by further reflecting internal and external parameters of the camera sensor. Here, the camera internal parameters may be internal parameters of the camera itself, such as a focal length and a center point of the camera. In addition, the external parameter of the camera may be physical geometrical information about the attachment position of the robot and the camera, such as the installation height and direction of the camera.

In addition, the non-avoidance point cloud coordinates may be converted based on the following formula.

[formula]

(Here, the image pixel coordinates are '2D (Dimensional) coordinates u, v for the position of the non-avoidable obstacle', 'fx, fy of the camera's internal parameters represent the focal length of the camera, and cx, cy are principal point, and 'r11 to r33 of camera external parameters are rotation matrix information and tx, ty, tz are translation information, and global point cloud coordinates can be calculated by these matrix information It may be point cloud coordinates (eg, location information of non-avoidable obstacles (clothes, etc.)) mapped to image pixel coordinates u and v.)

Accordingly, the second processor 240 calculates the 2D coordinates (u, v) of the non-avoidable obstacle in the image having the depth data in which the autonomous driving space is captured and the depth (Depth information (Z_w)) mapped to the image of the non-avoidable obstacle. ) can be used to calculate (X_w, Y_w, Z_w) of the global coordinate system through geometric transformation to obtain the coordinates of the non-avoiding point cloud.

The second processor 240 may generate at least one cluster by clustering non-avoidance point cloud coordinates for the non-avoidance obstacle in the point cloud data (fifth process).

Referring to FIG. 4 , the second processor 240 clusters non-avoidance point cloud coordinates for the side of clothes hanging on a chair, which is a non-avoidance obstacle, in point cloud data corresponding to an office, which is an autonomous driving space, and first to fourth Clusters 401, 402, 403, and 404 may be created.

The second processor 240 may determine at least some of the regions in the cluster as non-avoidance regions (sixth process).

Referring to FIG. 5 , the second processor 240 includes first to fourth regions 501, which are partial regions corresponding to the hem of the cluster regions, for each of the first to fourth clusters 401, 402, 403, and 404; 502, 503, and 504) may be determined as non-avoidance areas.

The second processor 240 may control the non-avoidance area of the non-avoidance obstacle to be moved without avoiding it while moving in the autonomous driving space based on the point cloud data including the cluster (seventh process).

More specifically, the second processor 240 may control a non-avoidance area, which is at least a partial area among areas in the cluster of point cloud data, to be moved without avoiding it.

The driving unit 250 may include a battery and a wheel or driving belt receiving power from the battery, and may provide power necessary for the movement of the mobile device 20 . Here, the driving unit 250 may be driven under the control of the second processor 240 .

The communication network 30 may transmit and receive various types of information between the server 10 and the mobile device 20 . The communication network 30 may use various types of communication networks, for example, WLAN (Wireless LAN), Wi-Fi (Wi-Fi), WiBro (Wibro), Wimax (Wimax), HSDPA (High Speed Downlink Packet Access), etc. A wired communication method such as a wireless communication method or Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), FTTH (Fiber To The Home) may be used.

On the other hand, the communication network 30 is not limited to the communication methods presented above, and may include all types of communication methods that are widely known or will be developed in the future in addition to the above-described communication methods.

6 is a flowchart illustrating an obstacle avoidance driving process in the second processor 240 of the mobile device 20 according to the present invention. Here, the operation of the second processor 240 may be performed in the mobile device 20 . Alternatively, the operation of the second processor 240 may be the same as that of the first processor 130 of the server 10 .

The second processor 240 may collect point cloud data within the autonomous driving space (S601).

Specifically, the second processor 240 may collect point cloud data within the autonomous driving space through the first sensor 221 of the mobile device 20 .

For example, the second processor 240 is a 3D point cloud representing the outline, shape, structure, etc. of the autonomous driving space of the mobile device 20 obtained by using the first sensor 221, LiDAR. The data may be point cloud data.

The second processor 240 may acquire an image of the autonomous driving space (S602).

Specifically, the second processor 240 may obtain an image of the autonomous driving space through the second sensor 222 of the mobile device 20 .

The second processor 240 determines the depth of an obstacle (eg, a wall, a pillar, a chair, a desk, a bookshelf, a multifunction device, a chair, or clothes hanging on a hanger) located in the autonomous driving space through the second sensor 222 . An image with data can be acquired.

The second processor 240 may determine whether the obstacle included in the image is a non-avoidable obstacle (S603).

The second processor 240 may determine whether at least one obstacle included in the acquired image is a non-avoidable obstacle.

More specifically, the second processor 240 may determine whether at least one obstacle included in the acquired image is a non-avoidable obstacle based on a preset learning model.

The learning model may be built by learning learning data based on images of a plurality of avoidance obstacles and a plurality of non-avoidance obstacles.

In addition, the plurality of avoidable obstacles may be obstacles that can be damaged when collided with, such as a wall, a pillar, a chair, a desk, a bookshelf, a multifunction device, a person (the front of a person wearing clothes), and the like.

In addition, the plurality of non-avoidable obstacles may be non-avoidable obstacles that are not damaged even when bumped into, such as a chair or the side of clothes hanging on a hanger.

Here, the learning data may be labeled with identification information representing avoidance obstacles or non-avoidance obstacles for each obstacle included in the image.

The second processor 240 may convert the coordinates of the position of the non-avoidance obstacle into non-avoidance point cloud coordinates (S604).

If the determination result is that the obstacle is a non-avoiding obstacle, the second processor 240 may convert the coordinates of the location of the non-avoiding obstacle in the image into non-avoiding point cloud coordinates corresponding to the point cloud data. .

Here, the coordinates of the location of the non-avoidable obstacle may include pixel coordinates of the non-avoidable obstacle in the image.

Also, the non-avoidance point cloud coordinates may be converted based on the pixel coordinates and the depth coordinates of the non-avoidance obstacle. Also, the non-avoidance point cloud coordinates may be converted by further reflecting internal and external parameters of the camera sensor.

The second processor 240 may generate a cluster by clustering non-avoidance point cloud coordinates for non-avoidance obstacles (S605).

The second processor 240 may generate at least one cluster by clustering non-avoidance point cloud coordinates for the non-avoidance obstacle in the point cloud data.

The second processor 240 may determine some of the regions within the cluster as non-avoidance regions (S606).

The second processor 240 may determine at least some of the regions in the cluster as non-avoidance regions.

The second processor 240 may control the non-avoidance area of the non-avoidance obstacle to be moved without avoiding it while moving the autonomous driving space (S607).

The second processor 240 may control the non-avoidance area of the non-avoidance obstacle to be moved without avoiding it while moving the autonomous driving space based on the point cloud data including the cluster.

Although it is described that steps S601 to S607 are sequentially executed in FIG. 6, this is merely an example of the technical idea of this embodiment, and those skilled in the art to which this embodiment belongs will Since it will be possible to change and execute the order described in FIG. 6 without departing from the essential characteristics or to execute one or more steps of steps S601 to S607 in parallel, it will be possible to apply various modifications and variations, so FIG. 6 is shown in a time-series order. It is not limited.

The method according to the present invention described above may be implemented as a program (or application) to be executed in combination with the mobile device 20, which is hardware, and stored in a medium.

The aforementioned program is such that a processor (CPU) of the mobile device 20 implements a device interface of the mobile device 20 so that the mobile device 20 reads the program and executes the methods implemented in the program. It may include a code coded in a language of the mobile device 20 such as C, C++, JAVA, or machine language that can be read through. These codes may include functional codes related to functions defining necessary functions for executing the methods, and the second processor 240 of the mobile device 20 executes the functions according to a predetermined procedure. It may include the control code related to the execution procedure required to do so. In addition, these codes refer to additional information or media necessary for the second processor 240 of the mobile device 20 to execute the functions at any location (address address) in the internal or external memory of the mobile device 20. It can further include code related to memory reference for what it should be. In addition, when the second processor 240 of the mobile device 20 needs to communicate with any other mobile device or server in the remote area in order to execute the functions, the code is 2 Communication-related codes for how to communicate with any other remote computer or server using the communication module of the communication unit 210 and what information or media should be transmitted/received during communication may be further included.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the mobile device 20 or various recording media on the user's mobile device 20 . In addition, the medium may be distributed to computer systems connected through a network, and a code readable by the mobile device 20 may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: server
110: first communication unit
120: first memory
130: first processor
20: mobile device
210: second communication unit
220: sensor unit
230: second memory
240: second processor
250: driving unit
30: communication network

Claims (10)

A lidar sensor that collects point cloud data within the driving space;
an RGB-D sensor acquiring an image of the driving space; and
A processor that performs an operation related to autonomous obstacle avoidance determination based on point cloud data in the driving space;
the processor,
Determine whether at least one obstacle included in the acquired image is a non-avoidable obstacle based on a preset learning model;
As a result of the determination, when the obstacle is the non-avoidance obstacle, at least one cluster is generated by clustering non-avoidance point cloud coordinates for the non-avoidance obstacle in the point cloud data;
Based on the point cloud data including the cluster, the driving unit is controlled so that the non-avoidable obstacle is moved without avoiding it while moving in the driving space. According to claim 1,
The coordinates of the position of the non-avoidable obstacle include pixel coordinates of the non-avoidable obstacle in the image,
The mobile device, characterized in that the non-avoidance point cloud coordinates are converted based on the pixel coordinates and the depth coordinates of the non-avoidance obstacle. According to claim 2,
The non-avoidance point cloud coordinates are:
Characterized in that the conversion is performed by further reflecting the internal and external parameters of the RGB-D sensor. According to claim 3,
The non-avoidance point cloud coordinates are:
A mobile device, characterized in that converted based on the following formula.

(Wherein u and v represent 2D (Dimensional) image pixel coordinates for the position of the non-avoidable obstacle, fx and fy represent focal lengths within camera internal parameters, and cx and cy represent principal points within camera internal parameters In addition, the r11 to r33 represent rotation matrix information within camera external parameters, the tx, ty, and tz represent translation information within the camera external parameters, and the Xw, Yw, and Zw represent u, Represents the non-avoiding point cloud coordinates mapped to v) According to claim 1,
the processor,
The movable device according to claim 1 , further determining a non-avoidance area including a partial area of a non-avoidance obstacle among areas within the cluster, and further controlling the drive unit to move the non-avoidance area without avoiding it. In the autonomous determination method of a mobile device equipped with an artificial intelligence and point cloud-based obstacle avoidance autonomous determination solution performed by a server,
collecting point cloud data in a driving space through a lidar sensor of the mobile device;
acquiring an image of the driving space through an RGB-D sensor of the mobile device;
determining whether at least one obstacle included in the acquired image is a non-avoidable obstacle based on a preset learning model through a processor of the mobile device;
generating at least one cluster by clustering non-avoiding point cloud coordinates for the non-avoiding obstacle in the point cloud data, through a processor of the mobile device, when the obstacle is the non-avoiding obstacle as a result of the determination; and
controlling a drive unit of the movable device through a processor of the movable device to move the unavoidable obstacle without avoiding it while moving in the driving space, based on point cloud data including the cluster; Including, method. According to claim 6,
The coordinates of the position of the non-avoidable obstacle include pixel coordinates of the non-avoidable obstacle in the image,
The non-avoidance point cloud coordinates are converted based on the pixel coordinates and the depth coordinates of the non-avoidance obstacle. According to claim 7,
The non-avoidance point cloud coordinates are:
Characterized in that the conversion is performed by further reflecting the internal and external parameters of the RGB-D sensor. According to claim 8,
The non-avoidance point cloud coordinates are:
Characterized in that the conversion is based on the following formula, a method.

(Wherein u and v represent 2D (Dimensional) image pixel coordinates for the position of the non-avoidable obstacle, fx and fy represent focal lengths within camera internal parameters, and cx and cy represent principal points within camera internal parameters In addition, the r11 to r33 represent rotation matrix information within camera external parameters, the tx, ty, and tz represent translation information within the camera external parameters, and the Xw, Yw, and Zw represent u, Represents the non-avoiding point cloud coordinates mapped to v) According to claim 6,
The control step is
Through the processor, further determining a non-avoidance area including a partial area of a non-avoidance obstacle among areas in the cluster, and further controlling the driving unit so that the non-avoidance area is moved without avoiding. .

Images