KR20210104530A - Robot and control method thereof - Google Patents

Abstract

A robot and a control method thereof are provided. The control method comprises steps of: acquiring first and second images of a plurality of objects at different positions; obtaining a plurality of candidate positions for a plurality of objects based on the directions of the plurality of objects in each of the first and second images and photographing positions of each of the first and second images; obtaining distance information between the robot and the plurality of objects in the first and second images by analyzing the first and second images; and determining positions of the plurality of objects from among the plurality of candidate positions based on distance information.

Description

Robot and control method thereof

The present disclosure relates to a robot and a control method thereof, and more particularly, to a robot and a control method thereof for identifying positions of a plurality of objects photographed while the robot is driving.

Conventionally, there have been various methods for generating a map corresponding to the environment in which the robot operates. For example, the robot may generate a map corresponding to the environment in which the robot operates by using a SLAM (Simultaneous Localization and Mapping) method using a lidar sensor and a vision SLAM method using a camera.

However, the map corresponding to the environment in which the conventional robot operates is not intuitive, and the location of objects such as home appliances and furniture located in the environment cannot be known. A problem existed.

The present disclosure has been devised to solve the above problems, and the present disclosure is to provide a robot capable of determining the position of an object using a plurality of images obtained by photographing a plurality of objects at different positions, and a method for controlling the same.

According to an embodiment of the present disclosure for achieving the above object, a method for controlling a robot includes: acquiring first and second images obtained by photographing a plurality of objects at different positions; obtaining a plurality of candidate positions for the plurality of objects based on the position directions of the plurality of objects in each of the first and second images and the photographing positions of the first and second images; obtaining distance information between the robot and the plurality of objects in the first and second images by analyzing the first and second images; and determining the positions of the plurality of objects from among the plurality of candidate positions based on the distance information.

On the other hand, according to an embodiment of the present disclosure for achieving the above object, the robot includes a memory for storing at least one instruction; a camera for photographing a plurality of objects; and a processor connected to the memory and the camera and controlling the robot, wherein the processor executes the at least one instruction, thereby capturing the plurality of objects at different positions through the camera. A second image is obtained, and a plurality of candidate positions for the plurality of objects are determined based on the position direction of the plurality of objects in each of the first and second images and the photographing positions of each of the first and second images. obtain, and analyze the first and second images to obtain distance information between the robot and the plurality of objects within the first and second images, and to obtain the distance information between the plurality of candidate positions based on the distance information. Determine the location of multiple objects.

As described above, according to various embodiments of the present disclosure, the robot may identify the location of the object in the environment, and may provide a map including the location of the object in the environment.

1 is a block diagram illustrating a configuration of a robot according to an embodiment of the present disclosure.
2A is a diagram illustrating that a robot captures an object at a first position, according to an embodiment of the present disclosure;
2B is a diagram illustrating a picture in which the robot captures an object at a first position.
2C is a diagram illustrating that the robot captures an object at a second position, according to an embodiment of the present disclosure;
FIG. 2D is a diagram illustrating a picture in which the robot captures an object at a second position.
3 is a diagram for describing a method of identifying a candidate location for one object according to the present disclosure.
4 is a diagram for describing a method of identifying candidate positions for two objects according to the present disclosure.
FIG. 5 is a diagram for describing a method of determining positions of two objects among a plurality of candidate positions according to the present disclosure.
6 is a diagram for explaining a method of providing a location of an object on a map according to the present disclosure.
7 is a diagram for explaining a method of identifying final positions of a plurality of objects through a statistical classification technique according to the present disclosure.
8 is a flowchart illustrating a method for controlling a robot according to the present disclosure.
9 is a block diagram for explaining a specific configuration of a robot according to the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of a robot according to an embodiment of the present disclosure. The robot 100 according to the present disclosure may determine positions of a plurality of objects photographed through the camera 120 .

Referring to FIG. 1 , the robot 100 may include a memory 110 , a camera 120 , and a processor 130 . According to the present disclosure, the robot 100 may be implemented as a robot cleaner, but is not limited thereto and may be implemented as various types of electronic devices capable of autonomous driving.

The memory 110 may store various programs and data necessary for the operation of the robot 100 . Specifically, at least one instruction may be stored in the memory 110 . The processor 130 may perform the operation of the robot 100 by executing a command stored in the memory 110 .

The memory 110 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 110 is accessed by the processor 140 , and reading/writing/modification/deletion/update of data by the processor 140 may be performed. In addition, programs and data for configuring various screens to be displayed in the display area of the display may be stored in the memory 110 .

The camera 120 may photograph an environment in which the robot 100 operates while the robot 100 is traveling. The robot 100 may acquire a plurality of images obtained by photographing a plurality of objects at different positions through the camera 120 .

The processor 130 may be electrically connected to the memory 110 to control the overall operation and function of the robot 100 . The processor 130 controls the overall operation of the robot 100 . To this end, the processor 130 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). . The processor 130 may be implemented in various ways. For example, the processor 130 may include an Application Specific Integrated Circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), or a digital signal processor (Digital Signal). Processor, DSP). Meanwhile, in the present disclosure, the term processor 130 may be used to include a central processing unit (CPU), a graphic processing unit (GPU), a main processing unit (MPU), and the like.

The processor 130 may control hardware or software components connected to the processor 140 by driving an operating system or an application program, and may perform various data processing and operations. In addition, the processor 140 may load and process commands or data received from at least one of the other components into the volatile memory, and store various data in the non-volatile memory.

In particular, the processor 130 may provide a location identification function for a plurality of objects photographed through the camera 120 . That is, through the location identification function, the robot 100 may determine the location of an object existing in the environment in which the robot 100 operates.

As an embodiment of the present disclosure, as shown in FIG. 1 , a location identification function may be implemented through a plurality of modules 1100 to 1600 included in the processor 130 . However, a plurality of modules for implementing the location identification function may be included in the robot 100, but this is only an embodiment, and at least some of the location identification function may be included in the external server.

As described above, the plurality of modules 1100 to 1600 may be located in the processor 130 , but the present invention is not limited thereto, and the plurality of modules 1100 to 1600 may be located in the memory 110 . When the plurality of modules 1100 to 1600 are located in the memory 110 , the processor 130 loads the plurality of modules 1100 to 1600 from the non-volatile memory to the volatile memory, and the plurality of modules 1100 to 1 1600) can be executed. Loading refers to an operation of loading and storing data stored in the non-volatile memory into the volatile memory so that the processor 130 can access it.

The image acquisition module 1100 is configured to acquire an image of an object existing in an environment in which the robot 100 operates through the camera 120 .

As an embodiment, the robot 100 may acquire first and second images obtained by photographing a plurality of objects.

In an embodiment, when an object is detected within the angle of view of the camera 120 , the image acquisition module 1100 may acquire an image by photographing the detected object, but is not limited thereto.

In one embodiment, the image acquisition module 1100 may acquire a plurality of images by photographing an environment in which the robot 100 operates during a preset period interval (eg, 5 seconds) while the robot 100 is driving. . In addition, the image acquisition module 1100 may identify whether an object is included in the acquired image, and may identify an image including the object from among a plurality of images.

FIG. 2A is a diagram illustrating that a robot captures an object 200 at a first position 10 according to an embodiment of the present disclosure. And, FIG. 2B is a diagram illustrating an image in which the robot 100 captures the object 200 at the first position 10 . The image acquisition module 1100 may acquire a first image obtained by photographing a TV shelf in an environment in which the robot 100 operates at a first position 10 as shown in FIG. 2B .

FIG. 2C is a diagram illustrating that the robot 100 captures the object 200 at the second position 20 according to an embodiment of the present disclosure. And, FIG. 2D is a diagram illustrating a picture in which the robot 100 captures the object 200 at the second position 20 . The image acquisition module 1100 may acquire a first image obtained by photographing a TV shelf in an environment in which the robot 100 operates at the second position 20 , as shown in FIG. 2D .

The object recognition region identification module 1200 is configured to identify a region in which an object exists in an image.

The object recognition region identification module 1200 may identify at least one object recognition region in the image by applying the image acquired by the image acquisition module 1100 to the artificial intelligence model. The artificial intelligence model according to the present disclosure is an artificial intelligence model trained to identify an object region in an image.

As an embodiment, the artificial intelligence model may be a first model trained to acquire a B-Box (Bounding Box) in an image through an object detection method. The first model may identify whether an object exists in a grid at regular intervals in the image, and the region in which the object exists may be identified as a B-Box area. In addition, the object recognition area identification module 1200 may identify the B-Box area in the image identified through the first model as the object recognition area.

As an embodiment, the artificial intelligence model may be a second model trained to acquire pixel regions in which an object is located in an image through a semantic segmentation method. The second model classifies all pixels in an image into a specific class, and classifies an area in which an object is located in units of pixels. In addition, the object recognition region identification module 1200 may identify pixels in which an object exists in the image obtained through the second model as the object recognition region.

As an embodiment, the artificial intelligence model may be a third model trained to acquire a mask of a region in which an object exists in an image through an instance segmentation method. The third model may identify regions in which a plurality of objects exist in the image, and obtain a mask corresponding to each object. That is, when the third model is used, overlapping objects in the image may be identified, respectively. In addition, the object recognition region identification module 1200 may identify a region corresponding to the mask in the image obtained through the third model as the object recognition region.

Referring to FIGS. 2B and 2D , regions 200 - 1 and 200 - 2 in which TV shelves are located in an image may be identified as object recognition regions.

Also, according to an embodiment, the object recognition area identification module 1200 may not identify the area corresponding to the TV positioned on the TV shelf as the object recognition area. That is, the object recognition area identification module 1200 may identify only the area 200 - 1 corresponding to the object in contact with the floor, such as the TV shelf of FIG. 2B , as the object recognition area.

2B and 2D illustrate that only an area corresponding to the TV shelf in contact with the floor in the image is identified as the object recognition area, but is not limited thereto. That is, the object recognition area identification module 1200 may identify the TV shelf and the area corresponding to the TV as one object recognition area.

The candidate position obtaining module 1300 is configured to identify the position direction of the object based on the position of the object recognition area in the image, and obtain the candidate position of the object through the position direction.

The candidate location obtaining module 1300 may identify the location direction of the object on the image based on the location of the object recognition area identified through the object recognition area identification module 1200 .

FIG. 3 is a diagram for explaining a method of identifying a candidate location for one object 300 according to the present disclosure. 3 shows a first position 10 of the robot 100 photographing a first image and a second position of the robot 100 photographing a second image on a map corresponding to an environment in which the robot 100 operates. (20), the actual position of the object 320 is shown.

As an embodiment, the map corresponding to the environment in which the robot 100 operates may be a map generated using a Simultaneous Localization and Mapping (SLAM) method using a lidar sensor or a vision SLAM method using a camera.

Referring to FIG. 3 , the candidate position obtaining module 1300 may identify the first direction 3100 through the first image in which the robot 100 captures the object 300 at the first position 10 . The first direction 3100 is a direction in which the robot 100 looks at the object 300 from the first position 10 , and the direction with respect to the object recognition area of the robot 100 in the first image is the first direction 3100 . ) can be identified.

As an embodiment, the first direction 3100 may be a direction pointing to any one of the object recognition areas of the robot 100 in the first image.

As an embodiment, the first direction 3100 may be a direction pointing to the midpoint of the object recognition area of the robot 100 in the first image.

As an embodiment, the first direction 3100 may be a direction pointing to the center of the area of the object recognition area of the robot 100 in the first image.

As an embodiment, the first direction 3100 may be a direction indicating a position determined based on the geometric shape of the object recognition area of the robot 100 in the first image.

In addition, the candidate position obtaining module 1300 may identify the second direction 3200 through the second image in which the robot 100 captures the object 300 at the second position 20 . The second direction 3200 is a direction in which the robot 100 looks at the object 300 at the second position 20 , and the second direction 3200 is a direction for the object recognition area of the robot 100 in the second image. ) can be identified.

As an example, the second direction 3200 may be a direction pointing to any one of the object recognition areas of the robot 100 in the second image.

As an embodiment, the second direction 3200 may be a direction pointing to the midpoint of the object recognition area of the robot 100 in the second image.

As an embodiment, the second direction 3200 may be a direction pointing to the center of the area of the object recognition area of the robot 100 in the second image.

As an embodiment, the second direction 3200 may be a direction indicating a position determined based on the geometric shape of the object recognition area of the robot 100 in the second image.

In addition, the candidate position obtaining module 1300 determines a position 30 where the first direction 3100 in the first position 10 and the second direction 3200 in the second position 20 intersect the object 300 . ) can be identified as a candidate location.

4 is a diagram for describing a method of identifying candidate positions for two objects according to the present disclosure.

4 is a first position 10 of the robot 100 photographing a first image, and a second position of the robot 100 photographing a second image, on a map corresponding to an environment in which the robot 100 operates. (20), the actual positions of the first object 410 and the second object 420 are shown.

Referring to FIG. 4 , the candidate position acquisition module 1300 performs the first-first direction through the image obtained by the robot 100 capturing the first object 410 and the second object 420 at the first position 10 . 4110 and a 1-2 direction 4120 may be identified. That is, the 1-1 direction 4110 is a direction in which the robot 100 looks at the first object 410 from the first position, and the 1-2 direction 4120 is a direction in which the robot 100 faces the first object 410 from the first position. It is a direction in which the second object 420 is viewed.

In addition, the candidate position acquisition module 1300 performs a 2-1 direction 4210 and A second-second direction 4220 may be identified. That is, the 2-1 direction 4210 is a direction in which the robot 100 looks at the first object 410 at the second position, and the 2-2 direction 4220 is a direction in which the robot 100 faces the first object 410 at the second position. It is a direction in which the second object 420 is viewed.

In addition, the candidate position obtaining module 1300 is configured in the 1-1 direction 4110 and 1-2 directions 4120 in the first position 10 and the 2-1 direction ( 4210 ) and a plurality of positions 40 - 1 , 40 - 2 , and 40 - 3 where the 2 - 2 direction 4220 intersect can be identified as candidate positions of the first object 410 and the second object 420 . can

Also, the candidate location acquisition module 1300 may match a plurality of candidate locations on a map corresponding to an environment in which the robot 100 operates. In addition, the candidate location obtaining module 1300 may obtain a plurality of candidate locations by excluding candidate locations outside a predetermined area on the map from among the plurality of candidate locations.

That is, referring to FIG. 4 , the candidate position obtaining module 1300 determines the candidate positions for the first object 410 and the second object 420 in a first position 40-1 and a second position 40-2. ) and the third position (40-3). Accordingly, when only the photographing position of the image and the direction of the object in the image are used, an error may occur as in the third position 40 - 3 . Accordingly, an error may be excluded from a plurality of candidate positions through the distance information obtaining module 1400 and the object position determining module 1500 according to the present disclosure.

The distance information acquisition module 1400 is configured to acquire distance information between the robot 100 and an object in an image. The distance information acquisition module 1400 may acquire distance information between the robot 100 and the object in the image by using the image captured by the camera 120 on which the calibration has been performed.

The distance information acquisition module 1400 may acquire pixel information corresponding to the lower end of the object recognition region in the image identified through the object recognition region identification module 1200 . The pixel information, for example, may refer to coordinate information at which a pixel is located in an image.

Referring to FIG. 2B , the distance information obtaining module 1400 may obtain first pixel information 210-1 corresponding to the lower end of the object recognition area 200-1 in the image. And, referring to FIG. 2D , the distance information obtaining module 1400 may obtain second pixel information 210 - 2 corresponding to the lower end of the object recognition area 200 - 1 in the image.

In addition, the distance information acquisition module 1400 may acquire distance information between the robot 100 and the object in the image by using the acquired pixel information. The distance information acquisition module 1400 may acquire distance information between an object existing in the image and the robot 100 photographing the image by using the position of the line corresponding to the pixel information in the image. For example, an object in which a position of a line corresponding to pixel information of a plurality of objects included in the image is relatively lower may be closer to the robot 100 than other objects.

The object location determination module 1500 is configured to determine the locations of a plurality of objects by using the plurality of candidate locations obtained by the candidate location obtaining module 1300 and the distance information obtained from the distance information obtaining module 1400 .

Specifically, the distance between the plurality of candidate positions and the positions at which the plurality of images are captured among the plurality of candidate positions acquired by the candidate position acquiring module 1300 is determined by determining the distance between the distance information acquired by the distance information acquiring module 1400 and the preset distance ( For example, positions of a plurality of objects may be obtained by excluding candidate positions that are different by 1 m or more or different by a preset range ratio (eg, 80%) or more.

FIG. 5 is a diagram for describing a method of determining positions of two objects among a plurality of candidate positions according to the present disclosure.

FIG. 5 further illustrates pixel areas 410-1, 410-2, 420-1, and 420-2 corresponding to the lower end of the object recognition area in the image on the map corresponding to the environment in which the robot 100 operates. It is a drawing. That is, in the 1-1 pixel area 410 - 1 , the pixel corresponding to the lower end of the object recognition area corresponding to the first object 410 in the first image captured at the first position 10 is the robot 100 . Indicates an area located on the map corresponding to the operating environment, and the 1-2 pixel area 410 - 2 corresponds to the second object 420 in the first image captured at the first location 10 . A pixel corresponding to the lower end of the object recognition area indicates an area located on the map corresponding to the environment in which the robot 100 operates. Also, the 2-1 pixel area 420-1 is a pixel corresponding to the lower end of the object recognition area corresponding to the first object 410 in the second image captured by the robot 100 at the second position 20 . Indicates an area located on the map corresponding to the environment in which the robot 100 operates, and the 2-2 pixel area 420 - 2 is a second image captured by the robot 100 at the second location 20 . A pixel corresponding to the lower end of the object recognition area corresponding to the second object 420 represents an area located on the map corresponding to the environment in which the robot 100 operates.

And, referring to FIG. 5 , the object positioning module 1500 is configured to determine a plurality of candidate positions 40-1, 40-2, and 40-3 among a plurality of candidate positions 40-1, 40-2, and 40-3. ) and the distance between the first location 10 and the second location 20 is different from the distance information obtained in the distance information acquisition module 1400 by more than a preset distance (for example, 1 m), or a preset range ratio (for example, 80%) or more, the position 40 - 1 of the first object 410 and the position 40 - 2 of the second object 420 may be obtained by excluding candidate positions different from each other.

Specifically, the distance between the third candidate position 40 - 3 and the first position 10 , the distance between the 1-2 th pixel area 410 - 2 and the first position 10 , and the third candidate position 40 - 3) The distance between the second location 20 and the second-first pixel area 420-1 and the second location 20 may be different from each other by a preset range ratio (eg, 80%) or more. Accordingly, the object positioning module 1500 determines the positions excluding the third candidate position 40-3 among the plurality of candidate positions 40-1, 40-2, and 40-3 for the first object 410 and the second position. The object 420 may be identified as positions 40 - 1 and 40 - 2 corresponding to the object 420 .

The object map generation module 1600 is configured to provide the identified object location on a map corresponding to the environment in which the robot 100 operates.

When the positions of the plurality of objects are determined through the object positioning module 1500 , the object map generating module 1600 may display the plurality of objects on a map corresponding to the environment in which the robot 100 operates.

The map corresponding to the environment in which the robot 100 operates may be a map generated using a Simultaneous Localization and Mapping (SLAM) method using a lidar sensor or a vision SLAM method using a camera.

The robot 100 may generate a map corresponding to the environment in which the robot 100 operates, but is not limited thereto. The robot 100 may receive a map corresponding to the environment in which the robot 100 operates from an external server and store it in the memory 110 .

The object map generating module 1600 may identify the location information of the plurality of objects on the map by matching the locations of the plurality of objects on the map corresponding to the environment in which the robot 100 operates.

As an embodiment, the object map generating module 1600 may identify size information of a plurality of objects by using pixel information at the bottom of the object recognition area obtained through the distance information obtaining module 1400 . That is, the object map generation module 1600 may use the length information of the bottom of the object recognition area obtained through the distance information obtaining module 1400 to identify the object as having a larger size as the length of the bottom of the object recognition area is longer. .

In addition, the object map generating module 1600 may display a plurality of objects on the map by using the identified size information and location information.

6 is a diagram for explaining a method of providing a location of an object on a map according to the present disclosure. As an embodiment, the object map generating module 1600 may display icons 610 to 640 corresponding to a plurality of objects on the map.

The icons 610 to 640 corresponding to the plurality of objects may include information on the plurality of objects. That is, the first icon 610 may include information that the object corresponding to the first icon 610 is a table, and the second icon 610 may include information that the object corresponding to the second icon is TV. have.

In an embodiment, information on a plurality of objects may be acquired through a plurality of images captured by the image acquisition module 1100 . When an image obtained by photographing an object is acquired through the image acquiring module 1100 , the object map generating module 1600 may identify the type of the object. For example, the object map generation module 1600 may input an image to the artificial intelligence model learned to identify the object to identify the type of object included in the image. For example, when the image of FIG. 2B is input to the artificial intelligence model learned to identify the object, the object map generating module 1600 may identify the type of object included in the image as a TV and a TV shelf.

As an embodiment, when the type of the object is identified through the artificial intelligence model learned to identify the object, a first user interface (UI) asking whether the type of the identified object is correct may be provided. When it is confirmed that the type of the identified object is correct through the first UI, the object map generation module 1600 may display information on the type of the identified object on the icon of the identified object. When it is confirmed that the types of objects identified through the first UI are different, the object map generation module 1600 may provide a second user interface (UI) for acquiring the types of objects.

As an embodiment, the robot 100 may obtain information on a plurality of objects by a user. The object map generation module 1600 may provide a second user interface (UI) for acquiring the type of object corresponding to the icon displayed on the map. The object map generation module 1600 may obtain the type of an object corresponding to the icon displayed on the map from the user through the second UI, and display information corresponding to the object on the icon.

In the above description, a method of determining the positions of a plurality of objects by using the first image and the second image captured by the robot 100 at the first position 10 and the second position 20 has been described. The present disclosure is not limited thereto.

As an embodiment, the image acquisition module 1100 may acquire a third image obtained by photographing a plurality of objects at a location different from that of the first image and the second image.

In addition, the object position determining module 1500 may determine positions of a plurality of objects based on the first image and the third image. That is, the candidate location obtaining module 1300 may obtain a candidate location based on the first image and the third image. Also, the distance information acquisition module 1400 may acquire distance information between the robot 100 and the plurality of objects based on the first image and the third image. In addition, the object position determining module 1500 may determine positions of a plurality of objects based on the first image and the third image.

Also, the object position determining module 1500 may determine positions of a plurality of objects based on the second image and the third image.

In addition, the object positioning module 1500 determines the positions of a plurality of objects determined based on the first image and the second image, and the object position determining module 1500 determines the positions of the plurality of objects determined based on the first image and the third image. The location and object location determination module 1500 may determine the final location of the plurality of objects by considering the second image and the locations of the plurality of objects determined based on the second image. For example, the object positioning module 1500 may be configured to set a preset range (eg, 2m) among coordinates corresponding to positions of a plurality of objects obtained based on different combinations of two of the first image, the second image, and the third image. The coordinates in the plurality of objects may be identified as coordinates corresponding to any one of the objects, and the object position determination module 1500 calculates the average value of the coordinates within a preset range (eg, 2m) of the one object. can be identified by the final location of

As an embodiment, the image acquisition module 1100 may acquire the first image and the second image, and further acquire at least one additional image obtained by photographing a plurality of objects.

In addition, the object position determining module 1500 may determine the positions of the plurality of objects, respectively, based on a combination of two different images from among the first image, the second image, and at least one additional image.

In addition, the object position determining module 1500 may statistically analyze positions determined for each combination to determine the final positions of the plurality of objects.

7 is a diagram for describing a method of identifying final positions of a plurality of objects through a statistical classification technique according to the present disclosure.

While the robot 100 is traveling in an environment in which the robot 100 operates, the robot 100 may acquire a plurality of images obtained by photographing any one of the plurality of objects at different positions. In addition, the object positioning module 1500 uses the object recognition area within the plurality of images, the location where the plurality of images are captured on the map corresponding to the environment in which the robot 100 operates, and the plurality of objects at the location. You can identify the final positions of objects by indicating the direction of . Referring to FIG. 7 , a plurality of unfilled circles displayed on the map indicate positions at which a plurality of images are captured, and a line corresponding to the empty circle indicates directions of a plurality of objects at the corresponding positions. And, the length of the line indicates the distance between the robot 100 and the object in the image obtained using pixel information corresponding to the lower end of the object recognition area included in the image captured at the corresponding position.

The object positioning module 1500 statistically classifies a plurality of empty circles and lines corresponding to the empty circles displayed on the map as shown in FIG. 7 to determine final positions 70-1 to 70-4 of the plurality of objects. can For example, the object positioning module 1500 statistically classifies the positions of a plurality of objects obtained using two different images among a plurality of images by using the K-means clustering technique to obtain the final positions 70 of the plurality of objects. -1 to 70-4) can be determined. The K-means clustering technique is a separable clustering algorithm. Each cluster has one centroid, and the centroid of each cluster can be identified as the average value of the distance within the cluster. For example, the object positioning module 1500 may identify a K value corresponding to the number of clusters through the Silhouette method. The Silhouette method is a method for identifying a value of K that minimizes dissmilarity between any one of a plurality of data belonging to a cluster and data within a cluster to which any one data belongs. As an embodiment, the number of identified K may be the number of a plurality of objects according to the present disclosure. That is, referring to FIG. 7 , the object positioning module 1500 classifies the positions of a plurality of objects obtained using two different images among a plurality of images into K (four in FIG. 7) clusters, The center point of each cluster may be identified as final positions 70 - 1 to 70 - 4 of the plurality of objects.

8 is a flowchart illustrating a method for controlling a robot according to the present disclosure.

The robot 100 may acquire first and second images obtained by photographing a plurality of objects at different positions (S810).

In addition, the robot 100 may obtain a plurality of candidate positions for the plurality of objects based on the directions of the plurality of objects in each of the first and second images and the photographing positions of each of the first and second images (S820). ).

For example, the robot 100 may identify a plurality of object recognition regions within the first and second images by applying the first and second images to the artificial intelligence model. The artificial intelligence model according to the present disclosure is an artificial intelligence model trained to identify an object region in an image.

In addition, the robot 100 may identify the directions of the plurality of objects based on the positions of the object recognition regions in the first image and the second image. In addition, the robot 100 is located at a position where the direction of each object at the capturing position of the first image and the direction of each object at the capturing position of the second image intersect on a map corresponding to the environment in which the robot 100 operates. Corresponding coordinates may be identified as a plurality of candidate positions.

When a plurality of candidate positions are identified, the robot 100 may analyze the first and second images to obtain distance information between the robot 100 and the plurality of objects within the first and second images ( S830 ).

For example, the robot 100 may acquire pixel information corresponding to the lower end of the object recognition area in the first and second images. In addition, the robot 100 may acquire distance information between the robot 100 and a plurality of objects in the first and second images by using the acquired pixel information.

Then, the robot 100 may determine the positions of the plurality of objects from among the plurality of candidate positions based on the distance information ( S840 ).

For example, the robot 100 excludes candidate positions in which a distance between a plurality of candidate positions among a plurality of candidate positions and coordinates corresponding to the plurality of candidate positions is different from the distance information obtained in step S830 by more than a preset range, and thereby the plurality of objects position can be determined. For example, in the robot 100, the distance between the plurality of candidate positions among the plurality of candidate positions and the coordinates corresponding to the plurality of candidate positions is different from the distance information obtained in step S830 by a preset distance (eg, 1 m) or more, or The positions of the plurality of objects may be determined by excluding candidate positions different by a range ratio (eg, 80%) or more.

9 is a block diagram illustrating a specific configuration of a robot according to the present disclosure. As an embodiment, FIG. 9 may be a block diagram for a case in which the robot 900 is a robot cleaner.

Referring to FIG. 9 , the robot 900 includes a memory 910 , a camera 920 , a processor 930 , a driving unit 940 , a suction unit 950 , a battery 960 , a sensor 970 and a communication unit ( 980) may be included. However, such a configuration is an example, and it goes without saying that a new configuration may be added or some configuration may be omitted in addition to this configuration in carrying out the present disclosure. The memory 910 , the camera 920 , and the processor 930 have been described with reference to FIG. 1 . Hereinafter, the rest of the configuration will be described.

The driving unit 940 is configured to move the robot 900 under the control of the processor 930 , and may include a motor and a plurality of wheels. Specifically, the driving unit 940 may change the moving direction and the moving speed of the robot 900 under the control of the processor 930 .

The suction unit 950 may suck in dust on the bottom surface of the robot 900 . For example, the suction unit 950 may perform cleaning by absorbing dust located below it while it is moving or stopped. The suction unit 950 may further include an air purification unit that purifies pollutants in the air.

The battery 960 is configured to supply power to the robot 900 , and the battery 960 may be charged by the charging station. In an embodiment, the battery 960 may include a reception resonator for wireless charging. In one embodiment, the charging method of the battery 960 is CCCV (Constant Current Constant Voltage) charging in which a preset capacity is rapidly charged through a CC (Constant Current) charging method, and the remaining capacity is charged through a CV (Constant Voltage) method. method, but is not limited thereto and may be charged in various ways.

The sensor 970 may include various sensors necessary for the operation of the robot 900 . For example, the sensor 970 may include a distance sensor, a lidar sensor, a geomagnetic sensor, and the like.

The distance sensor is a configuration for obtaining distance information from the charging station of the robot 900. The distance sensor is an infrared sensor, an infrared sensor, an ultrasonic sensor, an RF sensor, etc. It may be implemented, and may be provided on one side of the inside or outside of the robot 900 .

A LiDAR sensor emits a laser pulse and uses the time it takes for a laser pulse to be scattered or reflected from a target device to return, and changes in the intensity, frequency, and polarization state of the scattered or reflected laser to provide physical properties related to the target object. It is a sensor capable of acquiring information about (the position and direction of the robot 900 , the distance and direction between the robot 900 and the target object, the shape and movement speed of the target object, etc.).

Specifically, the robot 900 may acquire a lidar map by scanning the periphery of the robot 900 using a lidar sensor. The lidar map is a map that can be obtained by using information on the physical characteristics of the robot 900 obtained by using the laser pulse of the lidar sensor. In addition, the robot 900 is SLAM using the lidar sensor. , to obtain information on the position of the robot 900 on the lidar map.

The geomagnetic sensor is a sensor for detecting a value of the geomagnetism, and through the geomagnetic sensor, information about a geomagnetism direction around the geomagnetic sensor and information about a geomagnetism magnitude may be obtained.

The communication unit 980 may communicate with an external device and an external server through various communication methods. The communication connection of the communication unit 980 with an external device and an external server may include communicating through a third device (eg, a repeater, a hub, an access point, a gateway, etc.).

Meanwhile, the communication unit 980 may include various communication modules to communicate with an external device. For example, the communication unit 980 may include a wireless communication module, for example, LTE, LTE Advance (LTE-A), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications) system), a cellular communication module using at least one of Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). As another example, the wireless communication module may include, for example, at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), and Zigbee.

In an embodiment, the processor 930 may receive a lidar map or a geomagnetic map corresponding to an environment in which the robot 900 operates from an external device or an external server through the communication unit 980 and store it in the memory 910 . have.

As an embodiment, the processor 930 may provide a user interface (UI) for identifying the type of an object to the user terminal device through the communication unit 980 . In addition, the processor 930 may receive information on a plurality of objects from the user terminal device through the communication unit 980 .

The present embodiments can apply various transformations and can have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, like reference numerals may be used for like components.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted.

In addition, the above-described embodiments may be modified in various other forms, and the scope of the technical spirit of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to more fully and complete the present disclosure, and to fully convey the technical spirit of the present disclosure to those skilled in the art.

The terms used in the present disclosure are used only to describe specific embodiments, and are not intended to limit the scope of rights. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element).

On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this disclosure depends on the context, for example, "suitable for," "having the capacity to" ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware.

Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” refers to a dedicated processor (eg, an embedded processor) for performing the operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, various elements and regions in the drawings are schematically drawn. Accordingly, the technical spirit of the present disclosure is not limited by the relative size or spacing drawn in the accompanying drawings.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one. In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, the above-described method according to various embodiments of the present disclosure may be stored in a non-transitory readable medium. Such a non-transitory readable medium may be mounted and used in various devices.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the programs for performing the above-described various methods may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

According to an embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In addition, although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure belongs without departing from the gist of the present disclosure as claimed in the claims Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: robot 110: memory
120: camera 130: processor

Claims (18)

In the robot control method,
acquiring first and second images obtained by photographing a plurality of objects at different positions;
obtaining a plurality of candidate positions for the plurality of objects based on the directions of the plurality of objects in each of the first and second images and the photographing positions of each of the first and second images;
obtaining distance information between the robot and the plurality of objects in the first and second images by analyzing the first and second images; and
and determining positions of the plurality of objects from among the plurality of candidate positions based on the distance information.
The method of claim 1
obtaining a third image obtained by photographing the plurality of objects at a location different from the first image and the second image;
determining positions of the plurality of objects based on the first image and the third image;
determining positions of the plurality of objects based on the second image and the third image; and
Determining the final positions of the plurality of objects in consideration of all the determined positions; Control method comprising a further comprising.
The method of claim 1
acquiring at least one additional image obtained by photographing the plurality of objects;
determining positions of the plurality of objects based on combinations of two different images from among the first image, the second image, and the at least one additional image; and
The control method further comprising a; statistically classifying the positions determined for each combination to determine the final positions of the plurality of objects.
According to claim 1,
The step of obtaining a plurality of candidate positions for the plurality of objects comprises:
identifying a plurality of object recognition regions in the first and second images by applying the first and second images to an artificial intelligence model;
identifying directions of the plurality of objects based on positions of the object recognition regions in the first and second images; and
On a map corresponding to the environment in which the robot operates, coordinates at which the direction of each object at the capturing position of the first image and the direction of each object at the capturing position of the second image intersect are identified as the plurality of candidate positions Control method comprising the; step of identifying as.
5. The method of claim 4,
The step of obtaining the distance information includes:
obtaining pixel information corresponding to the lower end of the object recognition region in the first and second images; and
and obtaining distance information between the robot and the plurality of objects in the first and second images by using the acquired pixel information.
According to claim 1,
Determining the positions of the plurality of objects comprises:
obtaining the positions of the plurality of objects by excluding candidate positions in which the distances between the plurality of candidate positions and the coordinates corresponding to the plurality of candidate positions among the plurality of candidate positions differ from the distance information by more than a preset range Characterized control method.
According to claim 1,
The step of obtaining the plurality of candidate positions comprises:
acquiring a map corresponding to an environment in which the robot operates;
matching the plurality of candidate locations to the map; and
and obtaining the plurality of candidate positions by excluding candidate positions outside a preset range on the map from among the plurality of candidate positions.
6. The method of claim 5,
identifying location information of the plurality of objects on the map by matching the locations of the plurality of objects with the map;
identifying size information of the plurality of objects by using the pixel information; and
and displaying a plurality of objects on the map by using the location information and size information of the identified plurality of objects.
9. The method of claim 8,
Further comprising; providing information on a plurality of objects on the map;
The control method, characterized in that the information on the plurality of objects is obtained through the first and second images.
In the robot,
a memory storing at least one instruction;
a camera for photographing a plurality of objects; and
A processor connected to the memory and the camera to control the robot,
The processor by executing the at least one instruction,
Obtaining first and second images of the plurality of objects at different positions through the camera,
obtaining a plurality of candidate positions for the plurality of objects based on the directions of the plurality of objects in each of the first and second images and the photographing positions of each of the first and second images,
Analyze the first and second images to obtain distance information between the robot and the plurality of objects in the first and second images,
A robot that determines positions of the plurality of objects from among the plurality of candidate positions based on the distance information.
11. The method of claim 10,
The processor is
Obtaining a third image obtained by photographing the plurality of objects through the camera at a location different from the first image and the second image,
determining the positions of the plurality of objects based on the first image and the third image,
determining the positions of the plurality of objects based on the second image and the third image,
A robot that determines the final positions of the plurality of objects by considering all the determined positions.
11. The method of claim 10,
The processor is
Obtaining at least one additional image obtained by photographing the plurality of objects through the camera,
respectively determine the positions of the plurality of objects based on combinations of two different images from among the first image, the second image, and the at least one additional image,
A robot that statistically classifies the positions determined for each combination to determine the final positions of the plurality of objects.
11. The method of claim 10,
The processor is
applying the first and second images to an artificial intelligence model to identify a plurality of object recognition regions within the first and second images;
based on the positions of the object recognition regions in the first and second images, identify directions of the plurality of objects;
On a map corresponding to the environment in which the robot operates, coordinates at which the direction of each object at the capturing position of the first image and the direction of each object at the capturing position of the second image intersect are identified as the plurality of candidate positions A robot characterized in that it is identified as
14. The method of claim 13,
The processor is
acquiring pixel information corresponding to the lower end of the object recognition area in the first and second images;
A robot, characterized in that by using the obtained pixel information, to obtain distance information between the robot and the plurality of objects in the first and second images.
11. The method of claim 10,
The processor is
and obtaining the positions of the plurality of objects by excluding candidate positions where the distances between the plurality of candidate positions and the coordinates corresponding to the plurality of candidate positions among the plurality of candidate positions are different from the distance information by more than a preset range robot that does.
11. The method of claim 10,
The processor is
Obtaining a map corresponding to the environment in which the robot operates,
matching the plurality of candidate locations to the map;
The robot according to claim 1, wherein the plurality of candidate positions is obtained by excluding candidate positions outside a preset range on the map among the plurality of candidate positions.
15. The method of claim 14,
The processor is
Match the locations of the plurality of objects to the map to identify location information of the plurality of objects on the map,
using the pixel information to identify size information of the plurality of objects,
The robot, characterized in that by using the location information and size information of the identified plurality of objects to display a plurality of objects on the map.
18. The method of claim 17,
The processor is
providing information on a plurality of objects on the map,
The robot, characterized in that the information on the plurality of objects is obtained through the first and second images.

Images