KR20240028574A - Robot managing electornic device and controlling method thereof - Google Patents

Abstract

A robot and a method for controlling the robot are disclosed. The robot includes a camera, a communication interface, a memory storing 3D information corresponding to a specific space, and one or more processors. When an electronic device is identified in an image obtained through a camera while the robot is traveling in a specific space, one or more processors obtain identification information of the identified electronic device. One or more processors transmit an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device through a communication interface. When the operation of the electronic device according to the operation confirmation signal is identified, one or more processors identify location information of the electronic device within a specific space. One or more processors update three-dimensional information by mapping the electronic device to a specific space based on the identification information of the electronic device and the location information of the electronic device.

Description

Robot for managing electronic devices and its control method {ROBOT MANAGING ELECTORNIC DEVICE AND CONTROLLING METHOD THEREOF}

This disclosure relates to a robot that manages electronic devices placed in a certain space and a method of controlling the same.

With the advancement of electronic technology, various electronic devices are being commercialized. For example, electronic devices may include smartphones, tablet PCs, desktop computers, laptop computers, audio devices, speakers, TVs, microwave ovens, etc. In addition, existing home appliances such as refrigerators, washing machines, and dryers are also developing to include communication functions, voice recognition functions, and gesture recognition functions.

An electronic device that includes a communication function can be connected to another electronic device or server through a network and transmit and receive various information to the other electronic device or server. Alternatively, an electronic device including a voice recognition function or a gesture recognition function may recognize a user's voice or gesture and perform an operation corresponding to the recognized voice or gesture.

In addition, robots are not only used for industrial purposes such as product assembly and inspection, but are also widely used in daily life by performing functions such as serving, delivery, security, and management.

A robot according to an embodiment of the present disclosure includes a camera, a communication interface, a memory storing 3D information corresponding to a specific space, and one or more processors. When an electronic device is identified in an image obtained through the camera while the robot travels in the specific space, the one or more processors may obtain identification information of the identified electronic device. The one or more processors may transmit an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device through the communication interface. When the operation of the electronic device according to the operation confirmation signal is identified, one or more processors may identify location information of the electronic device within the specific space. One or more processors may update the 3D information by mapping the electronic device to the specific space based on the identification information of the electronic device and the location information of the electronic device.

The robot control method according to an embodiment of the present disclosure can obtain identification information of the identified electronic device when an electronic device is identified in an image acquired while the robot is traveling in a specific space. The control method may transmit an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device. The control method may identify location information of the electronic device within the specific space when the operation of the electronic device according to the operation confirmation signal is identified. The control method may update stored 3D information by mapping the electronic device to the specific space based on the identification information of the electronic device and the location information of the electronic device.

A program that performs a method for controlling a robot recorded on a non-transitory computer-readable storage medium according to an embodiment of the present disclosure is configured to: When an electronic device is identified in an image acquired while the robot is traveling in a specific space, the identified electronic device Device identification information can be obtained. The program may transmit an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device. When the operation of the electronic device according to the operation confirmation signal is identified, the program can identify location information of the electronic device within the specific space. The program may update the stored 3D information by mapping the electronic device to the specific space based on the identification information of the electronic device and the location information of the electronic device.

1 is a diagram illustrating a robot system according to an embodiment of the present disclosure.
Figure 2 is a block diagram explaining the configuration of a robot according to an embodiment of the present disclosure.
Figure 3 is a block diagram explaining the specific configuration of a robot according to an embodiment of the present disclosure.
4 to 6 are diagrams illustrating a process for identifying an electronic device according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating a process of mapping an identified electronic device to 3D information according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating a process for simultaneously identifying a plurality of electronic devices according to an embodiment of the present disclosure.
Figure 9 is a timing diagram explaining the operation of a robot system that identifies an electronic device according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating a process for identifying a moving electronic device according to an embodiment of the present disclosure.
11 and 12 are diagrams illustrating a process for confirming the influence of surrounding objects according to an embodiment of the present disclosure.
FIG. 13 is a timing diagram explaining the operation of a robot system that identifies a moving electronic device according to an embodiment of the present disclosure.
Figure 14 is a diagram explaining a robot that performs a user command according to an embodiment of the present disclosure.
15 and 16 are diagrams illustrating a terminal device executing a user command according to an embodiment of the present disclosure.
Figure 17 is a flowchart explaining a method of controlling a robot according to an embodiment of the present disclosure.

Hereinafter, various embodiments will be described in more detail with reference to the attached drawings. The embodiments described herein may be modified in various ways. Specific embodiments may be depicted in the drawings and described in detail in the detailed description. However, the specific embodiments disclosed in the attached drawings are only intended to facilitate understanding of the various embodiments. Accordingly, the technical idea is not limited to the specific embodiments disclosed in the attached drawings, and should be understood to include all equivalents or substitutes included in the spirit and technical scope of the disclosure.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but these components are not limited by the above-mentioned terms. The above-mentioned terms are used only for the purpose of distinguishing one component from another.

In this specification, terms such as “comprise” or “have” are intended to indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Meanwhile, a “module” or “unit” for a component used in this specification performs at least one function or operation. And, the “module” or “unit” may perform a function or operation by hardware, software, or a combination of hardware and software. Additionally, a plurality of “modules” or a plurality of “units” excluding a “module” or “unit” that must be performed on specific hardware or performed on at least one processor may be integrated into at least one module. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the description of the present disclosure, the order of each step should be understood as non-limiting unless the preceding step must be performed logically and temporally prior to the subsequent step. In other words, except for the above exceptional cases, even if the process described as a subsequent step is performed before the process described as a preceding step, the nature of disclosure is not affected, and the scope of rights must also be defined regardless of the order of the steps. In this specification, the term “A or B” is defined not only to selectively indicate either A or B, but also to include both A and B. In addition, the term "included" in this specification has the meaning of including additional components in addition to the elements listed as included.

In this specification, only essential components necessary for description of the present disclosure are described, and components unrelated to the essence of the present disclosure are not mentioned. And it should not be interpreted in an exclusive sense that includes only the mentioned components, but in a non-exclusive sense that can also include other components.

In addition, when describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description thereof is abbreviated or omitted. Meanwhile, each embodiment may be implemented or operated independently, but each embodiment may be implemented or operated in combination.

1 is a diagram illustrating a robot system according to an embodiment of the present disclosure.

Referring to FIG. 1 , the robot system 1000 may include a robot 100, an electronic device 200, a server 300, and a terminal device 400.

The robot 100 may include map data and 3D information of a specific space. The robot 100 can explore a specific space and generate map data and 3D information. Alternatively, the robot 100 may receive map data and 3D information of a specific space from the server 300. For example, specific spaces may include home, school, work, etc. As an example, the robot 100 can explore a room, kitchen, living room, etc. in a home. The robot 100 can generate two-dimensional map data while exploring each space. Additionally, the robot 100 can identify the floor, walls, and ceiling of a room, kitchen, and living room. The robot 100 can generate 3D information using information on the identified floor, wall, and ceiling. Additionally, the robot 100 may identify objects placed in each space. For example, objects may include a dining table, a table, a bookshelf, a desk, etc.

An electronic device 200 may be placed in each space. For example, the electronic device 200 may be a network device connected to a network, or may be a general device not connected to a network. General devices not connected to a network may include electronic devices controlled with a remote control or the like. In this specification, network devices are referred to as IoT (Internet of Things) devices, and general devices controlled by a remote control and not connected to the network are referred to as IR (Infra-Red) devices. For example, the electronic device 200 may include a refrigerator, TV, robot vacuum cleaner, washing machine, dryer, air conditioner, light, speaker, sensor, power supply, router, etc. The example of the electronic device 200 described above is an example, and is not limited to the example described above.

The server 300 may communicate with the electronic device 200 (eg, IoT device) and the robot 100. The server 300 may receive information from the electronic device 200 placed in a specific space and transmit the information of the electronic device 200 to the robot 100.

For example, the electronic device 200 placed in a specific space may transmit information to the server 300. Information transmitted to the server 300 may include identification information including type, name, model name, and control-related information. The server 300 may transmit the identification information received from the electronic device 200 to the robot 100.

The robot 100 can explore space and identify the placed electronic device 200. The robot 100 may identify the electronic device 200 based on an image acquired through a camera in a specific space. The robot 100 may transmit an operation confirmation signal corresponding to the type of the identified electronic device 200 to the electronic device 200. The electronic device 200 may perform an operation according to the received operation confirmation signal. When the robot 100 identifies the operation of the electronic device 200, it can identify the location information of the electronic device in space. For example, the robot 100 may receive information related to lights placed in a space from the server 300 . The robot 100 can identify lights placed on the walls of a room from images acquired through spatial exploration and a camera. The robot 100 can transmit an operation confirmation signal to the light and identify the operation of the light. The robot 100 can identify the location information of the light. For example, the location information of the light may include coordinate information of the room, wall, and light. The robot 100 can match the identification information and location information of the light. In addition, the robot 100 can update the 3D information by mapping the information matching the lamp's identification information and location information to space.

The robot 100 may transmit 3D information in which the identification information and location information of the electronic device 200 are mapped to space to the server 300 and/or the terminal device 400. Server 300 may store the received information. The terminal device 400 may display a control user interface (UI) based on the received information. The terminal device 400 may receive a command to control the electronic device 200 on the displayed control UI. When a user's control command is input, the terminal device 400 may transmit the input control command to the corresponding electronic device 200. The electronic device 200 may perform control operations according to received control commands.

Although FIG. 1 illustrates the robot system 300 including the server 300, some or all functions of the server 300 may be performed in the robot 100. For example, the robot 100 can perform all of the functions of the server 300 described above. Alternatively, the robot 100 may request additional necessary information from the server 300 while performing the functions of the server 300 described above.

Below, various examples of the robot system 1000 will be described.

Figure 2 is a block diagram explaining the configuration of a robot according to an embodiment of the present disclosure.

Referring to FIG. 2, the robot 100 includes a camera 110, a communication interface 120, a memory 130, and a processor 140.

The camera 110 can capture images of the surrounding environment of the robot 100. The processor 140 may identify a space, an electronic device 200, an object, etc. from a captured image. Alternatively, the camera 110 may capture an image of the user. The processor 140 may identify user-related information, including the user's location, gaze direction, and pointing direction, from the captured user's image. Alternatively, the processor 140 may recognize a control command based on the user's facial expression, motion, gaze, etc., and perform a control operation corresponding to the recognized control command or the recognized area. For example, the camera 1110 may include a CCD sensor or a CMOS sensor. Additionally, the camera 110 may include an RGB camera or a depth camera.

The communication interface 120 can communicate with an external device. The communication interface 120 may receive data from an external device and transmit identified data to the external device. For example, the external device may include an electronic device 200, a server 300, and a terminal device 400. The communication interface 120 may receive identification information of the electronic device 200 from the server 300. Additionally, the communication interface 120 may transmit 3D information mapped to a specific space to the terminal device 400 based on the identification information and location information of the electronic device 200. Additionally, the communication interface 120 may transmit an operation confirmation signal, a test signal, and a control signal to the electronic device 200.

For example, the communication interface 120 may use communication methods such as Wi-Fi, Wi-Fi Direct, Bluetooth, ZigBee, DLNA, Wide, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long Term Evolution). Communication with an external device can be performed using at least one of the following communication methods. The communication interface 120 may be referred to as a communication device, communication unit, communication module, transceiver, etc.

The memory 130 may store algorithms, data, software, etc. related to the operation of the robot 100. Algorithms, data, software, etc. stored in the memory 130 may be loaded into the processor 140 under the control of the processor 140 to perform a data processing process. Additionally, the memory 130 may store identification information, location information, 3D information, etc. of the electronic device 200. For example, the memory 135 may be implemented in the form of ROM, RAM, HDD, SSD, memory card, etc.

The processor 140 can control each component of the robot 100. For example, the processor 140 may control the camera 110 to capture images and control the communication interface 110 to transmit and receive data with an external device. The robot 100 may include one processor 140 or may include a plurality of processors 140.

The processor 140 may identify the electronic device 200 from the image acquired through the camera 110 and obtain information about the identified electronic device 200. For example, the processor 140 may obtain identification information of the electronic device 200 identified in the image from the identification information of the electronic device 200 received from the server 300. Alternatively, if there is no identification information of the electronic device 200 (e.g., the identification information of the electronic device is not stored, not received, or there is no identification information corresponding to the electronic device identified from the image), the processor 140 controls the electronic device 200. ) can transmit a signal requesting identification information of the electronic device 200 to the server 300, and control the communication interface 120 to receive identification information of the electronic device 200 from the server 300.

The processor 140 may control the communication interface 120 to transmit an operation confirmation signal corresponding to the type of the identified electronic device 200 to the electronic device 200 and identify the operation of the electronic device 200. . For example, if the type of the electronic device 200 is a light output device (e.g., a light), the processor 140 controls the communication interface 120 to transmit an operation confirmation signal for outputting light to the electronic device 200. And, the camera 110 can be controlled to capture an image of the electronic device 200 that outputs light. If the type of the electronic device 200 is a sound output device (e.g., a speaker), the processor 140 controls the communication interface 120 to transmit an operation confirmation signal for outputting sound to the electronic device 200, and the electronic device 200 The speaker can be controlled to receive sound output from (200). If the type of the electronic device 200 is a communication device, an operation confirmation signal outputting a response signal is transmitted to the electronic device 200, and the communication interface 120 is controlled to receive a response signal output from the electronic device 200. can do. Alternatively, the operation confirmation signal may include a signal that turns on/off the electronic device 200. The processor 140 identifies the operation of the electronic device 200 corresponding to the operation confirmation signal transmitted to the electronic device 200 and determines whether the electronic device 200 included in the image is the electronic device 200 corresponding to the identification information. You can check whether or not.

For example, the electronic device 200 may be a device connected to a network or a device not connected to a network. A device connected to a network may be an IoT device, and a device not connected to a network may be an IR device.

If the electronic device 200 is an IR device, the processor 140 may control the camera 110 to photograph the IR device and transmit the captured image to the server 300. The server 300 may identify the IR device based on the received image and obtain identification information of the identified IR device. The server 300 may transmit identification information of the IR device to the robot 100. Accordingly, the robot 100 can obtain identification information of the IR device from the server 300 based on the captured image of the IR device.

Alternatively, if the electronic device 200 is an IoT device and a plurality of IoT devices exist, the processor 140 may simultaneously identify the plurality of IoT devices. As used herein, the meaning of simultaneous includes not only time coinciding with a specific point in time, but also sequential time at very short time intervals. For example, 1 second after the start of operation, a control signal is transmitted to multiple IoT devices, as well as 1 second, 1.1 second, 1.2 second, 1.3 second, … This may also include cases where control signals are sequentially transmitted to each IoT device. The processor 140 can simultaneously transmit distinct operation confirmation signals to a plurality of IoT devices and simultaneously identify the operations of the plurality of IoT devices. For example, distinct operation confirmation signals may include signals with distinct colors, flashing patterns, gradients, frequencies, sound output patterns, etc.

The processor 140 may identify location information of the electronic device 200. For example, location information includes not only two-dimensional location information such as room, living room, and kitchen, but also left wall, right wall, front wall based on the direction the robot 100 is facing, ceiling, floor, object, and each area. It may contain three-dimensional location information including coordinate information. The processor 140 may update 3D information by mapping the electronic device 200 to a specific space based on the identification information and location information of the electronic device 200. When a plurality of electronic devices 200 are located, the processor 140 can simultaneously identify location information of the plurality of electronic devices 200.

The processor 140 may control the communication interface 120 to transmit 3D information to which the identification information and location information of the electronic device 200 are mapped to the terminal device 400 . The terminal device 400 may display a control UI for controlling the electronic device 200 based on the received information.

Additionally, the robot 100 may detect changes in the environment after generating (or updating) 3D information of the electronic device 200 placed in a specific space. For example, a change in the environment may include changes such as placement of a new electronic device 200, absence of an existing electronic device 200, placement of a new object, etc.

As an example, when a new electronic device 200 is identified in an image acquired through the camera 110, the processor 140 sends an existing operation confirmation signal corresponding to the type of the identified electronic device 200 to the new electronic device ( 200). When the operation of the new electronic device 200 according to the existing operation confirmation signal is identified, the processor 140 may identify the new electronic device 200 as the existing electronic device whose location has been changed. The processor 140 may update 3D information that modifies the location information of an existing electronic device.

Alternatively, the processor 140 may identify a new object in the image acquired through the camera 110. When the processor 140 identifies a new object located on the floor among the 3D information in the image, it can control the communication interface 120 to transmit information related to the location of the new object to the robot cleaner. The robot cleaner can update the cleaning map based on the received information. That is, the processor 140 may control the communication interface 120 to update the cleaning map of the robot cleaner by transmitting information related to the location of the new object to the robot cleaner.

Alternatively, when a new object is identified in an image acquired through a camera, the processor 140 may transmit a test signal corresponding to the type of the electronic device 200 to the electronic device 200 through the communication interface 120. The processor 140 identifies the operating performance of the electronic device 200 according to the test signal, and when the operating performance is outside a preset range, the processor 140 uses the communication interface 120 to transmit information related to the operating performance to the terminal device 400. You can control it. A detailed explanation will be provided later.

Alternatively, when the robot 100 is located near the user and receives the user's voice command, the processor 140 can control the camera 110 to photograph the user. The processor 140 may identify user-related information including the user's location, gaze direction, and pointing direction from the user's image acquired through the camera 110. The processor 140 may identify the control target electronic device 200 based on the identified user-related information. A detailed explanation will be provided later.

Figure 3 is a block diagram explaining the specific configuration of a robot according to an embodiment of the present disclosure.

Referring to FIG. 3, the robot 100 includes a camera 110, a communication interface 120, a memory 130, a processor 140, a user interface 150, a microphone 155, a sensor 160, and a speaker ( 165), a display 170, a driving unit 175, and a power supply unit 180. The camera 110, communication interface 120, and memory 130 may be the same as those described in FIG. 2.

The user interface 150 may receive commands, etc. from the user. The user interface 150 may be implemented in various forms. For example, the user interface 150 may include a keyboard, buttons, key pad, touch pad, touch screen, etc. The user interface 150 may also be called an input device, input unit, input module, etc.

The microphone 155 can receive the user's voice. The processor 140 may recognize a control command based on the input voice and perform a control operation corresponding to the recognized control command.

The sensor 160 may detect information related to the user or the surrounding environment. The processor 140 may perform control operations based on the sensed information. For example, the sensor 160 may include a distance sensor, an image sensor, a tracking sensor, an angle sensor, an acceleration sensor, a gravity sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, a motion recognition sensor, a proximity sensor, a voltmeter, an ammeter, a barometer, and a hygrometer. , a thermometer, an illumination sensor, a heat sensor, a touch sensor, an infrared sensor, an ultrasonic sensor, etc. Distance sensors can be implemented as ultrasonic sensors, laser sensors, LiDar, etc.

The speaker 165 outputs a sound signal on which sound signal processing has been performed. For example, the speaker 165 may output a user's input command, status-related information of the robot 100, operation-related information, and notification of abnormalities in the electronic device to be diagnosed, etc., as a voice or notification sound.

Display 170 may display information in a visual manner. For example, the display 150 may be implemented as a Liquid Crystal Display (LCD), Organic Light Emitting Diode (OLED), or a touch screen. When the display 170 is implemented as a touch screen, the robot 100 can receive control commands through the touch screen.

The driving unit 175 may move the robot 100 or perform a preset control operation. Alternatively, the driving unit 175 may rotate the camera 110 or the part including the camera 110 up, down, left and right. The driver 175 may operate under the control of the processor 140 based on stored map data and sensed information. For example, the driving unit 165 may be implemented in the form of a module including a wheel for movement, or a robot arm that performs a preset control operation.

The power supply unit 180 can receive power from the outside and supply power to various components of the robot 100.

The robot 100 may include all of the above-described configurations or may include some of the configurations. Alternatively, the robot 100 may include additional components other than those described above.

So far, the configuration of the robot 100 has been described. Below, the process by which the robot 100 operates will be described.

4 to 6 are diagrams illustrating a process for identifying an electronic device according to an embodiment of the present disclosure. And, FIG. 7 is a diagram illustrating a process of mapping an identified electronic device to 3D information according to an embodiment of the present disclosure.

4 to 6 show electronic devices 200a, 200b, 200c, and 200d and a robot 100 installed in a specific space. For example, the specific space may be a room 10.

Referring to FIG. 4 , new electronic devices 200a, 200b, 200c, and 200d may be installed in the room 10. The robot 100 may receive identification information of the newly installed electronic devices 200a, 200b, 200c, and 200d from the server 300. Since there is no location information for the newly installed electronic devices 200a, 200b, 200c, and 200d, the robot 100 may classify the electronic devices 200a, 200b, 200c, and 200d as unknown. The robot 100 can move to the room 10 while exploring each space. The robot 100 can identify the electronic devices 200a, 200b, 200c, and 200d installed in the room 10 using a camera. Since the robot 100 knows the identification information of the electronic devices 200a, 200b, 200c, and 200d, it can know the type of each electronic device 200a, 200b, 200c, and 200d. The robot 100 may transmit an operation confirmation signal corresponding to the type of the electronic device to each of the electronic devices 200a, 200b, 200c, and 200d. As shown in FIG. 4, the robot 100 may transmit an operation confirmation signal corresponding to the type of the second electronic device 200b to the second electronic device 200b. Since the second electronic device 200b has received an operation confirmation signal corresponding to its type from the robot 100, it can perform an operation based on the operation confirmation signal.

For example, if the type of the second electronic device 200b is a light output device, the operation confirmation signal may be a control signal for outputting light. If the type of the second electronic device 200b is a sound output device, the operation confirmation signal may be a control signal for outputting sound. If the type of the second electronic device 200b is a communication device, the operation confirmation signal may be a control signal that outputs a response signal.

The second electronic device 200b shown in FIG. 4 may be a lamp, and the operation confirmation signal may be a control signal that outputs light corresponding to the lamp type. The second electronic device 200b may output light based on the received operation confirmation signal. The robot 100 can capture an image of the second electronic device 200b that outputs light using a camera. And, the robot 100 can check the operation of the second electronic device 200b.

If the type of the second electronic device 200b is a sound output device, the operation confirmation signal may be a control signal for outputting sound. The robot 100 may transmit an operation confirmation signal including sound type and volume information to the second electronic device 200b and check the output sound of the second electronic device 200b. If the type of the second electronic device 200b is a communication device, the operation confirmation signal may be a control signal that outputs a response signal. The robot 100 may identify the second electronic device 200b based on the signal strength and response message of the second electronic device 200b. Additionally, the second electronic device 200b may include an indicator that displays an operating status. When performing an operation based on the operation confirmation signal, the second electronic device 200b may indicate the operation status using an indicator. Therefore, even when the second electronic device 200b is a sound output device or a communication device, the robot 100 captures an image of the second electronic device 200b operating and uses the captured image to create a sound output device or a communication device. The operation of (200b) can be identified.

Upon confirming the operation of the second electronic device 200b, the robot 100 can identify the location information of the second electronic device 200b. For example, room 10 may include a left wall 11, a right wall 12, a front wall 13, a ceiling 14, and a floor 15. The robot 100 can capture images of the room 10 using a camera and identify each side of the room 10. The robot 100 may identify the location of the second electronic device 200b as the ceiling 14.

As shown in FIG. 5, the robot 100 can draw a virtual grid on the ceiling 14. The robot 100 can set any point in the virtual grid as a reference point. Additionally, the robot 100 can extract the coordinates of each electronic device 200a, 200b, and 200c based on the reference point and the virtual grid. For example, as shown in FIG. 5, the robot 100 has coordinates of the first electronic device 200a (3, 3), coordinates of the second electronic device 200b (3, 5), and 3 The coordinates of the electronic device 200c can be determined to be (3, 7).

In the above-described example, the robot 100 may identify the second electronic device 200b based on the identification information of the second electronic device 200b received from the server 300. Additionally, the robot 100 may determine that the location of the second electronic device 200b is the room 10, the ceiling 14, and the coordinates (3, 5). The robot 100 can match the identification information and location information of the second electronic device 200b. Additionally, the robot 100 may generate three-dimensional information about a specific space before the electronic devices 200a, 200b, 200c, and 200d are installed. For example, the robot 100 may generate 3D information of a specific space through advance search, and may also generate 3D information by receiving information about a specific space from the server 300. Accordingly, the robot 100 can update the pre-generated 3-D information by mapping the matching information in which the identification information and location information are matched to the pre-generated 3-D information.

The robot 100 determines the location information for the first electronic device 200a, the third electronic device 200c, and the fourth electronic device 200d through a similar process, and determines the location information of each electronic device 200a, 200c, and 200d. Identification information and location information can be matched. Additionally, the robot 100 may update the 3D information by mapping the matching information of each electronic device 200a, 200c, and 200d to the 3D information.

The first to fourth electronic devices 200a, 200b, 200c, and 200d described above may be IoT devices connected to a network. However, there may be IR devices located in a room that are not connected to the network. An IR device may be a device that is not connected to a network and is controlled by a remote control.

Referring to FIG. 6, a fifth electronic device 200e, which is an IR device, is shown. The robot 100 may not know the identification information of the fifth electronic device 200e that is not connected to the network. When the robot 100 searches the room 10 using a camera, it can identify the fifth electronic device 200e. Since the robot 100 does not know the identification information of the fifth electronic device 200e, it cannot generate matching information for the fifth electronic device 200e.

The robot 100 may capture an image of the fifth electronic device 200e and transmit the captured image to the server 300. The robot 100 can capture one image and transmit one image to the server 300, and can also capture multiple images and transmit the multiple images to the server 300. For example, the robot 100 captures images of the fifth electronic device 200e from various angles, such as the front, side, and back of the fifth electronic device 200e, and sends the plurality of captured images to the server 300. It can be sent to . Additionally, the robot 100 may recognize text displayed on the fifth electronic device 200e and transmit an image capturing the recognized text area to the server 300. The server 300 may receive an image of the fifth electronic device 200e from the robot 100. The server 300 may recognize the shape, shape, and text of the fifth electronic device 200e from the received image. Additionally, the server 300 may obtain identification information of the fifth electronic device 200e through search. The server 300 may transmit the acquired identification information of the fifth electronic device 200e to the robot 100. The robot 100 may identify the location information of the fifth electronic device 200e based on the received identification information of the fifth electronic device 200e.

For example, the robot 100 may transmit an operation confirmation signal corresponding to the type of the fifth electronic device 200e to the fifth electronic device 200e based on the identification information of the fifth electronic device 200e. For example, the fifth electronic device 200e may be a fan, and the operation confirmation signal may be a control signal for turning the fan on/off or a control signal for operating the fan. The robot 100 can check the operation of the fan using a camera. When the robot 100 checks the operation of the fan, it can identify the location information of the fan. The robot 100 can confirm that the fan is located on the floor 15. The robot 100 can create a virtual grid on the floor 15 and set a reference point. The robot 100 can acquire coordinate information of the fan based on the virtual grid and reference point. For example, as shown in FIG. 6, the robot 100 can identify location information of the fan, such as room 10, floor 15, and coordinates (1, 8). The robot 100 can generate matching information by matching the identification information and location information of the fan. And, the robot 100 can update the 3D information by mapping the matching information to the 3D information.

Referring to FIG. 7, the robot 100 may set the electronic device 200, whose location information is unknown, to unknown (1a). For example, the first to fourth electronic devices 200a, 200b, 200c, and 200d shown in Figure 5 may be lights. The robot 100 may know the identification information of the first to fourth lights, but may not know the location information. Identification information may include type information of the electronic device. Accordingly, the robot 100 can set the first to fourth lights to unknown lighting (1a). The robot 100 can confirm its operation by transmitting an operation confirmation signal to the first to fourth lights, respectively. The robot 100 may check the operation of each of the first to fourth lights and generate matching information that matches the identification information and location information of each of the first to fourth lights. And, the robot 100 can update the 3D information 1b by mapping the generated matching information to the 3D information 1b. The robot 100 may transmit the updated 3D information 1b to the terminal device 400. The terminal device 400 may display a control UI including images of each space and deployed electronic devices based on the updated 3D information 1b.

Once the robot 100 completes the identification of the electronic device located in the room 10, it can move to another space and perform a similar process.

The robot 100 can simultaneously check the operation of a plurality of electronic devices.

FIG. 8 is a diagram illustrating a process for simultaneously identifying a plurality of electronic devices according to an embodiment of the present disclosure.

Referring to FIG. 8, a plurality of electronic devices 200a, 200b, 200c, 200d, 200e, and 200f are shown. As described above, the robot 100 knows identification information including the types of the plurality of electronic devices 200a, 200b, 200c, 200d, 200e, and 200f. For example, as shown in FIG. 8, the electronic device may be a light, and the robot 100 may set each light to unknown lighting (3a). The robot 100 can simultaneously transmit an operation confirmation signal corresponding to each light to a plurality of lights. Operation confirmation signals simultaneously transmitted to a plurality of lights may be distinct operation confirmation signals. For example, each distinct operation confirmation signal may be a signal with distinct colors, flashing patterns, gradations, frequencies, sound output patterns, etc.

For example, as shown in FIG. 8, the robot 100 transmits a signal of a first blinking pattern to the first electronic device 200a and transmits a signal of a second blinking pattern to the fifth electronic device 200e. Can be transmitted. The robot 100 may transmit a signal of a first color to the third electronic device 200c and a signal of a second color to the fourth electronic device 200d. The robot 100 may transmit a gradient signal to the sixth electronic device 200f. Each electronic device 200a, 200b, 200c, 200d, 200e, and 200f may perform an operation corresponding to the received signal. The robot 100 can simultaneously capture images of a plurality of operating electronic devices 200a, 200b, 200c, 200d, 200e, and 200f using a camera. The robot 100 can simultaneously identify a plurality of electronic devices 200a, 200b, 200c, 200d, 200e, and 200f by identifying the operations of each electronic device 200a, 200b, 200c, 200d, 200e, and 200f. Additionally, the robot 100 can simultaneously identify the locations of a plurality of electronic devices 200a, 200b, 200c, 200d, 200e, and 200f based on the acquired images. The robot 100 may simultaneously generate matching information 3b that matches the identification information and location information of the plurality of electronic devices 200a, 200b, 200c, 200d, 200e, and 200f. The robot 100 may update 3D information based on the generated matching information 3b. And, the robot 100 can transmit the updated 3D information 3b to the terminal device 400. The terminal device 400 may display a control UI including images of each space and deployed electronic devices based on the updated 3D information 3b.

A plurality of electronic devices (200a, 200b, 200c, 200d, 200e, 200f) simultaneously receive distinct operation confirmation signals from the robot 100, so they may be devices connected to the network, and the device connected to the network may be an IoT device. .

In FIG. 8 , an example in which the electronic device is a light is shown, and an example in which the operation confirmation signal is distinguished by color, flashing pattern, and gradation has been described. If the electronic device is an audio device, the robot 100 may transmit an operation confirmation signal distinguished by an on/off pattern, frequency, and sound output pattern to each audio device.

Figure 9 is a timing diagram explaining the operation of a robot system that identifies an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 9, the robot 100 may receive information on the electronic device 200 from the server 300 (S110). For example, when the electronic device 200 is installed in a specific space, the electronic device 200 may transmit identification information to the server 300. The server 300 may transmit the received identification information of the electronic device 200 to the robot 100. For example, identification information may include type, name, model name, control-related information, etc.

The robot 100 can search for electronic devices (S120). The robot 100 can search for the electronic device 200 located in a specific space using a camera. When the robot 100 discovers the electronic device 200, identification information can be obtained. For example, if the discovered electronic device 100 is an IoT device, the robot 100 may obtain identification information corresponding to the type of the discovered electronic device 100 from the received identification information. Alternatively, if the discovered electronic device is an IR device, the robot 100 may capture an image of the electronic device 200 and transmit the captured image to the server 300. And, the robot 100 can obtain identification information of the electronic device 200 from the server 300.

The robot 100 may transmit an operation confirmation signal to the electronic device 200 (S130). The operation confirmation signal may be a signal corresponding to the type of the electronic device 200. For example, if the type of electronic device 200 is a light output device, the operation confirmation signal may be a control signal that outputs light. If the type of electronic device 200 is a sound output device, the operation confirmation signal may be a control signal that outputs sound. If the type of electronic device 200 is a communication device, the operation confirmation signal may be a control signal that outputs a response signal. Alternatively, the operation confirmation signal may include a signal that turns on/off the electronic device 200.

If the electronic device 200 is an IoT device and there are a plurality of identical electronic devices, the robot 100 may simultaneously transmit a distinct operation confirmation signal to each of the plurality of electronic devices 200. The robot 100 can simultaneously identify the operations of a plurality of electronic devices 200. In this specification, ‘simultaneous’ may include not only the same time but also a sequential series of times that can be considered the same time.

The electronic device 200 operates according to the received operation confirmation signal (S140), and the robot 100 can identify the electronic device (S150). For example, if the electronic device 200 is a light output device, the robot 100 can use a camera to check operations such as light output and blinking of the electronic device 200. If the electronic device 200 is a sound output device, the robot 100 uses the operation image of the electronic device 200 captured using a camera and the speaker to determine the sound output, sound output pattern, frequency, and The operation of the electronic device 200 can be confirmed based on information such as volume. If the electronic device 200 is a communication device, the robot can check the operation of the electronic device 200 based on the operation image of the electronic device 200 captured using a camera and information such as frequency, signal strength, and response message. there is.

When the operation of the electronic device 200 according to the operation confirmation signal is identified, the robot 100 may identify location information of the electronic device 200 within a specific space.

The robot 100 synchronizes identification information and location information with the electronic device (S160), and the robot 100 can map 3D information (S170). The robot 100 may generate matching information that matches the identification information and location information of the electronic device 200. Additionally, the robot 100 can update 3D information by mapping the electronic device 200 to a specific space.

Additionally, the robot 100 can transmit matching information or 3D information to the server 300, and the server 300 can update the stored information in the electronic device 200 (S180). Additionally, the robot 10 may transmit updated 3D information to a terminal device by mapping the matching information.

FIG. 10 is a diagram illustrating a process for identifying a moving electronic device according to an embodiment of the present disclosure.

Referring to FIG. 10, an example of the electronic device 200 moving from the first space to the second space is shown. For example, the first space may be the room 10 and the second space may be the living room 20. Additionally, the electronic device 200 may be a device in which matching information is generated by the robot 100 described in the above example and updated to 3D information.

The robot 100 can take pictures of the space while exploring the space. When a new electronic device 200 is identified in the acquired image, the robot 100 may transmit an existing operation confirmation signal corresponding to the type of the identified electronic device to the new electronic device 200. As described above, when an electronic device is placed in a space, the robot 100 may receive identification information of the placed electronic device from the server 300. If the robot 100 discovers a new electronic device in a specific space but fails to receive identification information from the server 300, there is a possibility that the new electronic device is an electronic device that has moved from another space. Accordingly, when the robot 100 identifies the operation of the new electronic device 200 according to the existing operation confirmation signal, it can identify the new electronic device 200 as the existing electronic device whose location has been changed. The robot 100 includes matching information that matches the identification information and existing location information of the existing electronic device. Accordingly, the robot 100 can identify the new location information of the electronic device 200 and modify the existing matching information. Additionally, the robot 100 can update 3D information that modifies the location information of the electronic device 200.

When the location of the electronic device 200 changes, the performance of the electronic device 200 may vary depending on the influence of the surrounding environment. Alternatively, as a new object is placed around the electronic device 200, the performance of the electronic device 200 may be affected by the placed object. Accordingly, the robot 100 can test the performance of the electronic device 200 when the location of the electronic device changes or when a new object is placed.

11 and 12 are diagrams illustrating a process for confirming the influence of surrounding objects according to an embodiment of the present disclosure.

Referring to FIG. 11, a diagram showing a new object 600a arranged in a room 10 is shown. For example, the second electronic device 200b may be a lamp, and a bookshelf may be placed under the lamp. A bookshelf placed under a light can affect the brightness of the room caused by the light. The robot 100 may identify a new object 600a from an image acquired through a camera. The robot 100 may transmit a test signal corresponding to the type of the second electronic device 200b to the second electronic device 200b. For example, the test signal may be the same signal as the operation confirmation signal. Alternatively, the test signal may be a signal that turns on the electronic device. The second electronic device 200b may perform an operation according to the received test signal. The robot 100 may identify the operational performance of the second electronic device 200b according to the test signal. For example, if the second electronic device 200b is a lamp, illuminance, etc. can be measured. If the second electronic device 200b is a speaker, sound volume, etc. can be measured. If the second electronic device 200b is a communication device, signal strength and data loss rate can be measured.

The robot 100 may identify the operational performance of the electronic device by transmitting a test signal to the electronic device when the electronic device is first deployed. Accordingly, if the second electronic device 200b is an existing electronic device, the robot 100 may include information about the operating performance of the second electronic device 200b. The robot 100 may compare the measured operation performance of the second electronic device 200b with the stored operation performance. If the operational performance of the second electronic device 200b is outside a preset range, the robot 100 may generate information 5 related to the operational performance. Additionally, the robot 100 may transmit information 5 related to the generated motion performance to the terminal device 400 . The terminal device 400 may output information related to the received operational performance.

Referring to FIG. 12, a diagram showing a new object 600b placed on the floor of the room 10 is shown. The new object 600b shown in FIG. 12 is placed on the floor. The object 600b placed on the floor may affect the robot cleaner 200c when it cleans.

The robot 100 may identify a new object 600b located on the floor from the acquired image. The robot 100 may transmit floor information 7 among the 3D information related to the location of the new object 600b to the robot vacuum cleaner 200c among the electronic devices. The robot vacuum cleaner 200c can explore the floor and create a cleaning map. If a new object 600b is placed, the robot cleaner 200c must discover the new object 600b and update the existing cleaning map. Therefore, the robot cleaner 200c must use time to identify objects and update the cleaning map. However, when the robot 100 discovers the object 600b and transmits floor information 7 among the three-dimensional information on which the object 600b is placed to the robot cleaner 200c, the robot cleaner 200c quickly cleans. The map can be updated.

FIG. 13 is a timing diagram explaining the operation of a robot system that identifies a moving electronic device according to an embodiment of the present disclosure.

Referring to FIG. 13, the robot 100 can discover a new electronic device 200 (S205). The robot 100 can generate three-dimensional information by mapping the matching information that matches the identification information and location information of the installed electronic device to space. Afterwards, the robot 100 may discover a new electronic device 200 in addition to the existing electronic device during space exploration. If the robot 100 does not receive the identification information of the electronic device from the server 300, there is a possibility that the discovered electronic device 200 is an existing electronic device.

The robot 100 may transmit an existing operation confirmation signal to the discovered electronic device 200 (S210). The existing operation confirmation signal is a signal corresponding to the type of the discovered electronic device 200. If the discovered electronic device 200 is an electronic device that has moved from another space, it may operate according to an existing operation confirmation signal (S215).

The robot 100 can check the operation of the electronic device 200 and identify it as an existing electronic device (S220). If the electronic device 200 is an existing electronic device, only the location information among the matching information or 3D information may be different. Accordingly, the robot 100 can obtain location information where the electronic device is placed and modify the location information included in the existing information with the acquired location information. That is, the robot 100 can update 3D information related to the electronic device 200 (S225).

Meanwhile, the surrounding environment changes as the electronic device 200 moves, and the changed surrounding environment may affect the operating performance of the electronic device 200. Accordingly, the robot 100 may transmit a test signal to the electronic device 200 to check the operating performance of the electronic device 200 (S230). For example, the test signal may be a signal that controls light emission, sound output, data transmission and reception, etc. The electronic device 200 may perform an operation according to the received test signal (S235).

The robot 100 can check the operation of the electronic device 200 and identify the operating performance of the electronic device 200 (S240). The robot 100 may compare the identified operation performance with the operation performance of an existing stored electronic device. The robot 100 may generate information about operation performance (S245). And, if the identified motion performance is outside the preset range, the robot 100 may transmit information related to the motion performance to the terminal device 400.

Figure 14 is a diagram explaining a robot that performs a user command according to an embodiment of the present disclosure.

Referring to FIG. 14, a user 90 and a robot 100 located near the user are shown. The robot 100 can recognize the voice of the user 90 and perform control operations. For example, the first electronic device 200a and the second electronic device 200b may be lights, the first electronic device 200a may be light number 1, and the second electronic device 200b may be light number 2. When the user 90 says, “Turn on light number 2,” the robot 100 can turn on the second electronic device 200b, which is light number 2. That is, the robot 100 can recognize the user's utterance and transmit a turn-on control signal to the second electronic device 200b.

Alternatively, the user 90 may utter “Turn this on,” without referring to a specific electronic device. At this time, the robot 100 located near the user 90 can acquire the user's image. The robot 100 can recognize the user 90's utterance and identify user-related information from the acquired image. For example, user-related information may include the user's location, gaze direction, pointing direction, etc. As shown in FIG. 14, the robot 100 identifies the user's gaze direction and the user's pointing direction from the acquired user's image, and identifies the second electronic device (200b) where “this” uttered by the user is the number 2 light. ) can be identified. The robot 100 transmits a turn-on control signal to the second electronic device 200b, and the second electronic device 200b may perform a turn-on operation according to the received control signal.

Alternatively, the user 90 may control the electronic device 200 using the terminal device 400.

15 and 16 are diagrams illustrating a terminal device executing a user command according to an embodiment of the present disclosure.

Referring to FIG. 15, map data of a specific space is shown. The robot 100 may generate map data 50 based on navigation or data received from the server 300. Map data 50 may include each area. For example, as shown in FIG. 15, the map data 50 may display areas such as Bathroom 1, Room 1, Kitchen, Living room, Room 2, Bathroom 2, and Room 3. The user 90 can select an area where an electronic device for control is located. For example, user 90 may select Room 3.

The terminal device 400 can display 3D information on electronic devices placed in Room 3.

Referring to FIG. 16, three-dimensional information 51 where the electronic devices 200a, 200b, 200c, 200d, and 200e in Room 3 are located is shown. As described above, the robot 100 can identify the spatial location and coordinates of the wall, floor, and ceiling as well as the area where the electronic device is located. That is, the robot 100 can identify the plane and coordinates where the first to fourth electronic devices 200a, 200b, 200c, and 200d are located. Additionally, the robot 100 can identify the object 600c, the surface on which the object 600c is located, and its coordinates. And, the robot 100 can identify the location of the fifth electronic device 200e.

For example, the robot 100 has a position of the first electronic device 200a at the ceiling of coordinates (3, 3), a position of the second electronic device 200b at the ceiling of coordinates (3, 5), and a third electronic device 200b. The location of the device is the ceiling at coordinates (3, 7), the location of the fourth electronic device 200d is the ceiling at coordinates (5, 2) to (5, 8), and the location of the object 600c is at coordinates (1, 8) ), the location of the fifth electronic device 200e can be identified as the top surface of the object 600c. In addition, the robot 100 generates matching information by matching (synchronizing) the identification information and location information of each electronic device (200a, 200b, 200c, 200d, 200e), and three-dimensional information 51 that maps the matching information. can be created.

The robot 100 can transmit the generated 3D information 51 to the terminal device 400. And, as shown in FIG. 16, the terminal device 400 may display a control UI based on the received 3D information 51. The user 90 may select the first electronic device 200a. The terminal device 400 may transmit a control signal to the first electronic device 200a according to the user's input command. Alternatively, the terminal device 400 may control the first electronic device 200a through the robot 100.

As the terminal device 400 displays a control UI including 3D information 51, the user can intuitively control the electronic device.

So far, various embodiments of the robot system 1000 have been described. Below, the control method of the robot 100 will be described.

Figure 17 is a flowchart explaining a method of controlling a robot according to an embodiment of the present disclosure.

Referring to FIG. 17, the robot acquires identification information of the identified electronic device (S310). For example, while a robot is traveling in a specific space, an electronic device can identify it from an image acquired through a camera and obtain identification information of the electronic device. Electronic devices may include network-connected IoT devices or non-network-connected IR devices. If the electronic device is an IR device, the robot can transmit a captured image of the electronic device to the server and obtain identification information of the electronic device corresponding to the image from the server.

The robot can transmit an operation confirmation signal to the electronic device (S320). The robot may transmit an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device. For example, if the type of electronic device is a light output device, the operation confirmation signal may be a signal that outputs light. If the type of electronic device is a sound output device, the operation confirmation signal may be a signal that outputs sound. If the type of electronic device is a communication device, the operation confirmation signal may be a signal that outputs a response signal.

Alternatively, when a plurality of identical electronic devices are identified, the robot can simultaneously transmit distinct operation confirmation signals to each of the plurality of IoT devices and simultaneously identify the operations of the plurality of IoT devices. For example, a distinct operation confirmation signal may be a signal that is distinguished using color, flashing pattern, gradation, frequency, sound output pattern, etc.

The robot can identify location information of electronic devices in space (S330). Location information may include surface and coordinate information in three-dimensional space where the electronic device is located. For example, the robot may identify location information of the electronic device, such as the ceiling of the first electronic device at coordinates (3, 5). Coordinate information can be calculated individually for each side. For example, the position of the ceiling (1, 1) is calculated based on the first reference point set in the virtual grid of the ceiling, and the position of the right wall (1, 1) is calculated based on the second reference point set in the virtual grid of the right wall. It can be calculated based on .

Additionally, the robot can identify an object and identify location information of an electronic device located on the object. For example, the robot can identify a table and identify location information about the second electronic device located on the table, such as the top of the table.

The robot can update the stored 3D information by mapping the electronic device to space (S340). The robot can generate matching information that matches the identification information and location information of the electronic device. In addition, 3D information can be created (updated) by mapping the generated matching information to a specific space. The generated (updated) 3D information may be transmitted to the terminal device and/or server.

Additionally, when a new electronic device is identified in an image acquired through a camera, the robot may transmit an existing operation confirmation signal corresponding to the type of the identified electronic device to the new electronic device. When the operation of the new electronic device according to the existing operation confirmation signal is identified, the robot can identify the new electronic device as the existing electronic device whose location has changed. The robot can update 3D information that modifies the location information of existing electronic devices.

Additionally, when a new object located on the floor is identified among the 3D information in an image acquired through a camera, the robot can update the cleaning map by transmitting information related to the location of the new object to the robot cleaner.

Additionally, when a new object is identified in an image acquired through a camera, the robot can test the operational performance of the electronic device by transmitting a test signal corresponding to the type of the electronic device to the electronic device. If the operational performance of the electronic device is outside a preset range, the robot can transmit information related to the operational performance to the terminal device.

The robot can receive the user's voice commands near the user. The robot can identify user-related information from the user's image acquired through a camera and identify the electronic device to be controlled based on the identified user-related information. For example, user-related information may include the user's location, gaze direction, pointing direction, etc.

The various examples described above are merely divided for convenience of explanation and are not separate embodiments. For example, if a robot finds an electronic device, it can identify whether it is a new or existing electronic device. If the robot is a new electronic device, it can perform the process of matching identification information and location information and mapping it to map data. If it is an existing electronic device, a process of updating existing location information can be performed. Additionally, when the robot detects a new electronic device, a change in location, or a change in the surrounding environment, it can perform a process to test the operational performance of the electronic device.

The robot control method according to the various embodiments described above may be provided as a computer program product. A computer program product may include the S/W program itself or a non-transitory computer readable medium in which the S/W program is stored.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above may be stored and provided on non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

In addition, although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure pertains without departing from the gist of the present disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present disclosure.

1000: Robot system 100: Robot
110: camera 120: communication interface
130: memory 140: processor
200: electronic device
300: Server 400: Terminal device

Claims (20)

In robots,
camera;
communication interface;
A memory storing three-dimensional information corresponding to a specific space; and
Contains one or more processors;
The one or more processors:
If an electronic device is identified in an image obtained through the camera while the robot is traveling in the specific space, obtain identification information of the identified electronic device,
Transmitting an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device through the communication interface,
When the operation of the electronic device according to the operation confirmation signal is identified, location information of the electronic device within the specific space is identified,
A robot that updates the three-dimensional information by mapping the electronic device to the specific space based on the identification information of the electronic device and the location information of the electronic device. According to paragraph 1,
The one or more processors:
A robot that transmits the three-dimensional information to which the identification information of the electronic device and the location information of the electronic device are mapped to a terminal device through the communication interface. According to claim 1 or 2,
The one or more processors:
If the type of the electronic device is an optical output device, transmitting an operation confirmation signal for outputting light to the optical output device,
If the type of the electronic device is a sound output device, an operation confirmation signal for outputting sound is transmitted to the sound output device,
If the type of the electronic device is a communication device, the robot controls the communication interface to transmit an operation confirmation signal that outputs a response signal to the communication device. According to any one of claims 1 to 3,
Further including a microphone;
The one or more processors:
When the robot is located near the user and receives the user's voice command through the microphone, user-related information including at least one of the user's location, gaze direction, and pointing direction in the user's image acquired through the camera. A robot that identifies information and identifies a controlled electronic device based on the identified user-related information. According to any one of claims 1 to 4,
The one or more processors:
When a new electronic device is identified in the image acquired through the camera, an existing operation confirmation signal corresponding to the type of the identified electronic device is transmitted to the new electronic device, and the new electronic device is activated according to the existing operation confirmation signal. When the motion is identified, the robot identifies the new electronic device as an existing electronic device whose location has changed and updates the 3D information with the modified location information of the existing electronic device. According to any one of claims 1 to 5,
The one or more processors:
When a new object located on the floor is identified among the 3D information in the image acquired through the camera, information related to the location of the new object is transmitted to the robot cleaner among the electronic devices through the communication interface to perform cleaning of the robot cleaner. A robot that updates the map. According to any one of claims 1 to 6,
The electronic device is,
A robot including at least one of an Internet of Thing (IoT) device and an Infra-Red (IR) device. In clause 7,
The one or more processors:
When the electronic device is an IR device, a robot transmits an image captured of the electronic device to a server through the communication interface and obtains identification information of the electronic device corresponding to the image from the server. According to clause 7 or 8,
The one or more processors:
When a plurality of identical IoT devices are identified, a distinct operation confirmation signal is simultaneously transmitted to each of the plurality of IoT devices, and the operation of the plurality of IoT devices is simultaneously identified to simultaneously identify the location information of the plurality of IoT devices. robot. According to clause 9,
The distinct operation confirmation signal is,
A robot that is distinguished by at least one of color, flashing pattern, gradient, frequency, and sound output pattern. According to any one of claims 1 to 10,
The one or more processors:
When a new object is identified in the image acquired through the camera, a test signal corresponding to the type of the electronic device is transmitted to the electronic device through the communication interface, and the operating performance of the electronic device according to the test signal is identified. and controlling the communication interface to transmit information related to the operating performance to the terminal device when the operating performance is outside a preset range. In the robot control method,
When an electronic device is identified in an image obtained while the robot is traveling in a specific space, obtaining identification information of the identified electronic device;
transmitting an operation confirmation signal corresponding to the type of the identified electronic device to the electronic device;
When the operation of the electronic device according to the operation confirmation signal is identified, identifying location information of the electronic device within the specific space; and
A method of controlling a robot comprising: updating stored 3D information by mapping the electronic device to the specific space based on the identification information of the electronic device and the location information of the electronic device. According to clause 12,
A method of controlling a robot further comprising transmitting the 3D information to which the identification information of the electronic device and the location information of the electronic device are mapped to a terminal device. According to claim 12 or 13,
The step of transmitting the operation confirmation signal to the electronic device includes:
If the type of the electronic device is an optical output device, transmitting an operation confirmation signal for outputting light to the optical output device,
If the type of the electronic device is a sound output device, an operation confirmation signal for outputting sound is transmitted to the sound output device,
If the type of the electronic device is a communication device, an operation confirmation signal that outputs a response signal is transmitted to the communication device. According to any one of claims 12 to 14,
When the robot is located near the user and receives the user's voice command, identifying user-related information including at least one of the user's location, gaze direction, and pointing direction from the obtained user's image; and
A robot control method further comprising: identifying a control target electronic device based on the identified user-related information. According to any one of claims 12 to 15,
When a new electronic device is identified in the acquired image, transmitting an existing operation confirmation signal corresponding to the type of the identified electronic device to the new electronic device;
When the operation of the new electronic device according to the existing operation confirmation signal is identified, identifying the new electronic device as an existing electronic device whose location has changed; and
A method of controlling a robot comprising: updating the 3D information that modifies the location information of the existing electronic device. According to any one of claims 12 to 16,
When a new object located on the floor is identified among the 3D information in the acquired image, transmitting information related to the location of the new object to the robot cleaner among the electronic devices to update the cleaning map of the robot cleaner. How to control a robot. According to any one of claims 12 to 17,
The electronic device is,
A method of controlling a robot, including at least one of an IoT (Internet of Thing) device and an IR (Infra-Red) device. According to clause 18,
If the electronic device is an IR device, transmitting an image captured of the electronic device to a server; and
Obtaining identification information of the electronic device corresponding to the image from the server. According to claim 18 or 19,
When a plurality of identical IoT devices are identified, simultaneously transmitting distinct operation confirmation signals to each of the plurality of IoT devices; and
Simultaneously identifying the operations of the plurality of IoT devices and simultaneously identifying location information of the plurality of IoT devices.

Images