KR20220084721A - Method and apparatus for detecting abnormality in sensor included in robot - Google Patents

Abstract

The present disclosure relates to a method of detecting an abnormality in a sensor included in a robot using an elevator. A method for detecting an abnormality in a sensor included in the robot includes measuring first data through a sensor included in the robot, requesting second data associated with the elevator, in response to the request for second data, receiving the associated second data; generating a comparison result between the first data and the second data; and determining whether a sensor included in the robot is abnormal, based on the generated comparison result.

Description

Method and device for detecting abnormality of sensor included in robot

The present disclosure relates to a method and a system for detecting an abnormality of a sensor included in a robot, and specifically, by comparing data measured through a sensor included in the robot with data associated with an elevator, whether a sensor included in the robot is abnormal It relates to a method and system for making a decision.

With the development of the robot industry, robots are being used in various fields such as logistics services, delivery services, and care services. These robots must maintain the best sensor condition in order to provide high-quality service to users. However, in the case of a mobile service robot, a sensor with a relatively lower precision than the highest level is used due to the problem of limited cost and battery usage, and information deviation occurs in these sensors over time. In addition, if the robot is operated for a long time, information deviation of the MEMS sensor or deterioration of the semiconductor may occur, and the distance sensor or optical sensor may have foreign substances on the surface or inside of the lens, or the optical semiconductor sensor may be burned by a strong light source. -out can be done.

Even when such a problem occurs, the robot can be driven because it is composed of complementary sensors. However, a sensor malfunction may eventually lead to a decrease in the service quality of the robot. Therefore, in order to maintain the service quality of the robot above a certain level, the manager has the inconvenience of periodically checking the sensor of the robot to confirm that there is no abnormality. In addition, there is a problem that the robot provides the service with the sensor abnormality as it is until the manager checks it.

The present disclosure provides a method for detecting an abnormality in a sensor included in a robot, a computer program stored in a recording medium, and an apparatus (system) for solving the above problems.

The present disclosure may be implemented in various ways including a method, an apparatus (system), or a computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method of detecting an abnormality in a sensor included in a robot using an elevator, performed by at least one processor, includes measuring first data through a sensor included in the robot, the elevator requesting second data associated with the; receiving second data associated with the elevator in response to the request for the second data; generating a comparison result between the first data and the second data; and determining whether a sensor included in the robot is abnormal, based on the generated comparison result.

A computer program stored in a computer-readable recording medium is provided to execute a method for detecting an abnormality in a sensor included in a robot according to an embodiment of the present disclosure in a computer.

An electronic device according to an embodiment of the present disclosure includes at least one processor connected to a communication module, a memory, one or more sensors, and a memory, and configured to execute at least one computer-readable program included in the memory. The at least one program measures the first data through the one or more sensors, requests second data associated with the elevator, and in response to the request for the second data, receives second data associated with the elevator, the first data and instructions for generating a comparison result between the and second data, and determining whether one or more sensors are abnormal based on the generated comparison result.

In various embodiments of the present disclosure, since the detection of abnormality of the sensor can be performed automatically and/or repeatedly, robot maintenance cost can be reduced, and the abnormality of the sensor can be recognized quickly, so the service quality of the robot can always be kept in top condition.

In various embodiments of the present disclosure, it is possible to detect whether the robot's sensor is abnormal during the time the mobile robot boards the elevator and moves, that is, while the robot provides a service, and the robot performs self-calibration (or correction). can do. Accordingly, it is possible to minimize the facilities required for sensor calibration (or correction), and it is possible to detect whether the sensor is abnormal at any time while the robot is providing a service.

In various embodiments of the present disclosure, since the moving distance and direction between floors of the elevator on which the robot is boarded are different each time, it is possible to check more various sensor abnormalities (or errors) than detecting abnormalities in the sensors in a fixed state. That is, the abnormal state of the sensor in various environments or states may be confirmed.

In various embodiments of the present disclosure, an error of a sensor may be detected within a limited time resource according to the moving distance of the elevator.

In various embodiments of the present disclosure, since the environment inside the elevator in which other objects such as people are not boarded can generate spatial information that can be modeled, the abnormality of the optical sensor and the distance detection sensor through image processing learned through the artificial neural network can detect

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals denote like elements, but are not limited thereto.
1 is a diagram illustrating an example in which a robot moves using an elevator according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a configuration in which a robot control system according to an embodiment of the present disclosure is connected to communicate with a plurality of robots.
3 is a block diagram illustrating an internal configuration of a robot and a robot control system according to an embodiment of the present disclosure.
4 is a block diagram illustrating an internal configuration of a processor according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method of detecting an abnormality in a sensor included in a robot using an elevator according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of detecting an abnormality in a sensor included in the robot while the robot is positioned in an elevator according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of detecting an abnormality in an inertial sensor included in a robot according to an embodiment of the present disclosure.
8 is a diagram illustrating an example in which inertial data is measured through an inertial sensor associated with an elevator according to an embodiment of the present disclosure.
9 is a diagram illustrating an example of detecting an abnormality in the inertial sensor included in the robot and calibrating the inertial sensor included in the robot according to an embodiment of the present disclosure.
10 is a flowchart illustrating a method of detecting an abnormality in an optical sensor included in a robot according to an embodiment of the present disclosure.
11 is a diagram illustrating an example of detecting an abnormality of an optical sensor included in a robot according to an embodiment of the present disclosure.
12 is a diagram illustrating an example of extracting a first image feature by inputting optical data measured through an optical sensor associated with a robot to a feature extraction unit according to an embodiment of the present disclosure.
13 is a flowchart illustrating a method of detecting an abnormality of a distance sensor included in a robot according to an embodiment of the present disclosure.
14 is a diagram illustrating an example of detecting an abnormality of a distance detection sensor included in a robot according to an embodiment of the present disclosure.
15 is a diagram illustrating an example of extracting a first shape feature by inputting distance sensing data measured through a distance sensing sensor associated with a robot according to an embodiment of the present disclosure to a shape feature extracting unit.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, the same or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if descriptions regarding components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and the present disclosure provides those skilled in the art with the scope of the invention. It is provided for complete information only.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. Terms used in this specification have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the content throughout the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. In the entire specification, when a part includes a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or at least one of variables. Components and 'modules' or 'units' are the functions provided therein that are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some contexts, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, a 'sensor' may refer to one or more sensors. In addition, a 'sensor included in the robot' may refer to a sensor embedded in, connected to, and/or attached to the robot, and data measured by the sensor may be provided to the robot and/or the robot control system. The sensor data measured in this way can be used to determine whether the robot is abnormal.

1 is a diagram illustrating an example in which a robot 110 moves using an elevator 120 according to an embodiment of the present disclosure. The mobile service robot 110 used in a multi-story building, etc. must be capable of inter-floor movement (ie, vertical movement) in order to provide a service to all or some floors of a multi-story building. To this end, the robot 110 may use an elevator (or elevator) 120 such as a multi-story building. Here, the elevator 120 is a robot-only elevator, in which a person or other object may not ride. Alternatively, the robot 110 may use a general elevator used by humans.

The robot 110 may include various sensors such as an inertial sensor system, an optical sensor system, and a distance sensing sensor system to perform driving and work (eg, moving logistics, etc.). For example, the robot 110 may include a sensor of an inertial sensor system to perform precise driving through inertial modeling of the robot 110 itself. As another example, the robot 110 may include an optical sensor type sensor for image processing and visual SLAM. As another example, the robot 110 may include a sensor of a distance sensor system for detecting and avoiding far/near obstacles. In order to provide stable driving and service of the robot 110, a precise sensor and modeling through this may be required.

In the case of the mobile service robot 110, a relatively low-precision sensor may be used due to a problem of limited cost and battery usage. Information bias over time may occur in these sensors. For example, when the robot 110 is operated for a long time, information deviation of a MEMS (Micro Electro Mechanical Systems) sensor or semiconductor aging may occur. In addition, in the distance sensor or the optical sensor, foreign substances may be formed on the surface or inside of the lens, or the optical semiconductor sensor may be burned out by a strong light source. Even when such a problem occurs, the robot 110 configured with a complementary sensor may be able to run. However, for the quality of the service provided by the robot 110, one or more sensors included in the robot 110 should be frequently checked to ensure that there are no abnormalities or errors in these sensors.

The elevator 120 or a system (eg, elevator control system, etc.) associated with the elevator 120 that the robot 110 can use for inter-floor movement, etc. is equipped with a constant power source, and may include a more precise sensor. can In addition, if there is no object (eg, logistics, robot, person, etc.) riding in the elevator 120, a constant environment (eg, lighting, internal space, etc.) can be always maintained, and in this environment, The data may be stored in advance. The robot 110 compares or compares data (eg, data measured by a sensor associated with the elevator or data stored in advance for the elevator) associated with the elevator 120 with data measured by a sensor included in the robot 110 or By analyzing, it can be determined whether there is an abnormality in the sensor included in the robot 110 .

While the robot 110 is positioned in the elevator 120 , data measured through a sensor included in the robot 110 and data associated with the elevator 120 are compared, and whether the sensor included in the robot 110 is abnormal. can be determined. According to an embodiment, the robot 110 may determine whether a sensor included in the robot 110 in the moving elevator 120 is abnormal. Alternatively or additionally, the robot 110 may determine whether a sensor included in the robot 110 in the stopped elevator 120 is abnormal.

According to an embodiment, the robot 110 may take any action required to correct or repair the abnormality of the sensor according to the determination result of whether the sensor is abnormal. For example, a sensor included in the robot 110 may be calibrated (or corrected) by itself. Alternatively or additionally, the robot 110 may provide information about the abnormality of the sensor to the robot control system. Then, according to the information on the abnormality of the sensor, a person (eg, an administrator) may directly repair and/or calibrate the sensor included in the robot 110 . The robot 110 automatically and repeatedly checks for abnormalities in the sensors included in the robot 110 in the elevator 120, thereby reducing robot maintenance costs and ensuring that the service quality of the robot 110 is always in the best state. can keep

2 is a schematic diagram illustrating a configuration in which the robot control system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of robots 210_1 , 210_2 , and 210_3 . The robot control system 230 may include system(s) for robot control, elevator control, and/or sensor abnormality detection. In one embodiment, the robot control system 230 may store, provide, and execute one or more computer-executable programs (eg, downloadable applications) and data related to robot control, elevator control, and/or sensor abnormality detection, etc. It may include a server device and/or a database, or one or more distributed computing devices and/or a distributed database based on a cloud computing service. The robot control system 230 may provide information corresponding to a command input from a user through a robot application, an elevator application, and/or an artificial intelligence application or perform a corresponding process. For example, the robot control system 230 controls a plurality of robots 210_1, 210_2, 210_3 to provide a service corresponding to a command input from a user through a robot application, an elevator application, and/or an artificial intelligence application. can

The robot control system 230 may communicate with the plurality of robots 210_1 , 210_2 , and 210_3 through the network 220 . Additionally, the robot control system 230 may communicate with an elevator (or a system associated with the elevator, such as an elevator control system) (not shown) via the network 220 . The network 220 may be configured to enable communication between the plurality of robots 210_1 , 210_2 , and 210_3 and the robot control system 230 . Network 220 according to the installation environment, for example, Ethernet (Ethernet), wired home network (Power Line Communication), telephone line communication device and wired networks such as RS-serial communication, mobile communication network, WLAN (Wireless LAN), It may consist of a wireless network such as Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited, and the robot (210_1, 210_2, 210_3) as well as a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 220 may include Short-range wireless communication between the two may also be included.

In FIG. 2 , the robot for professional service 210_1 , the industrial robot 210_2 , and the robot for personal service 210_3 are illustrated as examples of the robot, but is not limited thereto, and the robots 210_1 , 210_2 , and 210_3 are wired and/or It may be any computing device capable of wireless communication and in which a robot application, an artificial intelligence application, etc. can be installed and executed. For example, the robots 210_1, 210_2, and 210_3 are artificial intelligence robots, manufacturing robots, personal service robots, professional service robots, partial robots, humanoid robots, medical robots, household robots, pet robots, military robots, and exploration robots. It may include robots of various purposes and/or forms, such as robots, cleaning robots, mobile robots, caring robots, social robots, and the like. In addition, in FIG. 2 , the three robots 210_1 , 210_2 , and 210_3 are illustrated as communicating with the robot control system 230 through the network 220 , but the present invention is not limited thereto, and a different number of robots is connected to the network 220 . It may be configured to communicate with the robot control system 230 through.

In one embodiment, the robots 210_1 , 210_2 , and 210_3 transmit a data request associated with the elevator to the robot control system 230 through the network 220 , and receive data associated with the elevator from the robot control system 230 . can In an embodiment, the robots 210_1 , 210_2 , and 210_3 may provide information about an abnormality of a sensor to the robot control system 230 through the network 220 .

3 is a block diagram showing the internal configuration of the robot 210 and the robot control system 230 according to an embodiment of the present disclosure. The robot 210 may refer to any computing device capable of executing a sensor abnormality detection application, a user service application, a robot application and/or an artificial intelligence application, and capable of wired/wireless communication, for example, as shown in FIG. It may include a robot for professional service 210_1 , an industrial robot 210_2 , and a robot for personal service 210_3 . As shown, the robot 210 may include a memory 312 , a processor 314 , a communication module 316 , and an input/output interface 318 . Similarly, the robot control system 230 may include a memory 332 , a processor 334 , a communication module 336 , and an input/output interface 338 . As shown in FIG. 3 , the robot 210 and the robot control system 230 may be configured to communicate information and/or data via the network 220 using the respective communication modules 316 and 336 . can In addition, the input/output device 320 may be configured to input information and/or data to the robot 210 or output information and/or data generated by the robot 210 through the input/output interface 318 .

The memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-volatile mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSDs), flash memory, and the like. (permanent mass storage device) may be included. In another embodiment, a non-volatile mass storage device such as a ROM, SSD, flash memory, disk drive, etc. may be included in the robot 210 and/or the robot control system 230 as a separate permanent storage device separate from the memory. . In addition, the memories 312 and 332 include an operating system and at least one program code (eg, a sensor abnormality detection application installed and driven in the robot 210 , a user service application, a robot application and/or an artificial intelligence application) code) can be stored.

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332 . The separate computer-readable recording medium may include a recording medium directly connectable to the robot 210 and the robot control system 230, for example, a floppy drive, disk, tape, DVD/CD-ROM. It may include a computer-readable recording medium such as a drive or a memory card. As another example, the software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into the memories 312 and 332 based on a computer program installed by files provided through the network 220 by developers or a file distribution system that distributes installation files of applications. can be

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 314 , 334 by the memory 312 , 332 or the communication module 316 , 336 . For example, the processors 314 and 334 may be configured to execute received instructions according to program code stored in a recording device, such as the memories 312 and 332 .

The communication modules 316 and 336 may provide a configuration or function for the robot 210 and the robot control system 230 to communicate with each other via the network 220, and the robot 210 and/or the robot control system ( 230) may provide configuration or functionality for communicating with other robots and/or other systems (eg, separate cloud systems, etc.). For example, a request or data generated by the processor 314 of the robot 210 according to a program code stored in a recording device such as the memory 312 (eg, a data request associated with an elevator, information on abnormality of a sensor, etc.) ) may be transmitted to the robot control system 230 through the network 220 under the control of the communication module 316 . Conversely, a control signal or command provided under the control of the processor 334 of the robot control system 230 is transmitted to the robot through the communication module 316 of the robot 210 via the communication module 336 and the network 220 . may be received at 210 . For example, the robot 210 may receive data associated with an elevator from the robot control system 230 through the communication module 316 .

The input/output interface 318 may be a means for interfacing with the input/output device 320 . As an example, the input device may include an accelerometer, an inertial sensor such as a gyroscope (i.e., an angular velocity sensor), an optical camera, an optical sensor such as an IR camera, and a distance sensing sensor such as a ToF, LiDAR sensor, or Depth camera. . Additionally, the input device may include a device such as a camera including an audio sensor and/or an image sensor, a keyboard, a microphone, a mouse, and the like, and the output device may include a device such as a display, a speaker, a haptic feedback device, etc. . As another example, the input/output interface 318 may be a means for an interface with a device in which a configuration or function for performing input and output, such as a touch screen, is integrated into one. For example, when the processor 314 of the robot 210 processes the command of the computer program loaded in the memory 312, the service screen configured by using the information and/or data provided by the robot control system 230 and the like may be displayed on the display via the input/output interface 318 . In FIG. 3 , the input/output device 320 is illustrated not to be included in the robot 210 , but the present invention is not limited thereto, and may be configured as a single device with the robot 210 . In addition, the input/output interface 338 of the robot control system 230 is connected to the robot control system 230 or means for an interface with a device (not shown) for input or output that the robot control system 230 may include. can be In FIG. 3, the input/output interfaces 318 and 338 are illustrated as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. have.

The robot 210 and the robot control system 230 may include more components than those of FIG. 3 . However, there is no need to clearly show most of the prior art components. According to an embodiment, the robot 210 may be implemented to include at least a portion of the above-described input/output device 320 . In addition, the robot 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, the robot 210 is implemented so that various components such as a camera module, various physical buttons, buttons using a touch panel, input/output ports, a vibrator for vibration, and wheels for movement are further included in the robot 210 . can be

According to an embodiment, the processor 314 of the robot 210 may be configured to operate a sensor abnormality detection application, a user service application, a robot application, and/or an artificial intelligence application. In this case, the code associated with the corresponding application and/or program may be loaded into the memory 312 of the robot 210 . While the application and/or program is being operated, the processor 314 of the robot 210 receives information and/or data provided from the input/output device 320 through the input/output interface 318 or through the communication module 316 to the robot Information and/or data may be received from the control system 230 , and the received information and/or data may be processed and stored in the memory 312 . In addition, such information and/or data may be provided to the robot control system 230 through the communication module 316 .

While the program for the sensor abnormal detection application is being operated, the processor 314 is inputted through an input device such as a touch screen connected to the input/output interface 318, a keyboard, a camera including an audio sensor and/or an image sensor, a microphone, or the like. It is possible to receive the selected text, image, video, voice, etc., and to store the received text, image, video and/or voice in the memory 312 or through the communication module 316 and the network 220 to the robot control system (230) may be provided. Alternatively or additionally, while a program for a sensor anomaly detection application or the like is being operated, the processor 314 may include an acceleration sensor, an inertial sensor such as a gyroscope (ie, an angular velocity sensor), an optical sensor such as an optical camera, an IR camera, and the like. / or may receive input or selected text, image, video, voice, etc. through a distance detection sensor such as ToF, LiDAR sensor, depth camera, etc., and store the received text, image, video and/or voice in memory 312 It can be stored in or provided to the robot control system 230 through the communication module 316 and the network 220 .

In an embodiment, the processor 314 may measure the first data through an input device (eg, a sensor included in the robot). In an embodiment, the processor 314 may request the robot control system 230 for second data associated with the elevator through the network 220 and the communication module 316 . In response to the request for the second data, the processor 314 may receive the second data associated with the elevator from the robot control system 230 through the network 220 and the communication module 316 . In an embodiment, the processor 314 may generate a comparison result between the first data and the second data, and based on the generated comparison result, determine whether a sensor included in the robot is abnormal.

The processor 334 of the robot control system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of robots and/or a plurality of external systems, including the robot 310 . . The information and/or data processed by the processor 334 may be provided to the robot 310 via the communication module 336 and the network 220 . In an embodiment, the processor 334 of the robot control system 230 may receive a request for information related to the elevator from the robot 310 through the communication module 336 and the network 220 . The processor 334 of the robot control system 330 includes information processed through an output device such as a display output capable device (eg, a touch screen, a display, etc.) of the robot 310, a voice output capable device (eg, a speaker), and / or may be configured to output data.

In an embodiment, the processor 334 of the robot control system 230 may receive data associated with the elevator from the elevator (or a system associated with the elevator) through the communication module 336 and the network 220 . For example, when the processor 334 receives a request for data associated with an elevator from the robot 210, the processor 334 transmits the request to the elevator (or system associated with the elevator), and the data associated with the elevator from the elevator (or system associated with the elevator). can receive The data associated with the elevator thus received may be provided to the robot 310 through the communication module 336 and the network 220 . In one embodiment, the processor 334 of the robot control system 230 may receive information about the abnormality of the sensor from the robot 310 through the communication module 336 and the network 220 .

4 is a block diagram illustrating an internal configuration of a processor 314 according to an embodiment of the present disclosure. In an embodiment, the processor 314 may include a data measuring unit 410 , a data receiving unit 420 , a data comparing unit 430 , a sensor abnormality determining unit 440 , and a sensor correcting unit 450 . The processor 314 may determine whether there is an object that is an obstacle inside the elevator. For example, the data measurement unit 410 of the processor 314 may determine whether there is an obstacle in the elevator based on data input through a sensor. Alternatively, the data receiving unit 420 of the processor may receive real-time information on the inside of the elevator to determine whether there is an object obstructing the inside of the elevator. Alternatively, the object determining unit (not shown) of the processor receives data from the data measuring unit 410 and/or the data receiving unit 420 (eg, data measured through a sensor or real-time information about the inside of the elevator) ), it is possible to determine whether there is an object that is an obstacle inside the elevator.

The data measuring unit 410 may measure data through a sensor included in the robot. In an embodiment, the data measuring unit 410 may measure the first inertial data through an inertial sensor included in the robot. In another embodiment, the data measuring unit 410 may measure the first optical data through an optical sensor included in the robot. In another embodiment, the data measuring unit 410 may measure the first distance sensing data through a distance sensing sensor included in the robot.

The data receiver 420 may request data associated with the elevator, and in response to the request, receive data associated with the elevator. In an embodiment, the data receiver 420 may receive second inertial data measured through an inertial sensor associated with the elevator. In another embodiment, the data receiver 420 may receive data stored in advance for the inside of the elevator.

The data comparator 430 may receive data measured through a sensor included in the robot from the data measurement unit 410 . Additionally, the data comparator 430 may receive data associated with the elevator from the data receiver 420 . Thereafter, the data comparator 430 may generate a comparison result between data measured through a sensor included in the robot and data associated with the elevator. In an embodiment, the data comparator 430 may include first inertial data (ie, inertial data measured through an inertial sensor included in the robot) and second inertial data (ie, inertia measured through an inertial sensor associated with the elevator). data) can be determined. For example, the data comparator 430 may calculate the covariance between the first inertia data and the second inertia data by using the state vector-based prediction model.

In another embodiment, the data comparator 430 may generate a comparison result using the first optical data (ie, optical data measured through an optical sensor included in the robot) and pre-stored data about the inside of the elevator. . For example, when it is determined that there is no obstacle in the elevator, the data comparator 430 may generate a comparison result using the first optical data and pre-stored data about the inside of the elevator. For example, the data comparator 430 extracts a first image feature from the first optical data, and compares the extracted first image feature and the second image feature included in the pre-stored data for the inside of the elevator. A comparison result can be generated based on the difference.

In another embodiment, the data comparator 430 compares the result using the first distance sensing data (that is, distance sensing data measured through a distance sensing sensor included in the robot) and pre-stored data about the inside of the elevator. can create For example, when it is determined that there is no obstacle in the elevator, the data comparator 430 may generate a comparison result using the first distance sensing data and pre-stored data about the inside of the elevator. For example, the data comparison unit 430 generates a first shape feature based on the first distance sensing data, and compares the generated first shape feature and the second shape feature included in the previously stored data for the inside of the elevator. similarity can be determined. That is, the data comparator 430 may determine the similarity between the 3D data included in the first shape feature and the 3D reference data included in the second shape feature. Here, the 3D data included in the first shape feature may be generated by inputting the first distance sensing data to the 3D reconstruction artificial neural network model. In addition, the 3D reference data included in the second shape feature may be generated by inputting some or all of pre-stored data about the inside of the elevator to the 3D reconstruction artificial neural network model.

The sensor abnormality determination unit 440 may receive the comparison result generated from the data comparison unit 430 and determine whether a sensor included in the robot is abnormal. In an embodiment, the sensor abnormality determining unit 440 may determine whether an inertial sensor included in the robot is abnormal based on information on a relationship between the first inertial data and the second inertial data. Additionally, when the covariance between the first inertial data and the second inertial data is equal to or greater than a predetermined threshold, the sensor abnormality determining unit 440 may provide information about the abnormality of the inertial sensor included in the robot to the robot control system.

In another embodiment, when the difference between the first image feature and the second image feature is greater than or equal to a predetermined threshold, the sensor abnormality determining unit 440 may determine that there is an abnormality of the optical sensor included in the robot. have. Additionally, when there is an abnormality of the optical sensor included in the robot, the sensor abnormality determining unit 440 may provide information about the abnormality of the optical sensor included in the robot to the robot control system. Here, the abnormality of the optical sensor included in the robot may include at least one of illuminance abnormality, outline abnormality, and screen abnormality.

In another embodiment, the sensor abnormality determining unit 440 determines whether the distance detection sensor included in the robot is abnormal based on the similarity between the first shape feature and the second shape feature included in the data stored in advance for the inside of the elevator. can decide for example. The sensor abnormality determining unit 440 may determine whether the distance detecting sensor included in the robot is abnormal based on the similarity between the 3D data included in the first shape feature and the 3D reference data included in the second shape feature. . Additionally, when there is an abnormality in the distance sensing sensor included in the robot, the sensor abnormality determining unit 440 may provide information about the abnormality in the distance sensing sensor included in the robot to the robot control system.

When there is an abnormality in the inertial sensor included in the robot, the sensor compensator 450 may perform calibration on the inertial sensor included in the robot. For example, when there is an abnormality in the inertial sensor included in the robot, the sensor compensator 450 may automatically correct (or calibrate) the inertial sensor included in the robot using the calculated covariance.

5 is a flowchart illustrating a method 500 for detecting an abnormality in a sensor included in a robot using an elevator according to an embodiment of the present disclosure. In an embodiment, the method 500 for detecting an abnormality in a sensor included in a robot using an elevator may be performed by a processor (eg, at least one processor of the robot). As shown, the method 500 may be initiated by the processor measuring first data through a sensor included in the robot (S510). Here, the sensor included in the robot may include any sensor required to operate the robot, for example, may include at least one of an inertial sensor, an optical sensor, and a distance sensor. The processor may request second data associated with the elevator (S520). For example, the processor may provide a request for the second data associated with the elevator to the robot control system.

In response to the request for the second data, the processor may receive second data associated with the elevator (S530). For example, the processor may receive second data associated with the elevator from the robot control system. Here, the second data associated with the elevator may include data measured through a sensor associated with the elevator and/or data stored in advance for the interior of the elevator.

Thereafter, the processor may generate a comparison result between the first data and the second data ( S540 ). According to an embodiment, the comparison result between the first data and the second data may include a difference between the first data and the second data, whether they match, a degree of similarity and/or a degree of matching, and the like. The processor may determine whether a sensor included in the robot is abnormal based on the generated comparison result (S550). Here, the sensor abnormality may include abnormalities in sensor internal components (for example, semiconductor elements, etc.), parameter abnormalities, abnormalities due to time shift phenomenon, abnormalities due to sensor contamination, abnormalities due to foreign substances on the lens surface, etc. .

In one embodiment, the processor may determine whether the robot is positioned within the elevator, and the processor may perform the method 500 while the robot is positioned within the elevator. For example, the processor may determine whether the robot is positioned in the elevator based on a change amount pattern of data (eg, acceleration, angular velocity, and/or barometric pressure, etc.) measured through a sensor of the robot. Alternatively or additionally, the processor compares vision data (eg, images, image features, etc.) measured via sensors on the robot with pre-stored data (eg, reference images, image features, etc.) about the interior of the elevator. By doing so, it can be determined whether the robot is located in the elevator. Alternatively, the processor may determine whether the robot is located in the elevator by receiving information about whether the robot is located in the elevator from a control system (eg, a robot control system or an elevator control system). In one embodiment, if it is determined that the robot is positioned in the elevator, the processor may repeat method 500 while the robot is positioned in the elevator.

According to an embodiment, the processor may determine whether the elevator moves more than a predetermined number of floors. If it is determined that the elevator moves more than a predetermined number of floors, the processor measures air pressure data through a barometric pressure sensor included in the robot while the robot is in the elevator, and receives data for the floor where the elevator is currently located and air pressure data for each floor can do. Then, the processor may generate a comparison result between the barometric pressure data measured at the currently located floor and the air pressure data for each floor based on the measured air pressure data, the data on the currently located floor, and the air pressure data for each floor. Based on these results, the processor may determine whether or not the barometric pressure sensor included in the robot is abnormal. In an embodiment, the processor may measure barometric pressure data through a barometric pressure sensor included in the robot while the robot is in the elevator, and receive data on a floor on which the elevator is currently located. In this case, the processor may generate air pressure data for each floor based on the measured air pressure data and the received data on the floor where the elevator is currently located.

6 is a flowchart illustrating a method 600 of detecting an abnormality in a sensor included in the robot while the robot is positioned in an elevator according to an embodiment of the present disclosure. The robot control system that controls the robot may call the elevator to which the robot is to be boarded (S612). In one embodiment, in response to the robot's boarding request, the robot control system may call the elevator. For example, the robot control system may call an elevator through the elevator control system. Here, the robot control system may include an elevator control system. In response to the call of the robot control system, the elevator may be disposed on the floor where the robot is disposed (S614). That is, the elevator may be disposed in a state in which the robot can ride. The robot control system may transmit an elevator boarding command to the robot in response to the arrangement of the elevator (S616).

In response to the elevator boarding command of the robot control system, the robot may board the arranged elevator (S618). After boarding the elevator, the robot may determine whether the elevator is a robot-only elevator (S620). When it is determined that the elevator on which the robot is boarded is not a robot-only elevator, the robot may determine whether a passenger, that is, an object that becomes an obstacle, exists in the elevator (S622). When it is determined that the corresponding elevator is a robot-only elevator or it is determined that there are no passengers in the elevator, the robot may request data related to the elevator to the robot control system (S624). In response to this request, the robot control system may request data associated with the elevator to the elevator (or the system associated with the elevator) (S626). When it is determined that the elevator on which the robot is boarded is a robot-only elevator, step S624 may be directly executed. Accordingly, the elevator (or the system associated with the elevator) may provide data associated with the elevator to the robot control system (S628). For example, the elevator (or a system associated with the elevator) may provide the robot control system with pre-stored data about the interior of the elevator or data measured through a sensor associated with the elevator. The robot control system may transmit the received data associated with the elevator to the robot (S630).

The robot may perform an operation of comparing the received data associated with the elevator with data measured through a sensor included in the robot (S632). Based on the result of the comparison operation performed in this way, the robot may determine whether there is a problem (ie, abnormality or error) in the sensor (S634). When it is determined that there is a problem with the sensor, the robot can perform data correction work, that is, calibrate (or calibrate) the sensor by itself (S636), or provide information about the problem with the sensor included in the robot to the robot control system. (S638). At this time, sensor calibration is performed when the robot can perform sensor calibration on its own. may be provided. Then, the robot waits in the elevator until it arrives at the destination, and can get off when it arrives at the destination (S640).

On the other hand, if it is determined in step S620 that the elevator on which the robot is boarded is not a robot-only elevator, and there is an object (for example, a passenger) that is an obstacle inside the elevator in step S622, the sensor included in the robot Since data due to an object that is an obstacle may be included in data measured through , steps S624 to S638 may not be performed.

7 is a flowchart illustrating a method 700 of detecting an abnormality in an inertial sensor included in a robot according to an embodiment of the present disclosure. In an embodiment, the method 700 for detecting an abnormality in an inertial sensor included in the robot may be performed by a processor (eg, at least one processor of the robot). Here, the inertial sensor may be a sensor for measuring inertial data such as an acceleration sensor and a gyroscope (angular velocity sensor).

As shown, the method 700 of detecting an abnormality of the inertial sensor included in the robot may be initiated by the processor measuring first inertial data through the inertial sensor included in the robot (S710). Also, the processor may receive second inertial data measured through an inertial sensor associated with the elevator ( S720 ). To this end, the processor may request the robot control system for second inertial data measured through an inertial sensor associated with the elevator. The processor may perform step S720 and simultaneously perform step S710. For example, the second inertial data may be data measured in real time through an inertial sensor associated with an elevator when the processor measures the first inertial data through an inertial sensor included in the robot.

Then, the processor may determine information about the relationship between the first inertia data and the second inertia data (S730). In an embodiment, the processor may calculate a covariance between the first inertia data and the second inertia data through a state vector-based prediction model. The processor may determine whether an inertial sensor included in the robot is abnormal based on the determined relationship information (S740). When an abnormality exists in the inertial sensor included in the robot, the processor may perform calibration on the inertial sensor included in the robot ( S750 ). In an embodiment, when an abnormality exists in the inertial sensor included in the robot, the processor may automatically calibrate the inertial sensor included in the robot using the calculated covariance. On the other hand, when the calculated covariance is equal to or greater than a predetermined threshold, the processor may provide information about the abnormality of the inertial sensor included in the robot to the robot control system.

8 is a diagram illustrating an example in which inertial data is measured through inertial sensors 812 , 822 , 832 , and 842 associated with an elevator according to an embodiment of the present disclosure. As described above, the processor may receive the measured inertial data through an inertial sensor associated with the elevator. Since the elevator is mainly provided with a power source at all times and is designed for user safety, it may include a sensor that is more precise than the sensor included in the robot. In addition, as shown, the elevator may include a boarding unit 810 on which the robot rides, a hoisting machine 820 , a counterweight 830 , and a control panel 840 .

In an embodiment, an inertial sensor may be built-in, connected, or attached to at least one of the boarding unit 810 , the hoisting machine 820 , the counterweight 830 , and the control panel 840 of the elevator. For example, the acceleration of the elevator may be measured through the acceleration sensor 812 included in the boarding unit 810 of the elevator. As another example, the acceleration of the elevator may be measured through the acceleration sensor 832 included in the counterweight 830 of the elevator. As another example, the acceleration of the elevator may be measured through an acceleration sensor or a gyro sensor included in the hoisting machine 820 of the elevator. Here, the acceleration sensor or the gyro sensor may be built in an accessory device of the hoisting machine 820 . As another example, the acceleration of the elevator may be measured through an acceleration sensor included in the control panel 840 of the elevator. Here, the acceleration sensor included in the control panel 840 may be a wired and/or wireless communication acceleration sensor. The measured inertia data may be provided to the robot control system and provided to the robot positioned in the corresponding elevator. Alternatively, such inertial data may be provided directly to a robot positioned within the elevator.

9 is a diagram illustrating an example of detecting an abnormality in the inertial sensor 910 included in the robot and calibrating the inertial sensor 910 included in the robot according to an embodiment of the present disclosure. The process to be described later may be performed while the robot is positioned in the elevator. When the robot moves on an elevator, since the elevator and the robot are located in the same inertial system, data measured through the inertial sensor 910 included in the robot is the same as or similar to data measured through the inertial sensor associated with the elevator Should be. If the data measured through the inertial sensor included in the robot is different from the data measured through the inertial sensor 910 associated with the elevator, the inertial sensor associated with the elevator is more precise and reliable, so the inertial sensor included in the robot ( 910), it can be determined that there is an abnormality.

)를 수신하고, 승강기와 연관된 관성 센서(920)를 통해 측정된 관성 데이터(

)를 수신할 수 있다. 데이터 비교부(930)는 로봇에 포함된 관성 센서를 통해 측정된 관성 데이터(

) 및 승강기와 연관된 관성 센서를 통해 측정된 관성 데이터(

) 사이의 관계에 대한 정보를 결정할 수 있다. 프로세서는 결정된 관계에 대한 정보를 기초로 로봇에 포함된 관성 센서(910)의 이상 여부를 판정할 수 있다. 프로세서는 로봇에 포함된 관성 센서의 이상이 존재하는 경우, 로봇에 포함된 관성 센서(910)에 대한 캘리브레이션을 수행할 수 있다.The data comparator (eg, 430 and 930 of FIG. 4 ) receives inertial data (

), and the inertial data measured through the inertial sensor 920 associated with the elevator

) can be received. The data comparator 930 provides inertial data (

) and inertial data measured through the inertial sensor associated with the elevator (

) can determine information about the relationship between The processor may determine whether the inertial sensor 910 included in the robot is abnormal based on the determined relationship information. When an abnormality exists in the inertial sensor included in the robot, the processor may perform calibration on the inertial sensor 910 included in the robot.

) 및 관성 데이터(

) 사이의 공분산(

)을 산출할 수 있다. 프로세서는 산출된 공분산(

)을 기초로 로봇에 포함된 관성 센서(910)의 이상 여부를 판정할 수 있다. 로봇에 포함된 관성 센서(910)의 이상이 존재하는 것으로 판정되는 경우, 프로세서는 산출된 공분산(

)을 이용하여 로봇에 포함된 관성 센서(910)를 자동 보정(또는 캘리브레이션)할 수 있다. 예를 들어, 프로세서는 산출된 공분산(

)을 센서 필터링에 이용하여 관성 센서(910)의 노이즈 및/또는 시간 편이 현상을 보정할 수 있다.As shown, the data comparator 430 uses the state vector-based prediction model to obtain the inertial data (

) and inertial data (

) with the covariance between (

) can be calculated. The processor calculates the covariance (

) based on whether the inertial sensor 910 included in the robot is abnormal. When it is determined that there is an abnormality of the inertial sensor 910 included in the robot, the processor calculates the calculated covariance (

) may be used to automatically calibrate (or calibrate) the inertial sensor 910 included in the robot. For example, the processor computes the covariance (

) may be used for sensor filtering to correct noise and/or time shift of the inertial sensor 910 .

') 및 승강기와 연관된 관성 센서를 통해 다시 측정된 관성 데이터(

) 사이의 공분산(

)을 산출할 수 있다. 이에 따라 프로세서는 산출된 공분산(

)을 기초로 로봇에 포함된 관성 센서(910)의 이상 여부를 다시 판정하고, 로봇에 포함된 관성 센서(910)의 이상이 존재하는 것으로 다시 판정되는 경우, 프로세서는 산출된 공분산(

)을 이용하여 로봇에 포함된 관성 센서(910)를 자동 보정(또는 캘리브레이션)할 수 있다.Thereafter, the data comparator 930 performs the inertial data (

') and the inertial data measured back through the inertial sensor associated with the elevator (

) with the covariance between (

) can be calculated. Accordingly, the processor computes the covariance (

) based on whether the inertial sensor 910 included in the robot is abnormal again, and when it is determined again that the inertial sensor 910 included in the robot is abnormal, the processor calculates the covariance

) may be used to automatically calibrate (or calibrate) the inertial sensor 910 included in the robot.

Accordingly, as shown, while the robot is positioned in the elevator, the processor may perform the above-described process one or more times. That is, during the time from when the robot gets on the elevator to the time it gets off, the processor may detect an abnormality in the inertial sensor 910 included in the robot and correct (or calibrate) the inertial sensor 910 . For example, while the robot is positioned in the elevator, the processor may repeat the above-described process a predetermined number of times. By repeating the sensor abnormality detection and sensor correction process, the precision of the inertial sensor 910 included in the robot may be improved, and the service quality of the robot may be improved.

)이 미리 결정된 임계치 이상인 경우(즉, 센서 필터링이 불가능한 경우), 프로세서는 로봇에 포함된 관성 센서(910)의 이상에 대한 정보를 로봇 관제 시스템으로 제공할 수 있다. 예를 들어, 로봇에 포함된 관성 센서(910)를 통해 측정된 관성 데이터(

)가 선형 예측 값 외의 값에 해당하여, 산출된 공분산(

)이 미리 결정된 임계치 이상인 경우, 해당 관성 센서(910)의 이상 정보(예를 들어, 관성 센서의 이상 정도, 여기서, 관성 센서를 더 이상 사용할 수 없다는 정보)를 로봇 관제 시스템으로 제공할 수 있다.In one embodiment, the calculated covariance (

) is greater than or equal to a predetermined threshold (ie, when sensor filtering is impossible), the processor may provide information about the abnormality of the inertial sensor 910 included in the robot to the robot control system. For example, inertial data measured through the inertial sensor 910 included in the robot (

) corresponds to a value other than the linear predicted value, so the calculated covariance (

) is greater than or equal to a predetermined threshold, abnormality information of the corresponding inertial sensor 910 (eg, the degree of abnormality of the inertial sensor, here, information indicating that the inertial sensor can no longer be used) may be provided to the robot control system.

10 is a flowchart illustrating a method 1000 for detecting an abnormality in an optical sensor included in a robot according to an embodiment of the present disclosure. In an embodiment, the method 1000 for detecting an abnormality of an optical sensor included in the robot may be performed by a processor (eg, at least one processor of the robot). Here, the optical sensor may include a sensor such as a vision sensor, an optical camera, or an IR camera. As shown, the method 1000 for detecting an abnormality of the optical sensor included in the robot may be initiated by the processor determining whether there is an obstacle in the elevator (S1010). Here, the obstacle object may be an object that did not exist when data about the elevator, such as an obstacle, logistics, a person, or another robot, was stored. When it is determined that there is no obstacle in the elevator, the processor may perform steps S1020 to S1080 to be described later.

The processor may measure the first optical data through the optical sensor included in the robot (S1020), and receive data stored in advance for the inside of the elevator (S1030). Here, the first optical data may include one or more images, and the pre-stored data for the elevator interior may include image features for the elevator interior image. Thereafter, the processor may extract a first image feature from the first optical data ( S1040 ). The processor may generate a comparison result based on a difference between the extracted first image feature and the second image feature included in the previously stored data for the inside of the elevator ( S1050 ). That is, the processor may generate the comparison result by using the first optical data and data stored in advance for the inside of the elevator.

The processor may determine that there is an abnormality in the optical sensor included in the robot based on the comparison result ( S1070 ). For example, the processor may determine whether the difference between the first image feature and the second image feature is equal to or greater than a threshold based on the comparison result ( S1060 ). Here, when it is determined that the difference is equal to or greater than the threshold, the processor may determine that there is an abnormality of the optical sensor included in the robot ( S1070 ). In response, the processor may provide information about an abnormality of the optical sensor included in the robot to the robot control system (S1080). Here, the abnormality of the optical sensor may include at least one of an illuminance abnormality, an outline abnormality, or a screen abnormality.

11 is a diagram illustrating an example of detecting an abnormality in the optical sensor 1110 included in the robot according to an embodiment of the present disclosure. The process to be described later may be performed while the robot is positioned in the elevator. In an embodiment, the object determining unit 1100 may determine that there are no people or other obstacles inside the elevator based on an image (or image) input through a sensor included in the robot, radio waves, distance sensing data, and the like. Alternatively, the object determination unit 1100 may determine that there are no people or other obstacles inside the elevator by receiving real-time information about the inside of the elevator from a control system (eg, a robot control system or an elevator control system). have. Here, the object determining unit 1100 may be included in a processor of the robot. When it is determined by the object determination unit 1100 that there is no obstacle inside the elevator, the processor may perform a process for detecting an abnormality in the optical sensor 1110, which will be described later.

The processor may measure optical data 1112 through one or more optical sensors 1110 included in the robot. Here, the optical data 1112 may be one or more images and/or image information. In an embodiment, the feature extractor 1120 may extract the first image feature 1122 from the optical data 1112 . For example, the feature extractor 1120 may extract a first image feature point as an image feature from the optical data 1112 . The feature extractor 1120 may be included in the robot's processor.

The processor may receive data 1130 stored in advance for the inside of the elevator. For example, the processor may receive pre-stored data 1130 about the inside of the elevator from the robot control system. Alternatively, the processor may receive pre-stored data 1130 about the interior of the elevator from a storage device associated with the elevator. When the lighting and the object inside the elevator are always the same, the pre-stored data 1130 for the inside of the elevator may include a learning dataset pre-photographed in the reference environment. That is, the pre-stored data 1130 for the inside of the elevator may include a training dataset pre-photographed in an environment where objects other than the reference lighting and the reference object that are always present inside the elevator do not exist. The pre-stored data 1130 for the inside of the elevator may include a reference image 1132 (ie, an image pre-captured in a reference environment) and/or a second image feature 1134 . Here, the second image feature 1134 may refer to a feature extracted from the reference image 1132 .

The data comparator 1140 compares based on the difference between the first image feature 1122 extracted from the optical data 1112 and the second image feature 1134 included in the pre-stored data 1130 for the inside of the elevator results can be generated. Here, the data comparison unit 1140 may correspond to or be included in the data comparison unit 430 of FIG. 4 . For example, the data comparison unit 1140 compares the result based on the difference between the feature points extracted from the optical data 1112 and the feature points of the reference image 1132 included in the previously stored data 1130 for the inside of the elevator. can create Based on the comparison result generated in this way, the processor may determine whether the optical sensor 1110 included in the robot is abnormal. For example, when the difference between the image features (or the difference between the feature points) is equal to or greater than a predetermined threshold, the processor may determine that there is an abnormality in the optical sensor 1110 , and the abnormality of the optical sensor 1110 . information can be provided to the robot control system. Accordingly, the processor frequently detects an illuminance abnormality (eg, a lens or image sensor abnormality), an outline abnormality (eg, a lens focus abnormality), or a screen abnormality of the image (or image) input through the optical sensor 1110 . can detect

11, the data comparison unit 1140 directly receives the second image feature 1134 included in the data 1130 stored in advance for the inside of the elevator from the control system (eg, the robot control system or the elevator control system) and However, the present invention is not limited thereto. For example, the feature extraction unit 1120 receives the pre-stored data 1130 for the inside of the elevator from the control system, and the second image feature 1134 from the reference image 1132 included in the pre-stored data 1130. may be extracted, and the extracted second image feature 1134 may be provided to the data comparator 1140 . In addition, although FIG. 11 illustrates the feature extraction unit 1120 and the data comparison unit 1140 as separate components, the present invention is not limited thereto. For example, the feature extractor 1120 may be included in the data comparison unit 1140 .

12 is a diagram illustrating an example of extracting a first image feature 1122 by inputting optical data 1112 measured through an optical sensor associated with a robot to the feature extraction unit 1120 according to an embodiment of the present disclosure; to be. In an embodiment, the feature extractor 1120 may include an outline-based feature point detection algorithm and/or an illuminance-based feature point detection algorithm. For example, the feature extractor 1120 may include an outline-based feature point detection algorithm such as Scale Invariant Feature Transform (SIFT) and/or an illuminance-based feature point detection algorithm such as LBP (Local Binary Pattern) and ferns. In an embodiment, the feature extractor 1120 may extract the first image feature 1122 from the optical data 1112 (ie, image or image information). For example, the feature extractor 1120 may acquire various feature point information from the optical data 1112 (ie, image or image information) by using an outline-based feature point detection algorithm and/or an illuminance-based feature point detection algorithm. . Additionally, the feature extractor 1120 may receive pre-stored data about the inside of the elevator and extract a second image feature therefrom.

In an embodiment, the feature extractor 1120 selects feature points that are easy to identify, such as a corner point, from the optical data 1112 using SIFT, and then image features for a local patch centered on each feature point. (eg, a feature vector) 1122 may be extracted. For example, the feature extraction unit 1120 may use SIFT to extract an image feature 1122 indicating the direction of the brightness change around the feature point and/or the sharp degree of the brightness change in each local patch of the optical data 1112. can

In one embodiment, the feature extractor 1120 extracts feature points from the optical data 1112 using ferns, and image features (eg, feature vectors) for a local patch centered on each feature point ( 1122) can be extracted. For example, the feature extractor 1120 uses ferns to generate image features (eg, features) indicating whether the pixel brightness difference of any two points in the patch of optical data 1112 is + or - vector) 1122 can be extracted. That is, an image feature extracted using ferns may indicate a difference in brightness in units of pixels, and may include only sign information, not a value of the difference in brightness. Also, the image features extracted using ferns are image features for patch-by-patch matching, and may represent a brightness pattern of an image patch. Since the image features extracted using ferns use only the sign information of the brightness difference, not the brightness value of each pixel, it may be less affected by changes in the contrast of the image (or image).

In an embodiment, the feature extractor 1120 may extract a texture feature from the optical data 1112 as the image feature 1122 by using the LBP. For example, for all pixels in the optical data 1112 , the feature extractor 1120 may extract an image feature 1122 indicating a relative change in brightness of a peripheral area of each pixel by using the LBP.

13 is a flowchart illustrating a method 1300 of detecting an abnormality of a distance detection sensor included in a robot according to an embodiment of the present disclosure. In an embodiment, the method 1300 for detecting an abnormality of a distance sensor included in the robot may be performed by a processor (eg, at least one processor of the robot). Here, the distance sensor may include a Time Of Flight (ToF) sensor, a Light Detection and Ranging (LiDAR) sensor, a depth camera, and the like.

As shown, the method 1300 for detecting an abnormality of the distance detection sensor included in the robot may be initiated by the processor determining whether there is an obstacle in the elevator (S1310). Here, the object that becomes an obstacle may be an object that did not exist when data about the elevator, such as an obstacle, logistics, a person, or another robot, was stored. When it is determined that there is no obstacle in the elevator, the processor may perform steps S1320 to S1370 to be described later. The processor may measure the first distance detection data through a distance detection sensor included in the robot (S1320), and receive data stored in advance for the inside of the elevator (S1330). Here, the pre-stored data on the inside of the elevator may include a second shape feature corresponding to the actual interior space of the elevator. Thereafter, the processor may determine or extract the first shape feature from the first distance sensing data ( S1340 ).

The processor may determine whether the distance detection sensor included in the robot is abnormal based on the extracted first shape feature and the second shape feature included in data previously stored in the elevator. For example, the processor may determine a degree of similarity between the extracted first shape feature and the second shape feature included in pre-stored data for the inside of the elevator ( S1350 ). Here, the first shape feature includes three-dimensional data generated by inputting the first distance sensing data to the three-dimensional reconstruction artificial neural network model, and the second shape feature is a three-dimensional reference included in pre-stored data for the inside of the elevator It may contain data. For example, the processor may determine a similarity between three-dimensional data included in the first shape feature and three-dimensional reference data included in the second shape feature. The processor may determine whether a distance detection sensor included in the robot is abnormal based on the similarity (S1360). When there is an abnormality in the distance detection sensor included in the robot, the processor may provide information about the abnormality in the distance detection sensor included in the robot to the robot control system (S1370).

14 is a diagram illustrating an example of detecting an abnormality in the distance detection sensor 1410 included in the robot according to an embodiment of the present disclosure. The process to be described later may be performed while the robot is positioned in the elevator. The object determiner 1400 shown in FIG. 14 may perform a function similar to that of the object determiner 1100 of FIG. 11 . When it is determined by the object determination unit 1400 that there is no obstacle inside the elevator, the processor may perform a process for detecting an abnormality in the distance detection sensor 1410 .

The processor may measure distance sensing data 1412 through one or more distance sensing sensors 1410 included in the robot. Here, the distance sensing data 1412 may be one or more depth images and/or data. In one embodiment, the processor may extract/generate the first shape feature 1414 from the distance sensing data 1412 . Here, the first shape feature 1414 may be an image obtained by simplifying the interior space and surface of the elevator recognized by the distance sensor. The first shape feature 1414 may include 3D data extracted/generated from the distance sensing data 1412 through a 3D reconstruction artificial neural network model. That is, the first shape feature 1414 may include a three-dimensional model of the interior space of the elevator recognized by the distance sensor 1410 .

The processor may receive pre-stored data 1420 for the inside of the elevator. For example, the processor may receive the pre-stored data 1420 about the inside of the elevator from the robot control system. Alternatively, the processor may receive pre-stored data 1420 about the interior of the elevator from a storage device associated with the elevator. When the lighting and the object inside the elevator are always the same, the pre-stored data 1420 for the inside of the elevator may include a learning dataset pre-photographed in the reference environment. That is, the pre-stored data 1420 for the inside of the elevator may include a training dataset pre-photographed in an environment in which objects other than the reference lighting and the reference object that are always present inside the elevator do not exist. The previously stored data 1420 for the inside of the elevator includes distance data (eg, space size, length, width, height data, etc.), image (or video) data, and/or second shape features 1422 for the inside of the elevator. ) may be included. Here, the second shape feature 1422 may be an image obtained by simplifying the internal space and surface of an actual elevator. Alternatively, the second shape feature 1422 may include 3D reference data extracted/generated from data on the inside of the elevator through the 3D reconstruction artificial neural network model. For example, the second shape feature 1422 may include a three-dimensional model of the interior space of the actual elevator.

The data comparison unit 1430 compares the similarity between the first shape feature 1414 extracted/generated from the distance sensing data 1412 and the second shape feature 1422 included in the pre-stored data 1420 for the inside of the elevator. can decide Here, the data comparison unit 1430 may correspond to or be included in the data comparison unit 430 of FIG. 4 . In an embodiment, the data comparator 1430 may determine a similarity between 3D data included in the first shape feature 1414 and 3D reference data included in the second shape feature 1422 . For example, the data comparator 1430 determines a similarity between 3D data included in the first shape feature 1414 and 3D reference data included in the second shape feature 1422 through an image matching algorithm or the like. can

The data comparison unit 1430 may generate a comparison result between the distance sensing data 1412 and the previously stored data 1420 for the inside of the elevator based on the determined similarity. Based on the comparison result generated in this way, it may be determined whether the distance detecting sensor 1410 included in the robot is abnormal. In an embodiment, the processor may determine whether the distance detecting sensor 1410 included in the robot is abnormal based on the similarity determined by the data comparator 1430 . For example, when the similarity determined by the data comparator 1430 is less than a predetermined threshold, the processor may determine that there is an abnormality in the distance detection sensor 1410 . When it is determined that there is an abnormality in the distance detection sensor 1410 included in the robot, the processor may provide information about the abnormality in the distance detection sensor 1410 included in the robot to the robot control system.

14, the data comparison unit 1430 directly receives the second shape feature 1422 included in the data 1420 stored in advance for the inside of the elevator from the control system (eg, the robot control system or the elevator control system) and However, the present invention is not limited thereto. For example, the processor receives the pre-stored data 1420 for the inside of the elevator from the control system, extracts/generates the second shape feature 1422 from the pre-stored data 1420, and the extracted/generated second shape The feature 1422 may be provided to the data comparator 1430 .

15 is an example of extracting a first shape feature 1414 by inputting distance detection data 1412 measured through a distance detection sensor associated with a robot to the shape feature extraction unit 1510 according to an embodiment of the present disclosure; It is a drawing showing The shape feature extraction unit 1510 may include a 3D reconstruction artificial neural network model and the like. In an embodiment, the shape feature extractor 1510 may receive the distance sensing data 1412 (ie, image or image information) and extract/create the first shape feature 1414 . For example, the shape feature extractor 1510 may extract/create the first shape feature 1414 by simply converting the interior space, surface, etc. of the elevator recognized by the distance sensor. As another example, the shape feature extraction unit 1510 may extract/generate a 3D model of the interior space of the elevator recognized by the distance sensing sensor as the first shape feature 1414 using the 3D reconstruction artificial neural network model. have. Here, the shape feature extractor 1510 may be included in the data comparator (eg, 1430 of FIG. 14 ) and/or the processor.

In another embodiment, the shape feature extracting unit 1510 receives pre-stored data (eg, 1420 of FIG. 14 ) for the inside of the elevator, and extracts the second shape feature 1422 from the pre-stored data for the inside of the elevator It can be extracted/created. For example, the shape feature extractor 1510 may extract/generate the second shape feature by making an actual internal space, a surface, and the like of an actual elevator into a simple figure, based on previously stored data. Also, the shape feature extraction unit 1510 may extract/create a 3D model of the internal space of the elevator as the second shape feature 1422 based on previously stored data using the 3D reconstruction artificial neural network model. . The extracted/generated second shape feature and/or the generated 3D model may be provided to a data comparator (eg, 1430 of FIG. 14 ).

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution by a computer. The medium may continuously store a computer executable program, or may be a temporary storage for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributedly on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store for distributing applications, sites supplying or distributing other various software, and servers.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques may include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer readable media such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not so limited and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: robot
120: elevator

Claims (20)

In the method of detecting an abnormality of a sensor included in a robot using an elevator, performed by at least one processor,
measuring first data through a sensor included in the robot;
requesting second data associated with the elevator;
receiving second data associated with the elevator in response to the request for the second data;
generating a comparison result between the first data and the second data; and
Based on the generated comparison result, comprising the step of determining whether a sensor included in the robot is abnormal,
How to detect abnormalities in the sensors included in the robot.
According to claim 1,
further comprising determining whether the robot is positioned within the elevator;
The method is performed while the robot is positioned in the elevator.
How to detect abnormalities in the sensors included in the robot.
3. The method of claim 2,
The step of determining whether the robot is positioned in the elevator includes determining whether the elevator moves more than a predetermined number of floors,
When it is determined that the elevator moves more than a predetermined number of floors, while the robot is in the elevator,
The step of measuring the first data through the sensor included in the robot includes measuring the atmospheric pressure data through the barometric pressure sensor included in the robot,
Receiving the second data associated with the elevator includes receiving data on the floor in which the elevator is currently located and air pressure data for each floor,
The generating of a comparison result between the first data and the second data may include: barometric pressure data measured at the currently located layer based on the measured barometric pressure data, the data for the currently positioned layer, and the barometric pressure data for each layer; Comprising the step of generating a comparison result between the air pressure data for each layer,
How to detect abnormalities in the sensors included in the robot.
According to claim 1,
Further comprising the step of performing a process corresponding to the comparison result when the abnormality of the sensor is determined,
How to detect abnormalities in the sensors included in the robot.
According to claim 1,
The step of measuring the first data through the sensor included in the robot includes measuring the first inertial data through the inertial sensor included in the robot,
Receiving the second data associated with the elevator includes receiving second inertial data measured through an inertial sensor associated with the elevator.
How to detect abnormalities in the sensors included in the robot.
6. The method of claim 5,
generating a comparison result between the first data and the second data includes determining information about a relationship between the first inertial data and the second inertial data;
Determining whether the sensor included in the robot is abnormal based on the generated comparison result includes determining whether the inertial sensor included in the robot is abnormal based on information on the determined relationship ,
How to detect abnormalities in the sensors included in the robot.
7. The method of claim 6,
When it is determined that the sensor is abnormal, further comprising the step of performing a process corresponding to the comparison result,
The performing of the method includes performing calibration on the inertial sensor included in the robot when there is an abnormality in the inertial sensor included in the robot.
8. The method of claim 7,
The determining of information about the relationship between the first inertial data and the second inertial data includes calculating a covariance between the first inertial data and the second inertial data through a state vector-based prediction model. do,
When there is an abnormality in the inertial sensor included in the robot, performing calibration on the inertial sensor included in the robot is performed using the calculated covariance when there is an abnormality in the inertial sensor included in the robot. to automatically calibrate the inertial sensor included in the robot,
How to detect abnormalities in the sensors included in the robot.
7. The method of claim 6,
The determining of information about the relationship between the first inertial data and the second inertial data includes calculating a covariance between the first inertial data and the second inertial data through a state vector-based prediction model. do,
When it is determined that the sensor is abnormal, the step of performing the processing corresponding to the comparison result may include: When the calculated covariance is equal to or greater than a predetermined threshold, information about the abnormality of the inertial sensor included in the robot is provided to the robot control system comprising the step of
How to detect abnormalities in the sensors included in the robot.
According to claim 1,
Measuring the first data through the sensor included in the robot includes measuring the first optical data through the optical sensor included in the robot,
Receiving the second data associated with the elevator includes receiving data stored in advance for the inside of the elevator
How to detect abnormalities in the sensors included in the robot.
11. The method of claim 10,
Prior to the step of measuring the first data through the sensor included in the robot, further comprising the step of determining whether there is an object that is an obstacle inside the elevator,
In the step of generating a comparison result between the first data and the second data, when it is determined that there is no object obstructing the inside of the elevator, the first optical data and the previously stored data about the inside of the elevator are used to generate the generating a comparison result;
How to detect abnormalities in the sensors included in the robot.
12. The method of claim 11,
The step of generating the comparison result using the first optical data and the data stored in advance for the inside of the elevator comprises:
extracting a first image feature from the first optical data; and
generating the comparison result based on a difference between the extracted first image feature and a second image feature included in pre-stored data for the inside of the elevator;
How to detect abnormalities in the sensors included in the robot.
13. The method of claim 12,
The step of determining whether a sensor included in the robot is abnormal, based on the generated comparison result, may include, when the difference between the first image feature and the second image feature is equal to or greater than a predetermined threshold, the robot Comprising the step of determining that an abnormality of the optical sensor included in the
The method is
Further comprising the step of performing a process corresponding to the comparison result,
The performing includes providing information about the abnormality of the optical sensor included in the robot to the robot control system when there is an abnormality in the optical sensor included in the robot,
How to detect abnormalities in the sensors included in the robot.
According to claim 1,
The step of measuring the first data through the sensor included in the robot includes measuring the first distance detection data through the distance detection sensor included in the robot,
Receiving the second data associated with the elevator, the method of detecting an abnormality of a sensor included in the robot comprising the step of receiving the previously stored data about the inside of the elevator.
15. The method of claim 14
Prior to the step of measuring the first data through the sensor included in the robot, further comprising the step of determining whether there is an object that is an obstacle inside the elevator,
In the step of generating a comparison result between the first data and the second data, when it is determined that there is no obstacle in the interior of the elevator, the first distance sensing data and pre-stored data for the interior of the elevator are used. generating the comparison result;
How to detect abnormalities in the sensors included in the robot.
16. The method of claim 15,
The step of generating the comparison result by using the first distance sensing data and data stored in advance for the inside of the elevator comprises:
generating a first shape feature based on the first distance sensing data; and
determining a degree of similarity between the generated first shape feature and a second shape feature included in pre-stored data for the inside of the elevator
A method of detecting an abnormality in a sensor included in the robot, including.
17. The method of claim 16,
The first shape feature includes three-dimensional data generated by inputting the first distance sensing data to a three-dimensional reconstruction artificial neural network model,
The second shape feature includes three-dimensional reference data included in pre-stored data for the inside of the elevator,
Determining the degree of similarity between the generated first shape feature and the second shape feature included in pre-stored data for the inside of the elevator may include:
determining a degree of similarity between three-dimensional data included in the first shape feature and three-dimensional reference data included in the second shape feature;
A method of detecting an abnormality in a sensor included in the robot, including.
18. The method of claim 17,
The step of determining whether the sensor included in the robot is abnormal based on the generated comparison result includes determining whether the distance detection sensor included in the robot is abnormal based on the degree of similarity,
The method is
When it is determined that the sensor is abnormal, further comprising the step of performing a process corresponding to the comparison result,
The performing includes providing information about the abnormality of the distance detection sensor included in the robot to the robot control system when there is an abnormality in the distance detection sensor included in the robot,
How to detect abnormalities in the sensors included in the robot.
A computer program stored in a computer-readable recording medium to execute the method for detecting an abnormality in a sensor included in the robot according to any one of claims 1 to 18 in a computer.
An electronic device comprising:
communication module;
Memory;
one or more sensors; and
At least one processor coupled to the memory and configured to execute at least one computer readable program included in the memory
including,
the at least one program,
measure first data via the one or more sensors, request second data associated with an elevator; in response to the request for second data, receive second data associated with the elevator; and instructions for generating a comparison result between the second data and determining whether the one or more sensors are abnormal based on the generated comparison result.
electronic device.

Images