RU2800529C1 - Mobile robot and method for mobile robot control - Google Patents

Abstract

FIELD: robotics.

SUBSTANCE: mobile robot contains a motor, a lidar sensor, a camera sensor, and a processor. A three-dimensional map of the robot's surroundings contains an indication of a pedestrian crossing, its waiting area, and a visual indicator of movement. When an object is detected in the waiting area, a bounding box of the object is formed using the sensor data. The position of the bounding box is displayed on the 3D map. The zone of visual overlap of the object in the waiting area is determined using the position of the bounding box and the orientation of the visual indicator of movement relative to the waiting area. The overlap zone is the calculated part of the waiting area where the visual motion indicator is covered by an object in the field of view of the camera sensor when the robot is in this calculated part. The reduced waiting area is determined as the difference between the waiting area and the visual overlap area. A visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area, after which the robot moves to the reduced waiting area.

EFFECT: improved robot control.

27 cl, 7 dwg

Description

The field of technology to which the invention belongs

[01] The present technology relates to computer methods and systems for controlling a robotic vehicle, in particular, to methods and systems for controlling a mobile robot.

State of the art

[02] Autonomous robotic vehicles are vehicles capable of autonomously moving around private and/or public areas. With the vehicle location and/or environment sensor system, an internal or vehicle-related logic system controls the speed and direction of the robotic vehicle based on the location and environment of the vehicle determined by the sensors.

[03] The robotic vehicle may use various sensor systems such as, but not limited to, camera systems, radar systems, and lidar systems. Various sensor systems may be used to collect various information about the location and surroundings of the vehicle and/or information in a different format. For example, lidar systems can be used to collect point cloud data to build 3D maps of the environment and other potential objects located in close proximity to a robotic vehicle.

[04] Such autonomous robotic vehicles are used for a variety of purposes, including the delivery of goods and other items. These tasks may require moving the robotic vehicle through a dynamic environment, such as a crowded sidewalk, a crosswalk where people and vehicles can move, and so on. In performing this navigational task, the autonomous robotic vehicle needs to avoid collisions with dynamic objects such as people and moving vehicles and static objects such as parked cars, buildings, lampposts, and other stationary objects commonly found in urban and suburban areas. where the use of a robotic vehicle is expected.

[05] Robotic vehicles are often relatively small, so the various objects they encounter during operation can block the visibility of part of the surrounding space for its sensors. This can lead to adverse consequences, since information important for navigation purposes is in inaccessible parts of space, which prevents the safe operation of the robotic vehicle.

[06] US8712624 describes methods and apparatus for driving autonomous vehicles using maps containing visibility information.

Disclosure of invention

[07] During the delivery of goods to the buyer, the mobile robot can follow the route to the location of the buyer. When moving along a route, the mobile robot may need to cross the road at a pedestrian crossing. When approaching a pedestrian crossing, the mobile robot must determine whether it is allowed (for example, a green light is on) or prohibited (for example, a red light is on) with a traffic light.

[08] One or more objects, such as pedestrians, may also be waiting to pass through the crosswalk and thus may obstruct the mobile robot's camera sensor field of view, making it difficult to observe the traffic light condition. In other words, the developers of the present technology have found that such objects can interfere with the mobile robot monitoring the state of a traffic light by blocking the traffic light in the field of view of one or more mobile robot camera sensors when the mobile robot is in close proximity to a pedestrian crossing. This inability of a mobile robot to track the state of a traffic light is undesirable for its operation.

[09] Therefore, the developers of the present technology have developed methods to solve this problem and mobile robots capable of determining one or more target positions in a given area in the vicinity of a pedestrian crossing, to which the mobile robot can move, and positions from which the mobile robot can observe the state of the traffic light and is not covered by one or more objects located in this area.

[010] In some embodiments of the present technology, the mobile robot can detect the presence of an object in a given area in the immediate vicinity of a pedestrian crossing and create a "bounding box" for this object. The position of this bounding box can be displayed on a 3D map of the surrounding space. The mobile robot is able to determine the "visual overlap zone" due to the presence of an object in this area. In one embodiment of the invention, this zone of visual overlap can be a "shadow projection" of the bounding box if the traffic light on the 3D map is considered a light source. The mobile robot can select a target position in a given area outside of the visual overlap zone and move to this area while being able to observe the traffic light status.

[011] In other embodiments of the present technology, the mobile robot can also determine if the visual overlap area formed based on the bounding box is too large and/or if the rest of the area is too small. In such cases, the mobile robot can perform a "deep" analysis of the area of visual overlap due to the presence of the object. During this in-depth analysis, the mobile robot can generate the "shell structure" of the object, for example, from lidar data, and use it instead of a bounding box when forming a zone of visual overlap. A shell structure more accurately represents the shape of an object than a bounding box, while being smaller than the bounding box. Therefore, its shadow projection is likely to have a smaller area than the bounding box's shadow projection, which frees up additional surface area into which the mobile robot can potentially move.

[012] In the first broad aspect of the present technology, a mobile robot is implemented, containing a laser detection and ranging system (LiDAR, Light Detection and Ranging) sensor - a lidar sensor, as well as a camera sensor with a certain field of view, a propulsion system for moving the mobile robot in the surrounding environment and a processor associated with the lidar sensor, the camera sensor and the propulsion system. The processor has access to a three-dimensional map of the surrounding space, containing the appropriate indication of the pedestrian crossing, the waiting area of the pedestrian crossing and a visual indicator of traffic on the pedestrian crossing. The 3D map indicates the position of the visual motion indicator and its orientation relative to the holding area. The processor is capable of detecting an object in the waiting area and generating a bounding box of the object from the camera sensor data and/or the lidar sensor data. The processor is able to map the position of the bounding box to the 3D map. The processor is capable of determining the area of visual overlap of an object in the holding area using the position of the bounding box and the orientation of the visual motion indicator relative to the holding area. The visual overlap zone is the calculated part of the waiting area in which the visual indicator of movement is covered by an object in the field of view of the camera sensor when the mobile robot is in this calculated part. The processor is able to determine the reduced holdout area as the difference between the holdout area and the visual overlap area. A visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area. The processor is capable of controlling the propulsion system to move the mobile robot into the reduced waiting area.

[013] In some embodiments of the mobile robot, the processor is further capable of comparing the reduced holding area with a predetermined threshold area and, if the reduced holding area is greater than the predetermined threshold area, controlling the propulsion system to move the mobile robot into the reduced holding area. If the reduced latency zone is less than a predetermined threshold zone, the processor may further determine, from the lidar sensor data, an object shell that is a more accurate estimate of the object's shape than the bounding box. If the reduced holdout area is less than a predetermined threshold area, the processor may further display the position of the shell on the 3D map, determine another visual overlap area of the object in the holdout area using the shell position and orientation of the visual indicator relative to the holdout area, where the other visual overlap area represents another calculated the part of the waiting area in which the visual motion indicator is obscured by an object in the field of view of the camera sensor when the mobile robot is located in this other calculated part, and where the other calculated part is smaller than said calculated part. If the reduced waiting area is less than a predetermined threshold area, the processor may further determine the other reduced waiting area as the difference between the waiting area and the other visual overlap, whereby a visual motion indicator is visible in the camera sensor's field of view when the mobile robot is in the other reduced waiting area. . If the reduced sleep zone is smaller than the predetermined threshold zone, the processor is further able to compare another reduced sleep zone with the predetermined threshold zone. If the other reduced holding area is larger than the predetermined threshold area, the processor is able to control the propulsion system to move the mobile robot to another reduced holding area.

[014] In some embodiments of the mobile robot, the processor is further capable of locally storing at least a portion of the 3D map.

[015] In some embodiments of the mobile robot, the processor is further capable of using an Object Detection Neural Network (ODNN) for camera data and/or lidar data to perform detection.

[016] In some embodiments of the mobile robot, the processor for determining the visual overlap area is further capable of projecting at least one corner of the bounding box onto the holding area by constructing a ray path on a 3D map from the visual indicator of movement to the holding area, the beam passing through at least at least one of the corners of the bounding box.

[017] In some embodiments of the mobile robot, multiple beams are used, and the processor is able to build multiple beams from the visual indicator of movement towards the bounding box to determine the shadow projection of the bounding box on the waiting area, while the shadow projection of the bounding box is the zone of visual overlap.

[018] In some embodiments of the mobile robot, the camera sensor is located at a predetermined distance above the ground surface on which the mobile robot is moving, wherein the processor is further able to use the predetermined distance to determine the visual overlap zone, and the visual overlap zone is located at a predetermined distance above the waiting area.

[019] In some embodiments of the mobile robot, the processor is further capable of controlling the propulsion system to stop the mobile robot in the reduced waiting area.

[020] In some embodiments of the mobile robot, the processor is further able to monitor the state of the visual motion indicator.

[021] In some embodiments of the mobile robot, the visual indicator of movement is a traffic light.

[022] In some embodiments of the mobile robot, the processor is an onboard processor of the mobile robot.

[023] In some embodiments of the mobile robot, the processor is a remote processor of a server associated with the mobile robot.

[024] In the second broad aspect of the present technology, a method for controlling a mobile robot in the surrounding space is implemented. The method provides for access to a three-dimensional map of the surrounding space containing a corresponding indication of a pedestrian crossing, a waiting area for a pedestrian crossing and a visual indicator of movement at a pedestrian crossing, while the three-dimensional map indicates the position of the visual indicator of movement and its orientation relative to the waiting area. The method includes detecting an object in a waiting area and forming a bounding box of the object based on data from at least one sensor of the mobile robot. The method includes constructing on a three-dimensional map the trajectories of two rays from a visual indicator of movement towards the bounding box to determine the shadow projection of the bounding box onto the waiting area, which is the zone of visual overlap of the object in the waiting area. The visual overlap zone is the calculated part of the waiting area in which the visual indicator of movement is covered by an object in the field of view of the camera sensor when the mobile robot is in this calculated part. The method includes determining a reduced holding area by subtracting a visual overlap area from the holding area. If the reduced waiting area is greater than a predetermined threshold area, the method includes controlling the mobile robot to stop inside the reduced waiting area, wherein a visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area.

[025] In some embodiments of the method, if the reduced waiting area is less than a predetermined threshold area, the method includes determining the shell of the object using at least one sensor, wherein the shell is a more accurate estimate of the shape of the object than the bounding box. The method includes plotting on a three-dimensional map the trajectories of two rays from a visual indicator of movement towards the shell to determine another shadow projection of the shell onto the waiting area, which is another visual overlap zone of the object in the waiting area, smaller than the said visual overlap zone. The method includes determining another reduced holding area by subtracting another visual overlap area from the holding area. If the other reduced waiting area is larger than the predetermined threshold area, the method includes controlling the mobile robot to stop the robot in the other reduced waiting area.

[026] In some embodiments of the method, it further includes continuously monitoring the status of the visual motion indicator using at least one mobile robot camera sensor.

[027] In the third broad aspect of the present technology, a method for controlling a mobile robot is implemented, comprising a lidar sensor, a camera sensor with a certain field of view, a propulsion system for moving the mobile robot in the surrounding space, and a processor associated with the lidar sensor, the camera sensor, and the propulsion system. The method is executed by the processor. The method provides for access to a three-dimensional map of the surrounding space containing a corresponding indication of a pedestrian crossing, a waiting area for a pedestrian crossing, and a visual indicator of movement at the pedestrian crossing. The 3D map indicates the position of the visual motion indicator and its orientation relative to the holding area. The method includes detecting an object in the waiting area and generating a bounding box for the object based on camera sensor data and/or lidar sensor data. The method includes displaying the position of a bounding box on a 3D map. The method includes determining a zone of visual overlap of an object in the waiting area using the position of the bounding box and the orientation of the visual motion indicator relative to the waiting area. The visual overlap zone is the calculated part of the waiting area in which the visual indicator of movement is covered by an object in the field of view of the camera sensor when the mobile robot is in this calculated part. The method includes determining the reduced waiting area as the difference between the waiting area and the visual overlap area, whereby a visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area. The method includes controlling a propulsion system to move the mobile robot to a reduced waiting area.

[028] In some embodiments of the method, it further includes comparing the reduced holding area with a predetermined threshold area and, if the reduced holding area is greater than the predetermined threshold area, controlling the propulsion system to move the mobile robot into the reduced holding area. If the reduced waiting area is less than a predetermined threshold area, the method further includes determining the shell of the object from the lidar sensor data, wherein the shell is a more accurate estimate of the shape of the object than the bounding box. If the reduced holding area is smaller than the predetermined threshold area, the method further includes displaying the position of the shell on a 3D map. If the reduced holding area is less than the predetermined threshold area, the method further includes determining another visual overlap area of the object in the holding area using the shell position and orientation of the visual motion indicator relative to the holding area. The other visual overlap zone is another calculated part of the holding zone in which the visual motion indicator is obscured by an object in the field of view of the camera sensor when the mobile robot is in this other calculated part. The other calculated part is less than the said calculated part. If the reduced holding area is smaller than the predetermined threshold area, the method further includes determining another reduced holding area as the difference between the holding area and the other visual overlap area. A visual indicator of movement is visible in the field of view of the camera sensor when the mobile robot is in another reduced waiting area. If the reduced waiting area is smaller than the predetermined threshold area, the method further includes comparing the other reduced waiting area with the predetermined threshold area. If the other reduced holding area is larger than the predetermined threshold area, the method includes driving the propulsion system to move the mobile robot to the other reduced holding area.

[029] In some embodiments of the method, at least a portion of the 3D map is stored locally.

[030] In some embodiments of the method, detection includes using an object detection neural network (ODNN) for camera data and/or lidar data.

[031] In some embodiments of the method, determining the visual overlap area includes projecting at least one corner of the bounding box onto the holding area by drawing a ray on the 3D map from the visual motion indicator to the holding area such that the ray passes through at least one of the corners of the bounding box.

[032] In some embodiments of the method, multiple ray trajectories are constructed from a visual indicator of movement towards the bounding box to determine the shadow projection of the bounding box onto the waiting area, with the shadow projection of the bounding box being the area of visual overlap.

[033] In some embodiments of the method, the camera sensor is located at a predetermined distance above the ground on which the mobile robot is moving, determining the visual overlap area includes using this predetermined distance, and the visual overlap area is located at a predetermined distance above the waiting area .

[034] In some embodiments of the method, it further includes controlling the propulsion system to stop the mobile robot in the reduced waiting area.

[035] In some embodiments of the method, it further includes monitoring the state of the visual motion indicator.

[036] In some embodiments of the method, the visual indicator of movement is a traffic light.

[037] In some embodiments of the method, the processor is an onboard processor of the mobile robot.

[038] In some embodiments of the method, the processor is a remote processor of a server associated with the mobile robot.

Brief description of the drawings

[039] These and other features, aspects, and advantages of the present technology are contained in the following description, in the appended claims, and in the following drawings.

[040] In FIG. 1 is a schematic representation of an example computer system for use in some embodiments of systems and/or methods in accordance with the present technology.

[041] In FIG. 2 depicts a networked computing environment containing a robotic vehicle suitable for use with some embodiments of the present technology.

[042] In FIG. 3 is a perspective top view of a 3D map accessed by the computer system of FIG. 1 in accordance with various embodiments of the described technology.

[043] In FIG. 4 is a perspective view of the 3D map of FIG. 3 with a bounding box in accordance with various embodiments of the described technology.

[044] In FIG. 5 is a top plan view of the 3D map of FIG. 4 in accordance with various embodiments of the described technology.

[045] In FIG. 6 shows the bounding box and shell structure generated by the computer system of FIG. 1 for the same object in accordance with various embodiments of the described technology.

[046] In FIG. 7 is a flow diagram of a method performed in accordance with various embodiments of the described technology.

Implementation of the invention

[047] Various illustrative embodiments of the present technology are more fully explained in the following description and in the accompanying drawings. However, the present technology may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. The absolute dimensions and relative dimensions of the layers and regions in the drawings may be enlarged for the sake of clarity. The same numerals refer to the same elements everywhere.

[048] The examples and conventions presented herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically given examples and terms. It should be understood that those skilled in the art are capable of developing various methods and devices that are not explicitly described or shown, but that implement the principles of the present technology within its spirit and scope.

[049] In addition, to facilitate a better understanding, the following description may contain simplified implementations of the present technology. Those skilled in the art will appreciate that other embodiments of the present technology may be significantly more complex.

[050] In some cases, useful examples of modifications to the present technology are provided. They contribute to understanding, but also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive, and a person skilled in the art can develop other modifications within the scope of this technology. In addition, if modifications are not described in some cases, this does not mean that they are impossible and / or that the description contains the only possible implementation of one or another element of the present technology.

[051] It should be understood that the terms "first", "second", "third", etc. may be used here to describe various elements, however, this does not impose restrictions on such elements. These terms are used to distinguish one element from another. Thus, the discussed first element may be referred to as the second element without departing from the essence of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the listed elements.

[052] It should be understood that if an element is "connected" to another element or "associated" with another element, it can be connected to or connected to another element directly or through intermediate elements. In contrast, when an element is listed as "directly connected" to another element or "directly connected" to another element, there are no intermediate elements. Similarly, other words used to describe the relationship between elements (eg, "between" and "immediately between", "adjacent" and "immediately adjoining", etc.) should be interpreted in a similar way.

[053] The terminology used here is only intended to describe specific illustrative implementations and is not intended to limit the present technology. The use of terms in the singular in the document also implies their use in the plural, unless the context clearly indicates otherwise. In addition, it should be understood that the use in this description of the terms "comprises" and/or "comprising" defines the presence of the specified properties, structures, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more others. properties, structures, steps, operations, elements, components and/or their groups.

[054] The functions of the various elements shown in the drawings, including any functional block designated as "processor", can be implemented using dedicated hardware, as well as hardware capable of executing software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor, such as a central processing unit (CPU), or a processor dedicated to a specific purpose, such as a digital signal processor (DSP, Digital Signal Processor). In addition, explicit use of the term "processor" should not be construed as referring solely to hardware capable of executing software, and may refer to an application specific integrated circuit (ASIC), field programmable gate array (FPGA), software ROM, RAM, and non-volatile memory, but not limited to. Other general purpose and/or custom hardware may also be contemplated.

[055] Software modules, or simply modules or blocks that are intended to be implemented in software, may be represented here as any combination of flowchart elements or other elements that indicate the steps of a process and/or contain a textual description. Such modules may be implemented in hardware, as shown or implied. In addition, it should be understood that a module may include, for example, computer program logic, computer program instructions, software, stack, firmware, hardware circuitry, or a combination thereof that provides the desired capabilities, but is not limited to them.

[056] In the context of this description, the term "database" means any structured set of data, regardless of its specific structure, database management software, or computer hardware for storing this data, using it, or otherwise providing its use. The database may reside on the same hardware as the process for storing or using the information stored in the database, or the database may reside on separate hardware such as a dedicated server or multiple servers.

[057] The present technology may be implemented as a system, method, and/or computer program product. The computer program product may include a computer-readable storage medium (or storage mediums) storing computer-readable program instructions that, when executed by a processor, implement aspects of the described technology. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of other examples of computer-readable storage media includes portable computer disk, hard disk drive, random access memory (RAM), read only memory (ROM), flash memory, optical disk, USB memory stick, floppy disk, mechanical storage media. or visual coding (eg punched card or barcode) and/or any combination thereof. In this description, a computer-readable storage medium refers to a computer-readable medium for long-term storage of data. This category does not include transient signals such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating in a waveguide or other transmission medium (for example, light pulses propagating in an optical fiber cable), or electrical signals transmitted over wires.

[058] It should be understood that computer-readable program instructions can be downloaded to appropriate computing or processing devices from a computer-readable storage medium or to an external computer or external storage device over a network, for example, over the Internet, a local area network, a wide area network and / or wireless network. A network interface in a computing or processing device is capable of receiving computer-readable program instructions over a network and forwarding them for storage on a computer-readable storage medium in a suitable computing or processing device.

[059] Machine-readable program instructions for performing the actions of the present invention may be assembly language instructions, machine instructions, firmware instructions, configuration data for integrated circuits, and source code or object code written in any combination of one or more programming languages. The machine-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer, and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer over any kind of network.

[060] The description of principles, aspects, and embodiments of the present technology, as well as their specific examples, are intended to cover their structural and functional equivalents, whether they are currently known or will be developed in the future. For example, those skilled in the art will appreciate that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams that implement the principles of the present technology. It should also be understood that any functional diagrams, block diagrams, state transition diagrams, pseudocodes, and the like. correspond to various processes that can be represented in machine-readable program instructions. These machine-readable program instructions may be sent to a processor or other programmable data processing device to form a mechanism in which these instructions are executed by a computer processor or other programmable data processing device and provide a means to implement the functions or actions indicated in the block diagram and / or in block or in several blocks of the block diagram. These computer-readable program instructions may also be stored on a computer-readable storage medium that allows you to control a computer, a programmable data processing device and/or other devices to ensure their operation in such a way that the computer-readable storage medium with the instructions stored thereon is an article containing instructions that implement aspects of functions or actions defined in function diagrams, block diagrams, state transition diagrams, pseudocode, and the like.

[061] Machine-readable program instructions may also be downloaded to a computer, other programmable processing device, or other devices to initiate a series of operating steps on the computer, other programmable device, or other devices to form a computerized process in which instructions executed on the computer other programmable device or other device, implement functions or actions defined in function diagrams, block diagrams, state transition diagrams, pseudocode, and the like.

[062] In some alternative embodiments of the invention, the functions indicated in the functional diagrams, block diagrams, state transition diagrams, pseudocode, and the like, may not be performed in the order in which they are indicated in the drawings. For example, two blocks shown in a block diagram in series may actually be executed in essentially parallel, and sometimes these blocks may be executed in reverse order, depending on the functions being implemented. It should also be noted that each of the functions indicated in the drawings, and combinations of such functions, may be implemented by specialized hardware systems that perform the indicated functions or actions, or by combinations of specialized hardware and computer instructions.

[063] In the following, in view of the foregoing principles, some non-limiting examples illustrating various embodiments of aspects of the present invention are discussed.

computer system

[064] In FIG. 1 depicts computer system 100. Computer system 100 can be a multi-user computer, single-user computer, laptop computer, tablet computer, smart phone, embedded control system, or any other computer system now known or subsequently developed. In addition, it should be understood that some or all of the elements of computer system 100 may be virtualized and/or cloud-based. As shown in FIG. 1, computer system 100 includes one or more processors 102, memory 110, storage interface 120, and network interface 140. These system elements are interconnected by bus 150, which may include one or more internal and/or external buses (not shown) such as a PCI bus, a USB bus, an IEEE 1394 "Firewire" bus, a SCSI bus, a Serial-ATA bus, etc., to which various hardware elements are electronically coupled.

[065] Memory 110, which may be random access memory or any other form of memory, may contain data 112, an operating system 114, and program 116. Data 112 may be any data that represents the input or output of any program on computer system 100 The operating system 114 is, for example, a Microsoft Windows or Linux operating system. Program 116 may be any program or set of programs with program instructions that can be executed by a processor to control the actions performed by computer system 100.

[066] Storage interface 120 is used to connect data stores, such as storage 125, to computer system 100. One type of storage 125 is a solid state drive that can use an integrated circuit node for long-term storage of data. Another type of storage 125 is a hard drive, such as an electromechanical device that uses a magnetic storage device to store and retrieve digital data. Similarly, storage 125 may be an optical drive, a memory card reader capable of reading information from a removable memory card, such as an SD card, or a flash memory device that can be connected to computer system 100, such as via Universal Serial bus (USB, Universal Serial Bus).

[067] In some embodiments, computer system 100 may use well-known virtual memory techniques to allow programs of computer system 100 to behave as if they were accessing a large, contiguous address space, rather than the space of many smaller storage devices, such as memory 110 and storage 125. Therefore, while data 112, operating system 114, and programs 116 are shown as residing in memory 110, those skilled in the art will appreciate that these elements do not necessarily reside entirely in memory. 110 at the same time.

[068] Processors 102 may include one or more microprocessors and/or other integrated circuits. Processors 102 execute software instructions stored in memory 110. Upon startup of computer system 100, processors 102 may first perform a boot procedure and/or execute software instructions that make up operating system 114.

[069] Network interface 140 is used to connect computer system 100 to other computer systems or network devices (not shown) over network 160. Network interface 140 may include a combination of hardware and software that allows information to be exchanged on network 160. In some embodiments, embodiment of the invention, the network interface 140 may be a wireless network interface. The software on the network interface 140 may include software that uses one or more network protocols to communicate over the network 160. For example, one of these network protocols may be the Transmission Control Protocol/TCP/IP Internet protocol).

[070] It should be understood that computer system 100 is only an example and that the technology described may be used with computer systems or other computer devices of various configurations.

networked computing environment

[071] In FIG. 2 shows a networked computing environment 200 suitable for use with certain non-limiting embodiments of the present technology. The networked computing environment 200 includes a computing device 210 associated with the robotic vehicle 220. The environment 200 also includes one or more servers 235 connected to the computing device 210 via a communication network 240 (eg, the Internet, etc.).

[072] In at least some non-limiting embodiments of the present technology, the computer device 210 is associated with the control systems of the robotic vehicle 220. The computer device 210 can control various operating systems of the robotic vehicle 220, including the engine control system, the system steering, as well as signaling and lighting systems, but not limited to them.

[073] In some non-limiting embodiments of the present technology, the networked computing environment 200 may include a GPS satellite (not shown) transmitting a GPS signal to and/or receiving a signal from the computing device 210. It should be understood that the present technology is not limited to the GPS system and may use a location technology other than the GPS system. It should be noted that the GPS satellite may not be available at all.

[074] The robotic vehicle 220 with which the computing device 210 is associated may be any robotic vehicle for delivery, warehouse use, and the like. Although the robotic vehicle 220 is depicted as a ground vehicle, it may be different in other non-limiting embodiments of the present technology. For example, in some non-limiting embodiments of the present technology, the robotic vehicle 220 may be an aircraft, such as a flying drone, or a waterborne vehicle.

[075] The robotic vehicle 220 may be a fully autonomous vehicle or a partially autonomous vehicle in which some aspects of the vehicle's operation are remotely controlled by a human operator and other aspects are automated. In one non-limiting example, the robotic vehicle 220 may operate autonomously until it encounters an unexpected or unusual situation that it cannot handle autonomously, at which time it may communicate with a remote human operator. It should be noted that no restrictions are imposed on specific parameters of the robotic vehicle 200, such as manufacturer, model, year of manufacture, weight, dimensions, weight distribution, surface area, height, motor type, tire type (if tires are used), power supply system and other characteristics or parameters of the vehicle.

[076] According to the present technology, the implementation of the computing device 210 is also not subject to any particular restrictions. For example, the computer device 210 may be implemented as a vehicle engine control unit, a vehicle CPU, a computer system embedded in the robotic vehicle 220, a plug-in control module, and the like. It should be noted that the computing device 210 may or may not be permanently associated with the robotic vehicle 220.

[077] Computer device 210 may include some or all of the elements of computer system 100 shown in FIG. 1, depending on the specific implementation. In some embodiments, computing device 210 is an on-board computing device and includes processors 102, storage 125, and memory 110. In other words, computing device 210 includes hardware and/or software and/or firmware, or a combination thereof, for processing data, as described in more detail below.

[078] In some non-limiting embodiments of the present technology, communication network 240 is the Internet. In other non-limiting embodiments of the present technology, communications network 240 may be implemented as any suitable local area network (LAN, Local Area Network), wide area network (WAN, Wide Area Network), private communications network, and the like. It should be understood that the embodiments of the communications network 240 are for illustrative purposes only. Between the computing device 210 and the communication network 240, a communication link (not specifically indicated) is provided, the implementation of which depends, among other things, on the implementation of the computing device 210. By way of a non-limiting example only, the communication link may be implemented as a wireless communication link. . Examples of wireless links may include, but are not limited to, a 3G network link, a 4G network link. Communication network 240 may also use a wireless connection to servers 235.

[079] In some embodiments of the present technology, servers 235 are implemented as computer servers and may contain some or all of the elements of the computer system 100 shown in FIG. 1. In one non-limiting example, the servers 235 are implemented as Dell™ PowerEdge™ servers running the Microsoft™ Windows Server™ operating system, but they may also be implemented using any other suitable hardware, software, and/or firmware or a combination thereof.

[080] In some non-limiting embodiments of the present technology, processors 102 of computing device 210 may be linked to servers 235 to receive one or more updates. Such updates may include, but are not limited to, software updates, map updates, route updates, geofencing updates, weather data updates, and the like. In some non-limiting embodiments of the present technology, computing device 210 may also send some operational data, such as routes travelled, traffic data, performance data, and the like, to server 235. Some or all of the data transmitted between robotic vehicle 220 and servers 235 may be encrypted and/or anonymised.

[081] It should be noted that the computing device 210 may use a variety of sensors and systems to collect information about the environment 250 of the robotic vehicle 220. As shown in FIG. 2, robotic vehicle 220 may be equipped with multiple sensor systems 280. It should be noted that various sensor systems from a plurality of sensor systems 280 may be used to collect various types of data about the environment 250 of the robotic vehicle 220.

[082] In one example, multiple sensor systems 280 may comprise various optical systems, including one or more camera-type sensor systems mounted on robotic vehicle 220 and connected to processors 102 of computing device 210. In general, one or more systems The camera sensors may collect image data about various parts of the environment 250 of the robotic vehicle 220. In some cases, image data provided by one or more camera sensor systems may be used by the computing device 210 to perform object detection procedures. For example, computing device 210 may transmit image data provided by one or more camera-type sensor systems to an object detection neural network (ODNN) trained to locate and classify potential objects in the environment 250 of robotic vehicle 220.

[083] In at least some embodiments of the invention, the ODNN network can be trained in a manner known from the aspects disclosed in patent application RU2021108149 "Methods and electronic devices for detecting objects in the surrounding space of a self-driving car" of the same applicant, filed on 03/26/2021, content which is incorporated herein by reference in its entirety.

[084] In another example, the plurality of sensor systems 280 may include one or more radar-type sensor systems mounted on the robotic vehicle 220 and connected to the processors 102. In general, one or more radar-type sensor systems may collect data about various parts of the environment. space 250 of the robotic vehicle 220 using radio waves. For example, one or more radar-type sensor systems may collect radar data about potential objects in the environment 250 of the robotic vehicle 220, and the data may relate to the distance from the radar-type sensor system to objects, the orientation of objects, the speed of objects, and the like. .

[085] In yet another example, a plurality of sensor systems 280 may include one or more lidar systems mounted on robotic vehicle 220 and associated with processors 102. In general, the lidar system is capable of collecting data about the environment 250 of robotic vehicle 220, for example, to building a multidimensional map of objects in the environment 250 of the robotic vehicle 220. The lidar system can be installed on the robotic vehicle 220 in various locations and/or in various versions to collect information about the environment 250 of the robotic vehicle 220.

[086] In the context of the present technology, computing device 210 is capable of detecting one or more objects in the environment 250 of robotic vehicle 220 based on data received from one or more camera systems and one or more lidar systems. For example, a computing device 210 capable of detecting an object in the environment 250 of the robotic vehicle 220 can identify lidar and camera data associated with the object, form a "pattern" characterizing the features of the object, and detect the object by forming a bounding box for the object.

Operation of the robotic vehicle

[087] Embodiments of the described technology provide navigation or motion planning when operating a robotic vehicle in the environment in the presence of both static and dynamic (ie, moving) objects. In the context of the present description, a small ground robotic vehicle is considered, designed to deliver goods, packages, boxes or other parcels in an urban or suburban area. Based on the nature of the tasks it performs, such a robotic vehicle typically travels on sidewalks and footpaths. Therefore, the motion planning module in the vehicle must take into account the behavior of pedestrians moving parallel to its path or crossing it.

[088] In addition, the robotic vehicle can cross roads. When driving cars and other vehicles on roads in urban and suburban areas, small robotic vehicles may be invisible, which can lead to collisions, as a result of which the robotic vehicle and its cargo can be damaged or destroyed. Therefore, the movement planning module of such a robotic vehicle must take into account the presence of objects on the roadway, including moving and parked cars and other vehicles.

[089] One example of a task for a delivery vehicle would be to deliver a package from an origin to a destination by a specific time. Therefore, the motion planning module must take into account the speed of the robotic vehicle and determine whether proper progress is being made towards the destination. These considerations are especially relevant when meeting delivery times is critical to the delivery task, or when the destination is far away.

[090] For illustrative purposes, an example of a robotic vehicle uses lidar and other sensors located on different sides of the vehicle. The computer device associated with the robotic vehicle receives data from the sensors and generates a three-dimensional point map (point cloud). This 3D point map can be used by the robotic vehicle, for example, to determine the distance to objects in the environment and to determine the trajectory and speed.

[091] It is contemplated that the robotic vehicle may also use the 3D map provided to it by the servers 235. be given remote access to it. Additionally or alternatively, the 3D map of the environment can also be transferred, at least in part, to the robotic vehicle for local storage and local access.

[092] It should be noted that the servers 235 may collect information from one or more robotic vehicles (eg, a fleet of vehicles) tasked with mapping the surrounding space and generating corresponding three-dimensional maps of the area. For example, one or more robotic vehicles may generate a three-dimensional map of a street, a city block, an urban area, a city, or the like. Information from one or more robotic vehicles may be collected by servers 235 and combined into a 3D map to be used in operation of the robotic vehicle.

[093] During operation, the robotic vehicle can access a three-dimensional map of its surrounding space to perform navigation tasks in this surrounding space. In general, the coordinate system of a three-dimensional map allows you to determine the location of objects located in this space, such as poles, mailboxes, curbs, roads, buildings, fire hydrants, road barriers, traffic lights, pedestrian crossings, trees, fences, billboards, landmarks, etc. P.

[094] In particular, the 3D map contains information indicative of the position of an object in the coordinate system of the 3D map, and may also contain information indicative of the orientation of the object in the coordinate system of the 3D map. It can also be said that the 3D map may contain information indicative of the relative position and relative orientation of one of a pair of objects with respect to another of the pair of objects.

[095] It should be noted that one or more objects on a three-dimensional map may contain natural objects of the surrounding space and artificial objects. In general, natural objects include real physical objects found in the surrounding space, and artificial objects include objects that can be added to a three-dimensional map to solve the navigation problems of a robotic vehicle. For example, artificial objects may include navigation paths, accessible areas, inaccessible areas, waiting areas, and the like.

[096] It is assumed that artificial objects can be displayed on a three-dimensional map in a separate "layer" of the three-dimensional map. For example, to display man-made objects in a layer on a 3D map, servers 235 may use the coordinate system associated with the man-made "layer" and the coordinate system associated with the 3D map. The following describes in more detail how some of the artificial objects may be used by the robotic vehicle for navigation.

[097] FIG. 3 is a schematic representation of a scenario in which various embodiments of the technology under consideration may be used. Shows a top view 300 of a 3D map accessed by mobile robot 306.

[098] The three-dimensional map contains natural objects such as a road 301, a pedestrian crossing 302, and a visual indicator 303 of movement. In addition, the three-dimensional map contains man-made objects such as waiting area 305 . It is assumed that the three-dimensional map contains information indicating the position of the visual indicator of movement 303 and its orientation relative to the waiting area 305. In particular, the position and orientation of the visual motion indicator relative to the waiting area 305 allows a light projection 304 to be determined indicating how the visual signal from the visual motion indicator 303 propagates in the waiting area 305 in three-dimensional space.

[099] Suppose that at a given time, the mobile robot 306 moves in the surrounding space in order to deliver goods to the buyer. On the section of the route along which the mobile robot 306 delivers goods, it must cross the pedestrian crossing 302. When approaching the pedestrian crossing 302, the mobile robot 306 may analyze the visual signal of the visual indicator of movement 303 to determine whether it is allowed (for example, a green light is on) or prohibited (for example, a red light is on) cross the 302 crosswalk.

[0100] The mobile robot 306 may move to the waiting area 305 under various circumstances. In one example, the mobile robot 306 may move to the waiting area 305 to cross the pedestrian crossing 302 if the visual signal from the visual motion indicator 303 allows it. In another example, the mobile robot 306 may move to the waiting area 305 to wait for the opportunity to cross the pedestrian crossing 302, if this is currently prohibited by the visual indicator 303 movement.

[0101] In some embodiments of the present technology, the mobile robot 306 may continuously monitor the visual signal or "state" of the visual motion indicator 303 during operation to determine whether or not crossing the pedestrian crossing 302 is currently allowed or prohibited.

[0102] It should be noted that if the visual indicator 303 currently prohibits the transition, the mobile robot 306, as well as other dynamic objects, may wait in the waiting area 305 until the visual indicator 303 movement allows it. Such dynamic objects may include pedestrians waiting for permission to cross a pedestrian crossing 302. The developers of the present technology have found that such dynamic objects can interfere with the mobile robot 306 to continuously monitor the status of the visual motion indicator 303 by covering the visual motion indicator 303 in the field of view of one or more the camera sensors of the mobile robot 306 when the mobile robot 306 is in the waiting area 305 . The inability for the mobile robot 306 to monitor the status of the visual indicator 303 may be undesirable for the operation of the mobile robot 306 in the environment.

[0103] Therefore, the developers of the present technology have developed ways to solve this problem and mobile robots capable of determining one or more target positions in the waiting area 305, which the mobile robot 306 can move to be able to continuously monitor the state of the visual indicator of movement 303, not closed one or more dynamic objects located in the waiting area.

[0104] In at least some embodiments of the present technology, it is contemplated that the determination of one or more target positions in the holding area 305 may also be performed considering static objects in addition to or instead of dynamic objects. Static objects, such as mailboxes, poles, and/or billboards, may also obscure the visual indicator of movement within the field of view of one or more of the mobile robot 306's camera sensors when the mobile robot is in the waiting area 305.

[0105] To do this, the mobile robot 306 may determine whether an object (or objects) is in the waiting area 305 . For example, mobile robot 306 may detect a specific object in the environment of mobile robot 306.

[0106] It should be noted that the mobile robot 306 may form a bounding box of the object based on camera sensor data and/or lidar sensor data. In one example, camera data and/or lidar data may be input to an ODNN capable of forming a bounding box of an object.

[0107] It should be noted that in some embodiments of the present technology, the ODNN can detect the presence of an object and form a bounding box for this object, indicating its position in the surrounding space. In other embodiments, the ODNN may also classify an object as a certain type of object (eg, pedestrian, bicycle, pram).

[0108] In one example, mobile robot 306 may use lidar data to detect an object by forming a bounding box for that object. In this example, the bounding box may be generated by the mobile robot 306 in the coordinate system of the lidar sensor.

[0109] The position of the bounding box may be inferred by the ODNN in a variety of ways. In one embodiment of the invention, the position of the bounding box can be output as a combination of a center point, a height offset from the center point, a length offset from the center point, and a width offset from the center point in the lidar sensor (and/or lidar point cloud) coordinate system. In another embodiment of the invention, the position of the bounding box may be output as a combination of the coordinates of the respective corners of the bounding box (eg, eight in total) in the lidar sensor (and/or lidar point cloud) coordinate system.

[0110] Regardless of the specific format in which the position of the bounding box is defined in terms of the lidar sensor (and/or lidar point cloud) coordinate system, the mobile robot 306 may display the position of the bounding box on a 3D map as described above. In other words, the mobile robot 306 is capable of converting the position of the bounding box from the lidar sensor coordinate system (and/or lidar point cloud) to its position in the 3D map coordinate system.

[0111] To this end, the mobile robot 306 may implement a plurality of transformation algorithms. In at least one embodiment, the mobile robot 306 may apply a transformation matrix to the position of the bounding box in the lidar sensor coordinate system to determine the position of the bounding box in the 3D map coordinate system.

[0112] In at least some embodiments of the present technology, the transformation algorithm may be implemented as an Iterative Closest Point (ICP) algorithm. In general, the goal of the ICP algorithm is to determine the reference point cloud and transform the source cloud so that it best matches the reference one. The algorithm iteratively revisits the transformation options (parallel translation and rotation combinations) to minimize the deviation score. Quantitative indicator of deviation can be, for example, the distance between the source and reference point cloud, for example, the sum of the squared differences in the coordinates of the respective pairs. The ICP algorithm can be used to match 3D models given an initial transformation assumption.

[0113] It is contemplated that when a modified transform between lidar sensor data and 3D map data is defined, that modified transform can be used to convert the bounding box data from the lidar sensor coordinate system to the 3D map coordinate system.

[0114] FIG. 4 shows a perspective view 400 of a 3D map instead of a top view 300 with a bounding box 410 applied to the 3D map. It can be seen that the bounding box 410 is located in the waiting area 305.

[0115] The mobile robot 306 may determine the visual overlap area 450 of an object in the waiting area 306 using the position of the bounding box 410 and the orientation of the visual motion indicator 303 relative to the waiting area 305 . In at least some embodiments, before determining the visual overlap zone formed based on the bounding box 410, the mobile robot 306 can confirm that the bounding box 410 is within the waiting area 305 and/or in close proximity to the waiting area 305 on a 3D map.

[0116] In general, the visual overlap area 450 is a calculated portion of the waiting area 305 in which the visual motion indicator 303 is obscured in the camera sensor field of view of the mobile robot 306 by an object when the mobile robot 306 is in that calculated portion. In other words, when the mobile robot 306 is to move in this calculated portion of the waiting area 305, the mobile robot 306's camera sensor will not be able to monitor or observe the state of the motion visual indicator 303 due to the presence of the object.

[0117] To this end, the mobile robot 306 is capable of projecting at least one corner of the bounding box 410 onto the waiting area 305 by plotting at least one ray path on a 3D map from the visual indicator 303 of the movement to the waiting area 305, in which this at least one the beam passes through the corresponding corner of the bounding box 410. In FIG. 4 shows a plurality of rays 420 that (1) exit from the motion visual indicator 303, (2) pass through a plurality of corners 430 of a bounding box 410, and (3) pass through a plurality of points 440 in the waiting area 305. The mobile robot 306 may designate an interior area defined by a plurality of dots 440 in the waiting area 305 as a visual overlap area 450 .

[0118] In some embodiments of the present technology, it is contemplated that the mobile robot 306 is capable of trajectorying a plurality of rays 420 from the visual indicator 303 towards the bounding box 410 to determine a "shadow projection" of the bounding box 410 onto the waiting area 305. To some extent, it can be said that it is possible to construct trajectories of a plurality of rays 420 that project the "shadow" of the bounding box 410 onto the waiting area 305, similar to the visual indicator of movement 303 if it were a light source, and the bounding box 410 if if it were an opaque object, it would create a shadow in the waiting area 305 .

[0119] In at least some embodiments of the present technology, the path of one or more beams can be said to be built from the motion visual indicator 303 to the holding area 305 such that one or more beams touch the respective corners 430 of the bounding box 410.

[0120] It should be noted that the camera sensor of the mobile robot 306 is located at a predetermined distance above the earth's surface on which the mobile robot 306 is currently operating. Therefore, in some embodiments of the invention, it is assumed that the mobile robot 306 can use this predetermined distance when determining the zone 450 of visual overlap.

[0121] Indeed, assuming that the waiting area 305 is located on the 3D map right at ground level, the mobile robot 306 can use this predetermined distance to refine the area occupied by the visual overlap area 450. This can be useful for refining the area occupied by visual overlap zone 405 based on the height at which the mobile robot's camera sensor 306 is actually located. , which is parallel to the waiting area 305 and located at a predetermined distance above the waiting area 305 on a three-dimensional map. In these embodiments, the corrected interior region defined by the plurality of corrected points may be used as the height-adjusted visual overlap zone. In other words, the predetermined distance may be used by the mobile robot 306 to determine a visual overlap zone at the camera level of the mobile robot 306 based on the height at which one or more camera sensors are above ground level.

[0122] The mobile robot 306 is also able to determine the reduced waiting area as the difference between the waiting area 305 and the visual overlap area 450. It should be noted that the visual motion indicator 303 is visible in the field of view of the camera sensor of the mobile robot 306 when the mobile robot 306 is in the reduced waiting area 460.

[0123] The mobile robot 306 may use the reduced waiting area 460 to control the movement of the mobile robot 306 into and/or through the waiting area 305 such that the camera sensor of the mobile robot 306 is able to continuously monitor the state of the motion visual indicator 303. For example, you can control the propulsion system of the mobile robot 306 to move the mobile robot 306 to a predetermined point in the reduced waiting area 460.

[0124] In FIG. 5 shows the first potential point 501, the second potential point 502 and the third potential point 503 in the waiting area 460. The processor 102 may control the propulsion system of the mobile robot 306 to move the mobile robot 306 to a selected first candidate point 501, second candidate point 502, or third candidate point 503 in the waiting zone 460.

[0125] In some embodiments, the mobile robot 306 may compare the reduced waiting area 460 with a predetermined threshold area. For example, mobile robot 306 may compare the surface size of the reduced waiting area 460 with a predetermined surface size threshold. Without wishing to be bound by any particular theory, the developers of the present technology have found that the surface size of the reduced holding area 460 can be small in cases where (1) the object is relatively large and forms a large visual overlap area and (2) a plurality of objects, resulting in a plurality of visual overlap zones within the holding area 305, which, when combined, form a large overall visual overlap zone.

[0126] In those embodiments where the mobile robot 306 compares the reduced waiting area 460 with a predetermined threshold area, if it is determined that the reduced waiting area 460 is greater than the predetermined threshold area, the mobile robot 306 may control the propulsion system to move it to the predetermined threshold area. position in the reduced waiting area 460 and/or to move through the reduced waiting area 460. In addition, if it is determined that the reduced waiting area 460 is smaller than the predetermined threshold area, the mobile robot 306 may proceed to some sort of refinement of the visual overlap zone determination in the waiting area 305 due to the presence of an object.

[0127] To do this, the mobile robot 306 may determine the "shell" structure for the object based on the lidar sensor data of the mobile robot 306 and use it instead of the bounding box 410 to determine another refined visual overlap zone in the waiting area 305 due to the presence of this object.

[0128] In FIG. 6 shows a picture of a bounding box 610 that may be generated by processor 102 during an object detection procedure, and a picture of a shell 630 that processor 102 may generate for the same object based on lidar data. The manner in which the processor may form the shell 630 is not particularly limited.

[0129] It is contemplated that the shell generated by the processor 102 for the object may be the convex shell of the object. In some embodiments, the processor 102 may compute a convex hull consisting of a finite set of points in 3D space by implementing a "fast hull algorithm". Such an algorithm can be briefly described as a sequence of the following steps. First, processor 102 may search for points with minimum and maximum x coordinates, since they will always be part of the convex hull. If there are many points with the same minimum/maximum x value, the processor 102 uses the points with the minimum/maximum y value, respectively. The processor 102 may then draw a line across two of these points to divide the set into two subsets of points processed recursively. The processor 102 can then determine the furthest point on one side of this line. This point forms a triangle with points on the line. The points lying inside this triangle cannot be part of the convex hull and can be excluded from the analysis in the next steps. Processor 102 may then repeat the previous two steps for the two lines formed by the triangle. The processor 102 may continue to perform the above steps until the dots run out. This means that the execution of the recursive procedure has ended and the selected points form a convex hull.

[0130] It should be noted that generating a bounding box 610 for an object may be less computationally expensive and/or may take less time than generating a shell 620 for the same object. However, bounding box 610 is, in a certain sense, a rougher estimate of the shape of a given object than shell 620, and is typically larger than shell 620.

[0131] Therefore, in those embodiments where the reduced waiting area 460 is smaller than the predetermined threshold area, the mobile robot 306 may instead use the bounding box 410 to use an object shell that is a more accurate estimate of the object's shape than the bounding box 410. Using a more accurate shape estimate object allows for a more accurate estimate of the area of visual overlap in the holding area 305 and/or a more accurate estimate of the reduced holding area.

[0132] In this regard, it can be said that in some embodiments of the present technology, mobile robot 306 can perform a two-step reduced waiting area estimation process, where in the first step of the two-step estimation process, mobile robot 306 uses a bounding box 410 to determine a reduced waiting area 460, and in the second step of the two-step evaluation process, mobile robot 306 uses the shell of this object (instead of bounding box 410) to determine another refined, reduced waiting area.

[0133] It is contemplated that mobile robot 306 can convert object shell data from a lidar sensor coordinate system (and/or lidar point cloud) to a 3D map coordinate system. This can be done similarly to mobile robot 306 converting the position of the bounding box from the coordinate system of the lidar sensor (and/or lidar point cloud) to the coordinate system of a 3D map. In addition, mobile robot 306 can define a different object visual overlap area in waiting area 305 using the shell position (instead of bounding box 410) and orientation of visual motion indicator 303 relative to waiting area 305 in a manner similar to mobile robot generating object visual overlap area 450 using the position of the bounding box. frame 410 and the orientation of the visual indicator 303 movement. The mobile robot 306 may also determine the other reduced waiting area as the difference between the waiting area 305 and the other shell visual overlap area. It should be noted that the other reduced wait area is likely to be larger than the reduced wait area 460 because the other visual overlap area calculated based on the shell is likely to be smaller than the visual overlap area 450 calculated based on the bounding box 410.

[0134] Once the mobile robot 306 determines another reduced holding area, it may compare the other reduced holding area with a predetermined threshold area. If the other reduced holding area is larger than the predetermined threshold area, the processor may activate the propulsion control system of the mobile robot 306 to move the mobile robot 306 to another reduced holding area. Otherwise, the mobile robot 306 may, in its current position, wait until one or more objects leave the waiting area 305 .

[0135] In FIG. 7 is a flow diagram of a method 700 for controlling a mobile robot 306 in accordance with various embodiments of the described technology. The mobile robot 306 includes a lidar sensor, a camera sensor, and a propulsion system for moving the mobile robot around. Mobile robot 306 also includes a processor associated with the lidar sensor, camera sensor, and propulsion system. Method 700 may be performed by a processor associated with mobile robot 306.

Step 702: Accessing a 3D map of the environment.

[0136] Implementation of method 700 begins at step 702 with processor 102 accessing a 3D map of the environment in which mobile robot 306 is currently operating. Fig. 3). The 3D map shows the position of the visual motion indicator and its orientation relative to the holding area.

[0137] In general, the coordinate system of a three-dimensional map allows you to determine the location of objects located in this space, such as poles, mailboxes, curbs, roads, buildings, fire hydrants, road barriers, traffic lights, pedestrian crossings, trees, fences, billboards, landmarks, etc. In particular, the 3D map contains information indicative of the position of an object in the 3D map coordinate system, as well as the orientation of that object in the 3D map coordinate system. It can also be said that the 3D map may contain information indicative of the relative position and relative orientation of one of a pair of objects with respect to another of the pair of objects. In at least some embodiments of the invention, a three-dimensional map may contain natural objects and artificial objects.

Step 704: detecting an object in the waiting area.

[0138] Method 700 continues at step 704 with processor 102 detecting an object located in waiting area 305 and generating bounding box 410. It should be noted that processor 102 may generate object bounding box 410 from camera sensor data and/or lidar sensor data.

[0139] For example, processor 102 may input at least camera data (image data) and/or lidar data (point cloud data) to an ODNN capable of detecting the presence of objects and determining their location by forming bounding boxes around the objects present.

Step 706: Display the position of the bounding box on the 3D map.

[0140] Method 700 continues at step 706 with processor 102 displaying the position of bounding box 410 on a 3D map. It should be noted that in some embodiments, the mobile robot 306 is capable of converting the position of the bounding box from the lidar sensor (and/or lidar point cloud) coordinate system to its position in the 3D map coordinate system.

[0141] To do this, the mobile robot 306 may apply a transformation matrix on the position of the bounding box 410 in the lidar sensor coordinate system to determine the position of the bounding box 410 in the 3D map coordinate system.

Step 708: Determining the area of visual overlap of the object in the waiting area.

[0142] Method 700 continues at step 708 with processor 102 determining visual overlap zone 450 within wait zone 305 . For example, the processor 102 may use the position of the bounding box 410 and the orientation of the visual motion indicator 303 relative to the hold zone 305 to determine the zone 450 of visual overlap.

[0143] In at least some embodiments of the present technology, processor 102 may execute a ray trajectory generation algorithm to generate a shadow projection of bounding box 410 in hold zone 305 where one or more ray trajectories are generated or plotted from a motion visual indicator.

[0144] In at least one embodiment of the invention, the processor 102 can project at least one corner of the bounding box 410 onto the waiting area 305 by plotting a ray path on a three-dimensional map from the visual indicator 303 of the movement to the waiting area 305, while the beam passes through the at least one of the corners of the bounding box 410.

[0145] In some embodiments of the invention, the processor 102 may perform the construction of the trajectories of several beams on a three-dimensional map. For example, processor 102 may perform multiple ray trajectories from motion visual indicator 303 toward bounding box 410 to determine a shadow projection of bounding box 410 onto wait area 305 .

[0146] In alternative embodiments of the present technology, when determining visual overlap zone 450, processor 102 may also take into account the height at which the mobile robot's camera sensor 306 is above the ground. As noted above, in such embodiments, the processor 102 may use a predetermined distance that the camera sensor is above the ground to determine a visual overlap zone that is a predetermined distance above the waiting area 305.

[0147] The visual overlap area 450 is assumed to be a calculated portion of the waiting area 305 in which the visual motion indicator 303 is obscured by an object in the camera sensor's field of view when the mobile robot 306 is in that calculated portion.

Step 710: Determining the reduced holding area as the difference between the holding area and the visual overlap area.

[0148] The implementation of method 700 continues at step 710 with processor 102 determining the reduced latency zone 460 as the difference between the latency zone 305 and the visual overlap zone 450.

Step 712: control of the propulsion system to move the mobile robot to the reduced holding area.

[0149] Method 700 continues at step 712 with propulsion control system processor 102 activating to move mobile robot 306 into reduced waiting area 460. In some embodiments, the processor 102 may cause the mobile robot 306 to move to stop in the reduced waiting area 460 and observe the current state of the visual motion indicator 303. In other embodiments, the processor 102 may cause the mobile robot 306 to move through the reduced waiting area 460 at the entrance to the pedestrian crossing 302.

[0150] For a person skilled in the art, possible changes and improvements in the above-described embodiments of the present technology may be obvious. In particular, despite the use of a predictive model when generating estimates for dynamic objects and an analytical model when generating estimates for static objects, it should be clear that these restrictions are not necessary. For example, an object normally classified as dynamic may remain stationary when a robotic vehicle is operating next to it. Examples are parked cars or people standing still, such as waiting for a bus. Although in this case such objects are static in that they do not move, they can also be manipulated using a predictive model.

[0151] the Foregoing description is for illustrative purposes only, and not to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the appended claims.

Claims (66)

1. A mobile robot containing a lidar sensor (LIDAR), a camera sensor with a field of view, a propulsion system for moving the mobile robot in the surrounding space, as well as a processor connected with the possibility of communication with the lidar sensor, the camera sensor and the propulsion system and configured to: - obtaining access to a three-dimensional map of the surrounding space containing the corresponding indication of a pedestrian crossing, a waiting area for a pedestrian crossing and a visual indicator of traffic on a pedestrian crossing, while the three-dimensional map indicates the position of the visual indicator of movement and its orientation relative to the waiting area; - detection of an object located in the waiting area, and the formation of a bounding box (bounding box) of the object using camera sensor data and/or lidar sensor data; - displaying the position of the bounding box on a three-dimensional map; - determining the zone of visual overlap (occlusion) of an object in the waiting area using the position of the bounding box and the orientation of the visual indicator of movement relative to the waiting area, while the visual overlap zone is the calculated part of the waiting area in which the visual indicator of movement is covered by the object in the field of view of the camera sensor, when the mobile robot is in this calculated part; - defining the reduced waiting area as the difference between the waiting area and the visual overlap area, whereby a visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area; And - control of the propulsion system to move the mobile robot to a reduced waiting area. 2. Mobile robot according to claim 1, in which the processor is additionally configured to: - comparing the reduced holding area with a predetermined threshold area; - if the reduced waiting area is greater than a predetermined threshold area: the processor is configured to control the propulsion system to move the mobile robot into the reduced waiting area, - if the reduced latency zone is less than a predetermined threshold zone, the processor is configured to: - determining the shell of the object using the lidar data of the lidar sensor, while the shell is a more accurate estimate of the shape of the object than the bounding box; - displaying the position of the shell on a three-dimensional image; - determining another zone of visual overlap of the object in the waiting area using the position of the shell and the orientation of the visual indicator relative to the waiting area, while the other visual overlap zone is another calculated part of the waiting area in which the visual indicator of movement is covered by the object in the field of view of the camera sensor, when the mobile robot is located in another calculated part, smaller than said calculated part; - defining another reduced waiting area as the difference between the waiting area and another visual overlap area, wherein a visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the other reduced waiting area; - comparing another reduced holding area with a predetermined threshold area; - controlling the propulsion system to move the mobile robot to another reduced waiting area if the other reduced waiting area is larger than a predetermined threshold area. 3. The mobile robot of claim 1, wherein the processor is further configured to store at least a portion of the 3D map locally. 4. The mobile robot of claim 1, wherein, to perform the detection, the processor is further configured to use an object detection neural network for camera data and/or for lidar data. 5. The mobile robot according to claim 1, in which, in order to determine the zone of visual overlap, the processor is additionally configured to project at least one corner of the bounding box onto the waiting area by constructing a ray path on a three-dimensional map from a visual indicator of movement to the waiting area, while the beam passes through at least one of the corners of the bounding box. 6. The mobile robot of claim. 5, which uses multiple beams, and the processor is configured to build trajectories of multiple beams from a visual indicator of movement towards the bounding box to determine the shadow projection of the bounding box on the waiting area, while the shadow projection of the bounding box is visual overlap zone. 7. Mobile robot according to claim 1, in which the camera sensor is located at a predetermined distance above the earth's surface on which the mobile robot moves, while the processor is additionally configured to use a predetermined distance to determine a zone of visual overlap located at a predetermined distance above the waiting area. 8. The mobile robot of claim 1, wherein the processor is further configured to control the propulsion system to stop the mobile robot in the reduced waiting area. 9. The mobile robot of claim 1, wherein the processor is further configured to monitor the state of the visual motion indicator. 10. The mobile robot of claim 1, wherein the visual indicator of movement is a traffic light. 11. The mobile robot of claim 1, wherein the processor is an on-board processor of the mobile robot. 12. The mobile robot of claim 1, wherein the processor is a remote processor of a server associated with the mobile robot. 13. A method for controlling a mobile robot in the surrounding space, including: - obtaining access to a three-dimensional map of the surrounding space containing the corresponding indication of a pedestrian crossing, a waiting area for a pedestrian crossing and a visual indicator of traffic on a pedestrian crossing, while the three-dimensional map indicates the position of the visual indicator of movement and its orientation relative to the waiting area; - detecting an object in the waiting area using at least one sensor of the mobile robot and forming a bounding box of the object; - construction on a three-dimensional map of the trajectories of two rays from a visual indicator of movement towards the bounding box to determine the shadow projection of the bounding box on the waiting area, which is a zone of visual overlap of the object in the waiting area, while the visual overlap zone is the calculated part of the waiting area, in which the visual the motion indicator is covered by an object in the field of view of the camera sensor when the mobile robot is in the calculated part; - determination of the reduced waiting area by subtracting the visual overlap zone from the waiting area; - control of the mobile robot, in which the robot stops inside the reduced waiting area, and a visual indicator of movement is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area, if the reduced waiting area is greater than a predetermined threshold zone. 14. The method of claim. 13, including, if the reduced waiting area is less than a predetermined threshold area: - forming the shell of the object using at least one sensor, while the shell is a more accurate estimate of the shape of the object than the bounding box; - construction on a three-dimensional map of the trajectories of two rays from a visual indicator of movement towards the shell to determine another shadow projection of the shell onto the waiting area, which is another zone of visual overlap of the object in the waiting area, smaller than the mentioned visual overlap zone; - determining another reduced holding area by subtracting another visual overlap area from the holding area; - controlling the mobile robot in which the robot stops in another reduced waiting area if the other reduced waiting area is larger than a predetermined threshold area. 15. The method of claim 13, further comprising continuously monitoring the status of the visual motion indicator using at least one mobile robot camera sensor. 16. A method for controlling a mobile robot containing a lidar sensor (LiDAR), a camera sensor with a field of view and a propulsion system for moving the mobile robot in the surrounding space, as well as a processor associated with the lidar sensor, the camera sensor and the propulsion system, including: - obtaining access to a three-dimensional map of the surrounding space containing the corresponding indication of a pedestrian crossing, a waiting area for a pedestrian crossing and a visual indicator of traffic on a pedestrian crossing, while the three-dimensional map indicates the position of the visual indicator of movement and its orientation relative to the waiting area; - detection of an object located in the waiting area, and the formation of a bounding box of the object according to the camera sensor data and/or lidar sensor data; - displaying the position of the bounding box on a three-dimensional map; - determining the zone of visual overlap of the object in the waiting area using the position of the bounding box and the orientation of the visual indicator of movement relative to the waiting area, while the visual overlap zone is the calculated part of the waiting area in which the visual indicator of movement is covered by the object in the field of view of the camera sensor when the mobile robot is in this calculated part; - defining the reduced waiting area as the difference between the waiting area and the visual overlap area, whereby a visual motion indicator is visible in the field of view of the camera sensor when the mobile robot is in the reduced waiting area; And - control of the propulsion system to move the mobile robot to a reduced waiting area. 17. The method of claim 16, further comprising: - comparison of the reduced waiting area with a predetermined threshold area; - controlling the propulsion system to move the mobile robot to the reduced waiting area if the reduced waiting area is greater than a predetermined threshold area; - if the reduced holding area is less than the pre-set threshold area: - determination of the object's shell from the lidar data of the lidar sensor, which is a more accurate estimate of the object's shape than the bounding box; - displaying the position of the shell on a three-dimensional image; - determination of another zone of visual overlap of the object in the waiting area using the position of the shell and the orientation of the visual indicator relative to the waiting area, while the other visual overlap zone is another calculated part of the waiting area in which the visual indicator of movement is covered by the object in the field of view of the camera sensor when the mobile the robot is located in another calculated part, smaller than the said calculated part; - defining another reduced waiting area as the difference between the waiting area and another visual overlap area, wherein a visual indicator of movement is visible in the field of view of the camera sensor when the mobile robot is in the other reduced waiting area; - comparing another reduced holding area with a predetermined threshold area; And - control of the propulsion system to move the mobile robot to another reduced waiting area if the other reduced waiting area is larger than a predetermined threshold area. 18. The method of claim 16, wherein at least a portion of the 3D map is stored locally. 19. The method of claim 16, wherein the detection includes using an object detection neural network for camera data and/or for lidar data. 20. The method of claim 16, wherein determining the visual overlap area includes projecting at least one corner of the bounding box onto the holding area by plotting on a 3D map a ray path from the visual indicator of movement to the holding area in which the beam passes through at least at least one of the corners of the bounding box. 21. The method of claim 20, wherein multiple ray paths are plotted from the visual indicator of motion toward the bounding box to determine a shadow projection of the bounding box onto the waiting area, where the shadow projection of the bounding box is the area of visual overlap. 22. The method of claim 16, wherein the camera sensor is located at a predetermined distance above the ground on which the mobile robot is moving, determining the visual overlap zone includes using a predetermined distance, and the visual overlap zone is located at a predetermined distance above the zone. expectations. 23. The method of claim 16, further comprising controlling the propulsion system to stop the mobile robot in the reduced holding area. 24. The method of claim 16, further comprising monitoring the status of the visual motion indicator. 25. The method of claim 16 wherein the visual traffic indicator is a traffic light. 26. The method of claim 16, wherein the processor is an onboard processor of the mobile robot. 27. The method of claim 16 wherein the processor is a remote processor of a server associated with the mobile robot.