RU2789923C2 - Methods and systems for synchronizing sensors of unmanned vehicles (sdv) online - Google Patents

Abstract

FIELD: unmanned vehicles.

SUBSTANCE: invention relates to computerized methods and systems for synchronizing sensors used by an unmanned vehicle (SDV) for navigation, in particular to synchronizing a lidar (LIDAR) and a camera. A method for synchronizing data for operation of an unmanned vehicle comprises: causing, by means of an electronic device, a camera system and a lidar system, to provide image data and map data to the electronic device at a common time reference, determining, by means of the electronic device, a first timestamp for map data, a second timestamp for the image data; and a time offset between the first timestamp and the second timestamp. The time offset is used to initiate a delay between successive snapshots generated by the camera system.

EFFECT: reliability of the system during operation is increased.

20 cl, 9 dwg

Description

The technical field to which the invention belongs

[0001] The present technology relates to computerized methods and systems for synchronizing sensors used by an unmanned vehicle (SDV) for navigation; more specifically, the present technology relates to lidar (LIDAR) and camera synchronization.

State of the art

[0002] In the prior art, several computer navigation systems have been proposed and implemented that are capable of assisting in navigation and/or driving. These systems range from more basic map-based localization solutions (i.e. using a computer system to assist the driver in navigating a route from a starting point to a destination) to more complex solutions (computerized and/or driver-autonomous). driving systems).

[0003] Some of these systems are implemented as what is commonly known as "optimum rate control" or "cruise control" systems. In these systems, the computer system on board the vehicles maintains a user-specified vehicle speed. Part of the optimal speed control system implements an "intelligent distance control" system whereby a user can set a distance to a potential vehicle ahead (eg, select a value expressed in terms of number of vehicles) and the computer system adjusts the vehicle's speed at least partially based on a vehicle approaching a potential vehicle ahead within a predetermined distance. Some speed control systems are additionally equipped with a collision control system, and these systems, when a vehicle (or other obstacle) is detected in front of a moving vehicle, slow or stop the vehicle.

[0004] Some advanced systems provide fully autonomous driving of a vehicle without the direct control of an operator (ie driver). These autonomous vehicles (also referred to herein as "unmanned vehicles", "SDVs") include computer systems that can cause a vehicle to accelerate, brake, stop, change lanes, and park automatically.

[0005] To efficiently perform these actions, computer systems include or are otherwise communicatively coupled to a plurality of sensors including, for example, a camera and a lidar (LIDAR) sensor based on optical detection and ranging. When installed on a vehicle, the lidar sensor and camera, by default, operate independently, collecting data indicating the vehicle's surroundings. The lidar sensor rotates, at a predetermined rate of rotation (eg, one rotation cycle of 200 ms), around its vertical central axis, collecting data in the form of lidar point clouds. The camera collects data in the form of taking snapshots in the same direction (because it is rigidly fixed in it) at a predetermined shooting frequency (for example, once every 30 ms). During an installation process (or, for example, a maintenance process), the lidar sensor and the camera may be synchronized to be able to collect data indicative of the same predetermined area of the vehicle's surroundings at an identical point in time.

[0006] However, during operation of the vehicle, the lidar sensor and the camera may become out of sync due to various reasons. These reasons may include, for example, different times of lidar and camera data updates, or some mechanical effects: sudden acceleration/deceleration of the vehicle, driving over bumps or in a sharp turn. All of these reasons can cause desynchronization of the lidar and the camera, which results, for example, in the vehicle computer system receiving inconsistent data regarding the object (another vehicle, a pedestrian, or any obstacle) from the lidar and the camera.

[0007] The severity of this problem is illustrated by the following hypothetical scenario. Assume that the SDV travels along a route determined by a computer system based on data received from the lidar and camera. At one point, the vehicle's computer system may receive, from the lidar sensor, data indicative of another vehicle moving in front of the vehicle; whereas, according to the data received from the camera, this vehicle may already be moving out of the SDV view. This discrepancy in the input data received from the various sensors associated with the SDV can complicate the formation of a trajectory for the SDV.

[0008] International application number WO 2019079511 A1, published April 25, 2019, owned by DeepMap, Inc. and entitled "LIDAR and Camera Synchronization" discloses a method and system for synchronizing lidar and camera on an autonomous vehicle. The system selects a set of trace samples for the route, including a lidar scan and an image. For each trace sample, the system calculates the time offset by iterating through a set of time deltas. For each time delta, the system adjusts the camera timestamp by that time delta, projects the lidar scan onto the image as a lidar projection according to the adjusted camera timestamp, and calculates the lidar projection alignment score for that time delta. The system specifies the time offset for each trace sample as the time delta with the largest overlap score. The system then models the time drift of the camera versus lidar based on the computed time offsets for trace samples, and synchronizes the lidar and camera according to the simulated time drift.

The essence of the invention

[0009] The objective of the present technology is to improve at least some of the inconveniences present in the prior art. Embodiments of the present technology may provide and/or expand the scope of approaches and/or methods for achieving the goals and objectives of the present technology.

[0010] The developers of the present technology take into account at least one technical problem associated with prior art approaches.

[0011] The developers have recognized that there is a need to dynamically synchronize the time when the camera takes a snapshot with the time when the lidar sensor projects its laser(s). In other words, the non-limiting embodiments of the present technology are developed based on the developers' assessment of the need to synchronize the data generated by the camera and the lidar sensor in real time while the vehicle is moving. More specifically, the developers intend to create a way to synchronize a camera and a lidar sensor without the need for special vehicle maintenance to synchronize, stop to synchronize, or otherwise need to synchronize in a garage or laboratory environment. Due to this, the computer system should receive additional resistance to the above mechanical effects that cause desynchronization of the lidar sensor and the camera, which allows to reduce the time (and financial costs) for vehicle maintenance.

[0012] The developers of the present technology have learned that an identified problem associated with lidar sensor and camera synchronization can be resolved by associating a predetermined lidar scan rate and a predetermined camera scan rate with astronomical time. Due to this, the computer system, firstly, can determine the respective first and second timestamps at which the lidar and the camera collect data of a predetermined area of the vehicle's vicinity; and secondly, the computer system can determine the time offset between the first and second timestamps. Once the time offset is determined, the computer system can use it to modify (eg, reduce) the predetermined camera scan rate to synchronize with the lidar sensor to collect data from the predetermined area of the vehicle's surroundings at the same time.

[0013] The developers take into account that iterative repetition of the above steps should keep the lidar sensor and camera synchronized "on the fly", i.e. in real time, at least partially resolving recurrent cases of desynchronization.

[0014] In contrast to some prior art approaches, the present technology uses a common astronomical time-based reference to determine time offset instead of common objects identified in both the camera snapshot and the corresponding lidar point cloud.

[0015] In accordance with one broad aspect of the present technology, a computer-implemented method for synchronizing data for operation of an unmanned vehicle (SDV) is provided. Data is generated by the camera system and the lidar system, and the camera system and the lidar system are communicatively coupled to the electronic device. The data includes: (i) image data of a camera system installed on the SDV to form image data about a predetermined area of the SDV's surroundings, the camera data being in the form of a sequence of snapshots generated by the camera system, the sequence of snapshots representing a predetermined area in different points in time, and (ii) mapping data from a lidar system mounted on the SDV to generate mapping data regarding the vicinity of the SDV including a predefined area, wherein the mapping data is in the form of data points generated by the lidar system, wherein the data points represent objects located in different parts of the SDV neighborhood at different times. The method comprises: causing, by means of an electronic device, a camera system and a lidar system to provide image data and mapping data to the electronic device at a common time reference, the common time reference being an indication of the absolute time during which the data is generated; determining, by means of an electronic device, a first timestamp for the map data, the first timestamp corresponding to a given point in time in the common time reference, during which the lidar system generates data points representing objects located in the predetermined area; determining, by means of an electronic device, a second timestamp for the image data, the second timestamps corresponding to another given point in time in the common time reference, during which the camera system generates a given snapshot of the predetermined area; determining, by means of an electronic device, a time offset between the first timestamp and the second timestamp, the time offset indicating the difference between a given time and another given time at a common time origin; using, by the electronic device, a time offset to initiate a delay between successive snapshots generated by the camera system such that (i) at least one successive snapshot is generated by the camera system, and (ii) successive data points are relative to the previously of a certain section are formed by the lidar system at an identical future moment of time at a common time reference.

[0016] In some implementations of the method, the use of the temporal offset is performed during operation of the SDV.

[0017] In some implementations of the method, data synchronization for SDV operation is performed substantially continuously during SDV operation.

[0018] In some implementations of the method, causing the camera system to provide image data is performed during the first iteration of the camera system operation, and the use is made for a subsequent camera system operation.

[0019] In some implementations of the method, the subsequent operation is a predetermined number of subsequent iterations.

[0020] In some implementations of the method, the above method is repeated after a predetermined number of subsequent iterations for an additional number of subsequent iterations.

[0021] In some implementations of the method, the camera system has a field of view that matches a predetermined portion of the SDV's neighborhood when mounted on the SDV.

[0022] In some implementations of the method, prompting includes using the PTP protocol by means of an electronic device to transmit an absolute time indicator to the camera system and the lidar system.

[0023] In some implementations of the method, the absolute time indicator is an astronomical time indicator.

[0024] In some implementations of the method, the lidar system is a rotational lidar system such that the lidar system generates data points representing objects located in a first region of the SDV neighborhood at a first point in time, and wherein the first region is associated with the angular position of the lidar system at first moment in time.

[0025] In some implementations of the method, the lidar system generates data points representing objects located in a second region of the SDV neighborhood at a second point in time, wherein the second region is associated with the angular position of the lidar system at the second point in time, wherein the angular position of the lidar system differs in the first time point and at the second time point, wherein the first section is different from the second section.

[0026] In some implementations of the method, determining the first timestamp comprises: obtaining, by means of an electronic device, an indicator of a predetermined angular position of the lidar system such that when the lidar is in a predetermined angular position, the lidar generates data points relative to objects located in a predetermined section, determining, by means of an electronic device, the first timestamp as the point in time in the common time reference at which the lidar system is at a predetermined angular position.

[0027] In some implementations of the method, the determination of the first timestamp is performed by the electronic device using a timestamp determination algorithm.

[0028] In some implementations of the method, a given snapshot is generated by the camera system during the time interval to generate the snapshot, wherein the second timestamp is a point in time in the common time origin that corresponds to the midpoint in the time interval to generate the snapshot.

[0029] In some implementations of the method, using a time offset electronically to initiate a delay between successive snapshots generated by the camera system comprises: causing the camera system to add a snapshot time delay after a given future snapshot.

[0030] In some implementations of the method, a snapshot time delay is a set of snapshot time delays added after respective future snapshots that are equal in time to the duration of the time offset.

[0031] In some implementations of the method, in which the lidar system operates in rotation cycles indicating the scan rate of the lidar, and the camera system operates in scan phases indicating the scan rate of the camera, and this rotation cycle of the lidar system is equal to a predetermined phase multiplier scanning the camera system, wherein the plurality of snapshot time delays are distributed over a number of future snapshots corresponding to a predetermined multiplier of the camera system scan phases.

[0032] In accordance with another broad aspect of the present technology, an electronic device is provided. The electronic device contains a processor; a computer-readable non-volatile storage medium containing data, wherein the data is generated by a camera system and a lidar system communicatively connected via a communication interface to a processor and mounted on an unmanned vehicle (SDV). The data includes: (i) image data of a camera system installed on the SDV to form image data about a predetermined area of the SDV's surroundings, the camera data being in the form of a sequence of snapshots generated by the camera system, the sequence of snapshots representing a predetermined area in different points in time, and (ii) mapping data from a lidar system mounted on the SDV to generate mapping data regarding the vicinity of the SDV including a predefined area, wherein the mapping data is in the form of data points generated by the lidar system, wherein the data points represent objects located in different parts of the SDV neighborhood at different times. The processor is configured to cause the camera system and the lidar system to provide image data and mapping data at a common time reference, the common time reference being an indication of the absolute time over which data is generated; determining a first timestamp for the map data, the first timestamp corresponding to a given point in time in the common time origin, during which the lidar system generates data points representing objects located in the predetermined area; determine a second timestamp for the image data, the second timestamps corresponding to another given point in time in the common time origin, during which the camera system generates a given snapshot of the predetermined area; determine, by means of an electronic device, a time offset between the first timestamp and the second timestamp, the time offset indicating the difference between a given time and another given time at a common time origin; use a time offset to initiate a delay between successive snapshots generated by the camera system such that (i) at least one successive snapshot is generated by the camera system, and (ii) successive data points relative to a predetermined area are generated by the lidar systems at an identical future point in time at the common origin.

[0033] In some implementations of the electronic device, the processor is configured to use the time offset during operation of the SDV.

[0034] In some implementations of the electronic device, the processor is further configured to cause the camera system to provide image data during the first iteration of the camera system operation, further configured to use a time offset for a subsequent camera system operation.

[0035] As used herein, a "server" is a computer program that runs on appropriate hardware and is capable of receiving requests (eg, from client devices) over a network and making those requests or causing those requests to be made. The hardware may be implemented as one physical computer or one physical computer system, but neither need be the case with respect to the present technology. In the present context, the use of the term "server" does not imply that every task (e.g., instructions or requests received) or any particular task must be received, executed, or caused to be executed by an identical server (i.e., identical software and/or hardware). ); it means that any number of software elements or hardware devices may be provided in receiving/sending, performing or inducing any task or request, or the results of any task or request; and all such software and hardware may be a single server or multiple servers, all of which is included in the expression "at least one server".

[0036] As used herein, an "electronic device" is any computer hardware that is capable of executing software that is appropriate for the relevant task in question. In the context of the present description of the invention, the term "electronic device" means that the device can function as a server for other electronic devices and client devices; however, this need not necessarily be the case with the present technology. Thus, some (non-limiting) examples of electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be understood that in the present context, the fact that a device functions as an electronic device does not mean that it cannot function as a server for other electronic devices. The use of the term "electronic device" does not preclude the use of multiple client devices in receiving/sending, performing or causing any task or request, or the results of any task or request, or steps in any method described herein.

[0037] As used herein, a "client device" is any computer hardware that is capable of executing software that is appropriate for the relevant task in question. In the context of the present description of the invention in General, the term "client device" is associated with the user of the client device. Thus, some (non-limiting) examples of client devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that a device serving as a client device in the present context is not excluded from serving as a server for other client devices. The use of the term "client device" does not preclude the use of multiple client devices in receiving/sending, performing or causing any task or request, or the results of any task or request, or steps in any method described herein.

[0038] In the context of the present description of the invention, the expression "information" includes information in general of any nature or kind, capable of being stored in a database. Thus, information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions , messages, etc.), documents, spreadsheets, etc.

[0039] In the context of the present description of the invention, the expression "software component" includes software (corresponding to a specific hardware context) that is necessary and sufficient in order to implement the specific referenced function(s).

[0040] In the context of the present description of the invention, the expression "computer storage media" (also referred to as "data storage media") includes media in general of any nature and form, including, without limitation, RAM, ROM, disks (CD-ROM , DVD, floppy disks, hard disk drives, etc.), flash drives, semiconductor drives, tape drives, etc. A plurality of components may be combined to form computer storage media including two or more media components of the same type and/or two or more media components of different types.

[0041] As used herein, a "database" is any structured collection of data, regardless of its particular structure, database management software, or computer hardware on which the data is stored, implemented, or otherwise made available for use. The database may reside on identical hardware with a process that stores or uses the information stored in the database, or it may reside on separate hardware such as a dedicated server or multiple servers.

[0042] In the context of the present description of the invention, the words "first", "second", "third", etc. are used as adjectives only for the purpose of distinguishing the nouns they modify from each other, and not for the purpose of describing specific relationships between those nouns. Thus, for example, it should be understood that the use of the terms "first database" and "third server" does not imply a particular order, type, chronology, hierarchy, or ranking (for example) relative to the server, and their use does not (directly) imply that the "second server" must necessarily exist in any given situation. Additionally, as explained herein in other contexts, reference to a "first" element and a "second" element does not preclude that the two elements represent an identical actual real world element. Thus, for example, in some cases, the "first" server and the "second" server may be identical software and/or hardware components, in other cases they may be different software and/or hardware components.

[0043] Implementations of the present technology have at least one of the above objectives and/or aspects, but do not necessarily have all of them. It should be understood that some aspects of the present technology resulting from an attempt to achieve the above goal may not satisfy this goal and/or may satisfy other goals not specifically set forth in this document.

[0044] Additional and/or alternative features, aspects, and advantages of implementations of the present technology should become apparent from the description below, the accompanying drawings, and the appended claims.

Brief description of the drawings

[0045] These and other features, aspects, and advantages of the present technology should become more apparent upon consideration of the following description, the appended claims, and the accompanying drawings, in which:

[0046] FIG. 1 illustrates a schematic diagram of an exemplary computer system for implementing specific embodiments of systems and/or methods of the present technology.

[0047] FIG. 2 illustrates a networked computing environment suitable for use with some implementations of the present technology.

[0048] FIG. 3 illustrates a schematic diagram of a timing system, in accordance with non-limiting embodiments of the present technology.

[0049] FIG. 4 illustrates a schematic top view of the vehicle of the networked computing environment of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[0050] FIG. 5 illustrates a timing diagram of operation by means of the processor of the computer system of FIG. 1 of the camera system of the network computing environment of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[0051] FIG. 6 illustrates a timing diagram of operation by means of the processor of the computer system of FIG. 1 of the lidar system of the networked computing environment of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[0052] FIG. 7 illustrates a combined timing diagram of operation by the processor of the computer system of FIG. 1 of the camera system of the networked computing environment of FIG. 2 and the lidar network computing environment system of FIG. 2 prior to their synchronization, in accordance with non-limiting embodiments of the present technology.

[0053] FIG. 8 illustrates a combined timing diagram of operation by the computer system processor of FIG. 1 of the camera system of the networked computing environment of FIG. 2 and the lidar network computing environment system of FIG. 2 after they are synchronized, in accordance with non-limiting embodiments of the present technology.

[0054] FIG. 9 illustrates a flowchart for synchronizing data for operation of the networked computing environment vehicle of FIG. 2 according to non-limiting embodiments of the present technology.

Detailed description of the invention

[0055] The examples and conventions set forth herein are intended primarily to assist the reader in understanding the principles of the present technology, and not to limit its scope to such specifically set forth examples and terms. It should be appreciated that those skilled in the art may devise various arrangements that, although not explicitly described or shown herein, nevertheless implement the principles of the present technology and are included within its spirit and scope.

[0056] In addition, as an aid to understanding, the following description may describe relatively simplified implementations of the present technology. Those skilled in the art will appreciate that various implementations of the present technology may be highly complex.

[0057] In some cases, what are considered useful examples of modifications to the present technology may also be set forth. This is merely as an aid to understanding and furthermore, it is not intended to limit the scope or set forth the limitations of the present technology. These modifications are not intended to be an exhaustive list, and other modifications may be made by those skilled in the art while staying within the scope of the present technology. Additionally, if examples of modifications are not set forth, this should not be interpreted in such a way that modifications are not possible, and/or in such a way that what is described is the only way to implement this element of the present technology.

[0058] In addition, all statements in this document that set forth the principles, aspects and implementations of the technology, as well as their specific examples, cover their design and functional equivalents, regardless of whether they are currently known or developed in the future. Thus, for example, those skilled in the art will recognize that the flowcharts herein represent conceptual views of an illustrative circuit embodying the principles of the present technology. Likewise, it should be appreciated that all flowcharts, block diagrams, state transition diagrams, pseudocode, and the like. represent various processes, which may be represented for the most part on computer-readable media, and are therefore executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0059] The functions of the various elements shown in the drawings, including all functional blocks labeled "processor", may be provided by dedicated hardware, as well as hardware capable of executing software, in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of separate processors, some of which may be shared. In addition, explicit use of the term "processor" or "controller" should not be construed as meaning solely hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, dedicated an integrated circuit (ASIC), a field programmable gate array (FPGA), a read-only memory (ROM) for storing software, a random access memory (RAM) and a non-volatile storage device. Other hardware, conventional and/or specialized, may also be included.

[0060] Software modules, or simply modules that are meant to be software, may be represented herein as any combination of flowchart elements, or other elements indicating the execution of process steps, and/or textual description. Such modules may be implemented by hardware, which is shown explicitly or implicitly.

[0061] With these basic principles in mind, some non-limiting examples are discussed below to illustrate various implementations of aspects of the present technology.

computer system

[0062] Referring initially to FIG. 1, a computer system 100 is shown suitable for use with some implementations of the present technology, the computer system 100 comprising various hardware components including one or more single or multi-core processors collectively represented by a processor 110, a semiconductor storage device 120, a storage device 130 , which can be random access memory or any other type of storage device. Communication between various components of computer system 100 may be via one or more internal and/or external buses (not shown) (e.g., PCI bus, Universal Serial Bus, IEEE1394 Firewire bus, SCSI bus, Serial ATA bus, etc.). to which various hardware components are electronically connected. In accordance with embodiments of the present technology, semiconductor drive 120 stores program instructions suitable for loading into storage device 130 and executed by processor 110 to determine the presence of an object. For example, the program instructions may be part of a vehicle control application executed by processor 110. It should be noted that computer system 100 may have additional and/or optional components (not illustrated), such as network communication modules, localization modules, and the like. .

networked computer environment

[0063] FIG. 2 illustrates a networked computing environment 200 suitable for use with some embodiments of systems and/or methods of the present technology. The networked computing environment 200 includes an electronic device 210 associated with the vehicle 220 or associated with a user (not illustrated) that can operate the vehicle 220. The networked computing environment 200 further comprises a server 235 communicating with the electronic device 210 via a communications network 245 ( for example, via the Internet, etc., as described in more detail later in this document). Optionally, the networked computing environment 200 may also include a GPS satellite (not illustrated) transmitting and/or receiving a GPS signal to/from the electronic device 210. It should be understood that the present technology is not limited to GPS and may use positioning technology other than from GPS. It should be noted that the GPS satellite can go down at all.

[0064] The vehicle 220 with which the electronic device 210 is associated may include any touring or transportation vehicle, such as a personal or commercial vehicle, truck, motorcycle, or the like. The vehicle may be a user driven or driverless vehicle. It should be noted that the specific parameters of the vehicle 220 are not limiting, and these specific parameters include: vehicle manufacturer, vehicle model, year of manufacture of the vehicle, vehicle weight, vehicle dimensions, vehicle weight distribution, vehicle surface area vehicle, vehicle height, drive train type (eg 2x or 4x), tire type, brake system, fuel system, mileage, vehicle identification number and engine displacement.

[0065] The implementation of the electronic device 210 is not specifically limited, but as an example, the electronic device 210 may be implemented as a vehicle engine control unit, a vehicle CPU, a vehicle navigation device (e.g., ^TomTomTM , ^GarminTM ), a tablet computer, a personal computer built into the vehicle 220, or the like. Thus, it should be noted that electronic device 210 may or may not be permanently associated with vehicle 220. Additionally or alternatively, electronic device 210 may be implemented in a wireless communication device such as a mobile phone (eg, smartphone or cordless phone). In specific embodiments, the electronic device 210 has a display 270.

[0066] The electronic device 210 may include some or all of the components of the computer system 100 illustrated in FIG. 1. In particular embodiments, electronic device 210 is an on-board computer device and includes a processor 110, a semiconductor drive 120, and a storage device 130. In other words, electronic device 210 includes hardware and/or software and/or firmware, or a combination of the foregoing. to determine the presence of an object around the vehicle 220, as described in more detail below.

[0067] In accordance with non-limiting embodiments of the present technology, the electronic device 210 further comprises or has access to a first sensor 230. Additionally, the electronic device 210 may further comprise a second sensor 240. The first sensor 230 is configured to capture an image of the surrounding area 250, and the second sensor 240 is configured to capture the lidar point cloud of the surrounding area 250 and other sensors (not illustrated). First sensor 230 and second sensor 240 are communicatively coupled to processor 110 to transfer such captured information to processor 110 for processing, as described in more detail hereinafter.

First sensor

[0068] In a specific non-limiting example, the first sensor 230 includes a camera. How the camera is implemented is not limited in a particular way. For example, in specific non-limiting embodiments of the present technology, the camera may be implemented as a single camera with sufficient resolution to detect objects at predetermined distances of up to about 30 m (although cameras with other resolutions and ranges are within the scope of this disclosure). The camera may be mounted in the cabin, in the upper portion of the windshield of the vehicle 220, but other locations are within the scope of this disclosure, including the rear window, side windows, front hood, roof, grille, or front bumper of the vehicle 220. In some non-limiting embodiments of the present technology, the first sensor 230 may be mounted in a dedicated housing (not illustrated) mounted on the roof of the vehicle 220.

[0069] In some non-limiting embodiments of the present technology, the first sensor 230 may be implemented as a plurality of cameras. For example, a plurality of cameras may have a sufficient number of cameras to capture an ambient/panoramic image of the surrounding areas 250.

[0070] In some non-limiting embodiments of the present technology, the camera (or one or more cameras that constitute an implementation of the first sensor 230) is configured to capture a predetermined portion of the surrounding area 250 around the vehicle 220. In some embodiments of the present technology, the camera is configured to capture an image (or sequence of images) that represents about 90 degrees of the surrounding area 250 around the vehicle 220, i. e. along the path of the vehicle 220.

[0071] In other non-limiting embodiments of the present technology, the camera is configured to capture an image (or sequence of images) that represents about 180 degrees of the surrounding area 250 around the vehicle 220, i. along the path of the vehicle 220.

[0072] In still further embodiments of the present technology, the camera is configured to capture an image (or sequence of images) that represents about 360 degrees of the surrounding area 250 around the vehicle 220, i. along the path of the vehicle 220 (in other words, the entire surrounding area around the vehicle 220).

[0073] In a specific non-limiting example, the first sensor 230 may be implemented, may be of the type offered by FLIR Integrated Imaging Solutions Inc., 12051 Riverside Way, Richmond, BC, V6W 1K7, Canada. It should be clearly understood that the first sensor 230 may be implemented in any other suitable equipment.

[0074] In non-limiting embodiments of the present technology, the first sensor 230 may be configured to take snapshots at a predetermined camera scan rate (e.g., one snapshot every 30ms), thereby generating image data of a predetermined area (illustrated in FIG. 4) the surrounding area 250 of the vehicle 220.

Second sensor

[0075] In a specific non-limiting example, the second sensor 240 includes a lidar (LIDAR) measuring device based on optical detection and ranging. Lidar stands for optical detection and ranging. It is intended that those skilled in the art understand the functionality of a lidar measurement instrument, but in brief, the transmitter (not illustrated) of the second sensor 240 implemented as a lidar sends a laser pulse and the light particles (photons) are scattered back to the receiver (not illustrated) of the second sensor 240 implemented as a lidar measuring instrument. The photons that return to the receiver are collected by the telescope and counted as a function of time. Using the speed of light (~3×10 ⁸ m/s), processor 110 can then calculate how far the photons have traveled (when going around). Photons can be reflected back from many different objects surrounding the vehicle 220, such as other particles (aerosols or molecules) in the atmosphere, another vehicle, stationary objects, or potential obstructions in the surrounding area 250 of the vehicle 220. These photons comprise so-called lidar clouds. points and indicate map data generated by the second sensor 240.

[0076] In a specific non-limiting example, the second sensor 240 may be implemented as a lidar sensor, which may be of the type offered by Velodyne LiDAR, Inc. 5521 Hellyer Avenue, San Jose, CA 95138, USA. It should be clearly understood that the second sensor 240 may be implemented in any other suitable equipment.

[0077] In some embodiments of the present technology, the second sensor 240 may be implemented as a plurality of lidar sensors, such as three, for example, or any other suitable number. In some embodiments of the present technology, the second sensor 240 (regardless of whether it is implemented as a single lidar sensor or multiple lidar sensors) may be housed in the above housing (not illustrated separately) located on the roof of the vehicle 220.

[0078] In those embodiments of the present technology in which the second sensor 240 is implemented as multiple lidar sensors placed in a housing (not illustrated), the spatial placement of the multiple lidar sensors can be designed taking into account the specific technical configuration of the multiple lidar sensors, the configuration of the housing, the weather conditions of the area in which the vehicle 220 is to be used (for example, frequent rain, snow, and other elements), and the like.

[0079] In non-limiting embodiments of the present technology, the second sensor 240 may be mounted on the vehicle 220 and rotate thereon at a predetermined rate (e.g., one rotation cycle of 200 ms), which is referred to herein as the predetermined lidar scan rate. In this regard, the second sensor 240 may be configured to generate mapping data of the surrounding area 250 of the vehicle 220 at a predetermined lidar scan rate.

Other sensors

[0080] In additional non-limiting embodiments of the present technology, the electronic device 210 may further comprise or have access to a third sensor (not illustrated). The third sensor (not illustrated) may contain a (radar) measuring device based on radio detection and ranging. Briefly, a radar meter is a detecting meter using radio waves to determine the range, angle, and/or speed of objects. The radar instrument includes a transmitter for generating electromagnetic waves, an antenna used for transmitting and receiving electromagnetic waves, a receiver, and a processor for determining properties of detected objects. In alternative embodiments of the present technology, a separate antenna for receiving waves and a separate antenna for transmitting waves may be provided. The processor used to determine the properties of surrounding objects may be processor 110.

[0081] In some embodiments of the present technology, the third sensor (not illustrated) may comprise long, medium, and short range radar sensors. As a non-limiting example, a long range radar sensor can be used for adaptive speed control, automatic emergency braking, and forward collision warning, while medium and short range radar sensors can be used for parking assistance, traffic alerts. cross traffic, crossroads assistance and blind side detection.

[0082] In a specific non-limiting example, the third sensor (not illustrated) may be of the type offered by Robert Bosch GmbH Robert-Bosch-Platz 1, 70839 Gerlingen, Germany. It should be clearly understood that the third sensor (not illustrated) may be implemented in any other suitable equipment.

[0083] In some embodiments of the present technology, the first sensor 230 and the second sensor 240 can be calibrated such that for an image captured by the first sensor 230 and a lidar point cloud captured by the second sensor 240, the processor 110 is configured to identify a given area image that corresponds to a given area of the lidar point cloud captured by the second sensor 240. In other embodiments of the present technology, the first sensor 230, the second sensor 240, and the third sensor (not illustrated) are calibrated such that for an image captured by the first sensor 230, of the lidar point cloud captured by the second sensor 240 and the radar data captured by the third sensor (not illustrated), the processor 110 is configured to identify a given area of the image that corresponds to the given area of the lidar point cloud and the radar data.

[0084] A third sensor (not illustrated) may therefore provide additional information or supplement what is obtained by the first sensor 230 and/or the second sensor 240. As a non-limiting example, it is shown that in certain cases, a radar sensor can perform better than a lidar instrument under specific weather conditions such as fog, rain, snow and dust. Additionally, the radar sensor can accurately determine the relative speed of traffic or the speed of a moving object using Doppler frequency shift. Thus, the third sensor (not illustrated) may be used in combination with the first sensor 230, in combination with the second sensor 240, or in combination with the first sensor 230 and the second sensor 240. In some embodiments of the present technology, the third sensor (not illustrated), for example , can only be used temporarily, for example, in difficult weather conditions.

[0085] In non-limiting embodiments of the present technology, the electronic device 210 may additionally contain or have access to other sensors (not illustrated). Other sensors (not illustrated) include one or more of the following: an inertial measurement unit (IMU), a global navigation satellite system (GNSS) instrument, ground speed radars, ultrasonic sonar sensors, odometer sensors including accelerometers and gyroscopes, mechanical tilt sensors, magnetic compass, and other sensors that enable vehicle 220 to operate.

[0086] As a non-limiting example, the IMU may be attached to vehicle 220 and include three gyroscopes and three accelerometers to provide rotational and linear motion data of vehicle 220 that may be used to calculate the movement and position of vehicle 220.

[0087] This calibration may be performed during the manufacture and/or installation of the vehicle 220. Alternatively, the calibration may be performed at any suitable time thereafter, or, in other words, the calibration may be performed during the upgrade of the vehicle 220 by the first sensor 230, a second sensor 240 and a third sensor 236, in accordance with the non-limiting embodiments of the present technology contemplated herein. Alternatively, calibration may be performed while equipping vehicle 220 with first sensor 230, second sensor 240, and third sensor 236 in accordance with the non-limiting embodiments of the present technology contemplated herein.

Communication network

[0088] In some embodiments of the present technology, communication network 245 is the Internet. In alternative non-limiting embodiments, the communications network may be implemented as any suitable local area network (LAN), wide area network (WAN), private communications network, and the like. It should be clearly understood that the communications network implementations are for purposes of illustration only. How the communication link (without a separate reference number) between the electronic device 210 and the communication network 245 is implemented will depend on, among other things, how the electronic device 210 is implemented. Just by way of example, and not as a limitation, in In those embodiments of the present technology in which the electronic device 210 is implemented as a wireless communication device, such as a smartphone or navigation device, the communication link may be implemented as a wireless communication link. Examples of wireless links include, but are not limited to, a 3G network link, a 4G network link, and the like. Communication network 245 may also use a wireless connection to server 215.

Server

[0089] In some embodiments of the present technology, the server 215 is implemented as a traditional server computer and may include some or all of the components of the computer system 1 of FIG. 1. In one non-limiting example, server 112 is implemented as a Dell™ PowerEdge™ server running the Microsoft™ Windows Server™ operating system, but may also be implemented in any other suitable hardware, software and/or firmware, or a combination of the foregoing. In the illustrated non-limiting embodiments of the present technology, the server is a single server. In alternative non-limiting embodiments of the present technology (not shown), the functionality of server 215 may be distributed and may be implemented across multiple servers.

[0090] In some non-limiting embodiments of the present technology, the processor 110 of the electronic device 210 may communicate with the server 215 to receive one or more updates. Updates may include, but are not limited to, software updates, map updates, route updates, weather updates, and the like. In some embodiments of the present technology, processor 110 may also be configured to transmit certain operational data, such as driving routes, traffic data, performance data, and the like, to server 215. Some or all of the data transmitted between vehicle 220 and server 215 may be encrypted and/or anonymized.

[0091] In non-limiting embodiments of the present technology, server 215 is configured to provide, via communications network 245, absolute time indicators to electronic device 210, as described below with reference to FIG. 3.

Synchronization system

[0092] Referring now to FIG. 3, a synchronization system 300 implemented in accordance with a non-limiting embodiment of the present technology is illustrated. The synchronization system 300 is configured to synchronize the operation of the first sensor 230 and the second sensor 240, in accordance with non-limiting embodiments of the present technology. Synchronization system 300 includes a processor (e.g., processor 110 of electronic device 210), a camera system 302 including a first sensor 230, and a lidar system 304 including a second sensor 240.

[0093] According to non-limiting embodiments of the present technology, the camera system 302 may further comprise a camera synchronization component 306. The lidar system 304 may further comprise a lidar synchronization component 308 .

[0094] Generally speaking, the timing system 300 is configured to coincide, in time, capturing image data 310 of a predetermined area of the surrounding area 250 generated by the camera system 302 with capturing map data 312 representing data points of objects located in the predetermined area. , map data that is generated by lidar system 304. To this end, in non-limiting embodiments of the present technology, processor 110 is configured to provide camera system 302 and lidar system 304 with a common time reference for capturing and processing data.

[0095] In this regard, as mentioned above, the processor 110 may be configured to receive, from the server 215, the absolute time indicator 320. In non-limiting embodiments of the present technology, absolute time is astronomical time.

[0096] In non-limiting embodiments of the present technology, astronomical time is the UT1 version of UTC. More specifically, astronomical time is mean solar time at 0° longitude. Astronomical time is calculated from observations of distant quasars using long baseline interferometry, laser ranging of the Moon and artificial satellites, and determination of the orbits of GPS satellites. Astronomical time, as mentioned in this document, is identical throughout planet Earth and is proportional to the angle of rotation of the Earth relative to distant quasars, namely, according to the International Celestial Reference Frame (ICRF), ignoring some small adjustments.

[0097] In some non-limiting embodiments of the present technology, astronomical time may further be approximated by a UTC version of Universal Time (Coordinated Universal Time). In these embodiments, astronomical time represents the atomic time scale for UT1, where it is counted using seconds provided by the International Atomic Time (TAI) standard, resulting in 86400 seconds per day with occasional introductions of a lost second.

[0098] As mentioned above, the camera system 302 includes a first sensor 230 and a camera synchronization component 306. The camera sync component 306 is communicatively coupled to the first sensor 230 using any compatible interface depending on the type of camera sync component 306 and the type of the first sensor 230 and associated software (drivers). To the processor 110, the camera synchronization component 306 is communicatively coupled via a communication link 314 .

[0099] In specific non-limiting embodiments of the present technology, the camera synchronization component 306 may be of the Jetson TX2 type available from NVIDIA Corporation, located at 2788 San Tomas Expy, Santa Clara, CA 95051, USA. It should be clearly understood that the camera synchronization component 306 may be implemented in any other suitable hardware.

[0100] Generally speaking, the camera synchronization component 306 is configured to transmit to the processor 110, via the communication line 314, data indicative of the scanning phases of the first sensor 230. Additionally, the camera synchronization component 306 is configured to receive, from the processor 110, based on the indicator 320 absolute time adjusted schedule for the operation of the first sensor 230. The schedule is generated such that at least one of the subsequent snapshots of the image data 310 is taken, by the first sensor 230, at a future point in time, simultaneously with the second sensor 240 generating the map data 312 objects in it (ie, to ensure that subsequent snapshots of image data 310 are coordinated in time with the generation of map data 312). The scanning phases of the first sensor 230 and scheduling are described below with reference to FIG. 4-8.

[0101] In some non-limiting embodiments of the present technology, link 314 may be based on Precision Time Protocol (PTP). Briefly, the PTP protocol is specified in the IEEE 1588 standard as a high-precision clock for network measurement and control systems and is designed to synchronize clocks in packet-switched networks that include distributed device clocks with varying accuracy and stability. The IEEE 1588 standards that define the PTP protocol describe a hierarchical master-slave architecture for distributing common time references. According to this architecture, the components of the timing system 300 can be represented as any source (eg, "master node", processor 110 in the synchronization system 300) or destination (eg, "slave node", camera timing component 306) for a timing reference (indicator 320 absolute time). The PTP protocol is based on the TAI time standard, in fact, UTC time can be calculated from the time received via the PTP grandmaster (root origin) time. Accordingly, in a local area network, the PTP protocol can achieve clock accuracy in the sub-microsecond range.

[0102] In other non-limiting embodiments of the present technology, link 314 may be based on Network Time Protocol (NTP). Briefly, the NTP protocol is a protocol used to synchronize computer clock times in a network that typically uses a server-client architecture. The NTP client initiates the exchange of temporary requests with the NTP server. As a result of this exchange, the client is able to calculate the link delay and its local offset and adjust its local clock to match that of the server computer. The NTP protocol uses UTC to synchronize computer clock times, and, in fact, synchronization accuracy of less than 1 millisecond on a local area network and synchronization accuracy of tens of milliseconds over the Internet can be achieved.

[0103] In non-limiting embodiments of the present technology, the lidar timing component 308 is configured to transmit to processor 110, via link 316, data indicative of times and corresponding angular positions of second sensor 240 in a given cycle. Then, the lidar timing component 308 is configured to receive, from the processor 110, lidar timestamps generated based on the absolute time indicator 320 for subsequent cycles of operation of the second sensor 240. Each of the lidar timestamps corresponds to future times when the second sensor 240 is in a predetermined angular position, to generate map data 312 of objects located in a predefined area of the surrounding area 250. The following describes in detail how the lidar system 304 is configured to generate map data 312, and how the processor 110 is configured to generate temporal lidar tags with reference to FIG. 4 and 6.

[0104] According to non-limiting embodiments of the present technology, link 316 may be based on the PTP protocol described above.

[0105] Generally speaking, in non-limiting embodiments of the present technology, the lidar timing component 308 may be implemented as a separate board including a microcontroller (not illustrated). A microcontroller (not illustrated) is connected to processor 110 via link 316 as described above, for example using an Ethernet interface. A microcontroller (not illustrated) further couples to the second sensor 240 using a universal asynchronous transceiver (UART) interface and pulses per second (PPS) signals generated exactly once per second. This allows the lidar timing component 308 to transmit the lidar timestamps received from the processor 110, thereby driving the second sensor 240 in a common time reference, as described in more detail below with reference to FIG. 4-8.

[0106] Referring to FIG. 4 is a schematic top view of a vehicle 220 including a camera system 302 and a lidar system 304 mounted thereon, in accordance with non-limiting embodiments of the present technology. In the embodiments of FIG. 4, the camera system 302 is rigidly mounted on the vehicle 220 such that the first sensor 230 is directed to a predetermined portion 402 of the surrounding area 250. In other words, the first sensor 230 has a field of view that coincides with the predetermined portion 402 of the surrounding area 250. From this purpose, the camera system 302 is configured to capture, using the first sensor 230, image data 310 of a predetermined portion 402 of the surrounding area 250 at various points in time.

[0107] In non-limiting embodiments of the present technology, the lidar system 304 mounted on the roof of the vehicle 220 and operating simultaneously with the camera system 302 may be a rotational lidar system. To this end, the second sensor 240 rotates in direction 414 about its central vertical axis. Accordingly, the lidar system 304 may be configured to generate, through the use of the second sensor 240, map data of objects located in the surrounding area 250 of the vehicle 220. To receive both image data 310 and map data 312 of the predefined area 402 at the same time, the processor 110 may be configured to synchronize the operation of the camera system 302 and the lidar system 304 based on the absolute time indicator 320 received from the server 215. The following describes how the synchronization is implemented with reference to FIG. 7-8.

[0108] Referring now to FIG. 4 and 5. In FIG. 5, illustrated is a timing diagram 500 of operation, by processor 110, of first sensor 230 of camera system 302, in accordance with non-limiting embodiments of the present technology. The first sensor 230 is configured, via the processor 110, to operate in scan phases; in each scanning phase, the first sensor 230 takes a corresponding snapshot of the predetermined area 402, thereby generating image data 310 that includes a plurality of snapshots of the predetermined area 402.

[0109] The scan phase 502 of the first sensor 230 includes a capture phase 504 and a pause phase 506. The duration of the scan phase 502 depends on the operating frequency of the first sensor 230, which may be one snapshot of a predefined area 402 per 30 ms, for example. The capture phase 504 is the phase in which the first sensor 230 takes a given snapshot of the predefined area 402. The pause phase 506 is the phase in which the first sensor 230 prepares to take the next snapshot of the predefined area 402. How the phase The scan 502 is broken down into a capture phase 504 and a pause phase 506, is not limited, and as such, their duration can be predetermined by a particular configuration of the first sensor 230.

[0110] In non-limiting embodiments of the present technology, the duration of the idle phase 506 may be modified (eg, increased) which causes a corresponding change in the duration of the scan phase 502.

[0111] Referring to FIG. 3 and 5, in non-limiting embodiments of the present technology, the camera synchronization component 306 of the camera system 302 may be configured to receive, from the first sensor 230, data indicative of the start and end of the shooting phase 504. Additionally, the camera timing component 306 may be configured to pass received data to processor 110. Accordingly, processor 110, based on the common time reference 510 received from server 215, may be configured to generate camera timestamps 508 for subsequent scan phases of the first sensor 230 and transmit data indicating them back to the camera synchronization component 306 . Due to this, the processor 110 is configured to cause the camera system 302 to generate image data 310 of a predetermined area 402 in a common reference 510 in time.

[0112] In some non-limiting embodiments of the present technology, each of the camera timestamps 508 may be selected as the midpoint of the respective shooting phase. For example, the camera timestamp t ₁ of the camera timestamps 508 is selected as the midpoint of the shooting phase 504 . In non-limiting embodiments of the present technology, the first sensor 230 is configured to generate each snapshot of the image data 310, in the capture phase 504, by pixel lines: line by line, top to bottom. In this regard, the midpoint of the shooting phase 504 can be calculated by the processor 110 so that it corresponds to the point in time when the first sensor 230 scans half of the pixel lines of the corresponding snapshot of the image data 310.

[0113] However, the method of selecting a camera timestamp 508 by the processor 110 is not limited, and in other non-limiting embodiments of the present technology, may include selecting a given camera timestamp at the start or end of the corresponding capture phase.

[0114] Now, referring to FIG. 4 and 6. In FIG. 6, illustrated is a timing diagram 600 of operation, by means of processor 110, of second sensor 240 of lidar system 304, in accordance with non-limiting embodiments of the present technology. The second sensor 240 is configured to generate map data 602 of the surrounding area 250 of the vehicle 220 in rotation cycles. One rotation cycle (eg, rotation cycle 604), for example, may be 200 ms. The rotation cycle 604 corresponds to the time of a full 360 degree rotation of the second sensor 240 about its central vertical axis, which can be predetermined by the configuration of the second sensor 240. To this end, at each time in the rotation cycle 604, the second sensor 240 has a different corresponding angular position, after which the second sensor 240 is configured to generate map data of objects located in the respective areas of the surrounding area 250 of the vehicle 220.

[0115] Thus, in the embodiments of FIG. 4 and 6, three angular positions of the second sensor 240 are illustrated corresponding to three respective times in the rotation cycle 604 of the second sensor 240: the first angular position 408 at the first time T ₁ , the second angular position 410 at the second time T ₂ and the third angular position at the third moment T ₃ of time. In this regard, the sensor 234, at the first angular position 408, is configured to generate map data 312 of objects located in the predetermined area 402 at the first time T ₁ . At the second angular position 410 corresponding to the second time T ₂ , the second sensor 240 is configured to generate map data (not illustrated) of objects located in the second portion 404 of the surrounding area 250. Finally, at the third angular position 412 corresponding to the third time T ₃ times, the second sensor 240 is configured to generate map data (not illustrated) of objects located in the third area 406 of the surrounding area 250.

[0116] Also referring to FIG. 3, according to non-limiting embodiments of the present technology, the lidar timing component 308 is configured, by the processor 110, to record, at each of times T ₁ , T ₂ , and T ₃ , data indicative of a first angular position 408, a second angular position 410, and a third corner position 412, respectively. Additionally, the lidar synchronization component 308 is configured to transmit recorded data indicative of the angular positions of the second sensor 240 to the processor 110. The processor 110, based on the common time reference 510, is configured to generate lidar timestamps 608 for subsequent rotation cycles of the second sensor 240 and send back their indicator to the lidar synchronization component 308 . Each of the lidar timestamps 608 corresponds to a proper point in time in the rotation cycle of the second sensor 240 when the second sensor 240 is at the first angular position 408. For example, the lidar timestamp Tl ₂ of the lidar timestamps 608 corresponds to the first angular position of the second sensor 240 in the rotation cycle after the rotation cycle 604. Due to this, the processor 110 is configured to cause the lidar system 304 to generate map data 312 of a predetermined area 402 in a common reference 510 in time.

[0117] It should be clearly understood that the first angle position 408, the second angle position 410, and the third angle position 412 of the second sensor 240 are illustrated in FIG. 4 just as an aid to understanding the present technology; and in non-limiting embodiments of the present technology, the lidar synchronization component 308 may be configured to record and transmit data indicative of more than three angular positions of the second sensor 240, which may be in the hundreds, thousands, or even hundreds of thousands.

[0118] After generating camera timestamps 508 and lidar timestamps 608 based on a common time reference 510, processor 110 can further be configured to synchronize the scan phases of the camera system 302 and at least one subsequent rotation cycle of the lidar system 304.

[0119] With reference to FIG. 7 illustrates a combined timing diagram 700 including timing diagram 600 of lidar system 304 and timing diagram 500 of camera system 302'. According to the timing diagram 700, the camera system 302 is configured to generate each snapshot of the image data 310 at a corresponding one of the camera timestamps 508; and the lidar system 304 is configured to generate mapping data 312 of objects located in the predetermined area 402 at a corresponding one of the lidar timestamps 608 . Thus, both camera system 302 and lidar system 304 are driven, by processor 110, to provide image data 310 and map data 312 at a common time reference 510.

[0120] Assume that in the embodiments of FIG. 7, the camera timestamp t ₁ from the camera timestamps 508 corresponds to the current scanning phase of the camera system 302. From the timing diagram 700, it can be taken into account that the scan phases of the camera system 302 and the rotation cycles of the lidar system 304 are desynchronized because the camera timestamp t ₁ from the time the camera labels 508 do not match the lidar timestamp Tl ₁ of the lidar timestamps 608 . Accordingly, the processor 110 may be configured to synchronize the immediately following lidar timestamp Tl ₂ from the lidar timestamps 608 with its nearest successor from the camera timestamps 508, eg, the camera timestamp t ₇ . In other words, processor 110 may be configured to determine the lidar timestamp for synchronization as the nearest lidar timestamp corresponding to the point in time when the second sensor 240 is at the first angular position 408. To this end, in non-limiting embodiments of the present technology, the processor 110 also may be configured to receive, from the lidar timing component 308, an indicator of the first angular position 408.

[0121] In some non-limiting embodiments of the present technology, processor 110 may be configured to apply a timestamping algorithm to determine a lidar timestamp for synchronization.

[0122] First, according to non-limiting embodiments of the present technology, the processor 110 is configured to determine the time offset 702 between the timestamp t₇ cameras and timestamp Tl₂ lidar in a common binding of 510 in time. Second, once the time offset 702 is determined, the processor 110 is configured to use it to initiate a delay between successive snapshots of the image data 310 such that the timestamp t₇ cameras and timestamp Tl₂ lidars must match at a future point in time, thereby generating an adjusted schedule for the operation of the first sensor 230.

[0123] In some non-limiting embodiments of the present technology, processor 110 is configured to use real-time time offset 702 at the time the vehicle is moving 220. In other non-limiting embodiments of the present technology, processor 110 is configured to use time offset 702 at that time. the time when the vehicle 220 is not moving (ie stopped, parked, etc.).

[0124] With reference to FIG. 8 illustrates a timing diagram 800 of the operation of the camera system 302 and lidar system 304 in a common time reference 510. Timing chart 800 differs from timing chart 700 only in that timing chart 500 of camera system 302 has been replaced by timing chart 802.

[0125] In some non-limiting embodiments of the present technology, processor 110 may be configured to use time offset 702 by dividing it equally between all scan phases of camera system 302 prior to the lidar timestamp, whereby processor 110 is configured to perform synchronization, i.e., .e. in the embodiments of FIG. 8, timestamp Tl ₂ of the lidar. Therefore, from the timing diagram 802, it can be appreciated that the processor 110 can be configured to increase the pause phases of the corresponding scan phases corresponding to camera timestamps t ₁ , t ₂ , t 3 , t _{4 ,} t ₅ , and t ₆ of the camera timestamps 508 by time delay 804, where time delay 804 is one sixth of time offset 702.

[0126] Thus, in those non-limiting embodiments of the present technology, in which the rotation cycle of the second sensor 240 is equal to a predetermined scan phase multiplier of the first sensor 230, the processor 110 is configured to use the time offset 702 by evenly distributing it between a predetermined number -multiplier of subsequent scan phases of the first sensor 230 preceding the lidar timestamp Tl ₂ in a common time reference 510.

[0127] Thus, by adding a time delay 804 to all scan phases preceding the lidar timestamp Tl ₂ in the common time reference 510 of the camera system 302, the processor 110 is configured to shift the camera timestamp t ₇ such that it should coincide with the timestamp Tl ₂ of the lidar.

[0128] In other non-limiting embodiments of the present technology, processor 110 is configured to use time offset 702 to initiate a delay between only two given consecutive snapshots of image data 310. To this end, the processor 110 may be configured to increase the pause phase in the operation of only one scan phase corresponding to a proper one of the camera timestamps 508 preceding the lidar timestamp Tl ₂ in the common time reference 510.

[0129] In some non-limiting embodiments of the present technology, processor 110 may be configured to use time offset 702 to initiate a delay between successive snapshots of image data 310 by implementing a frame-level delay that may be inserted between the end of reading the last row and the start of flushing the first lines of the next frame. This is typically referred to as the horizontal blanking period in camera manufacturer's datasheets.

[0130] If such an option is not available, in some non-limiting embodiments of the present technology, the processor 110 may be configured to use the time offset 702 to initiate a delay between subsequent snapshots of the image data 310 by causing the camera synchronization component 306 to generate an appropriate electrical signal at times , corresponding to the appropriate ones from the plurality of camera timestamps 508 . Accordingly, the first sensor 230 is configured to perform a subsequent snapshot of the image data 310 in a corresponding electrical signal generated by the camera timing component 306 . In some non-limiting embodiments of the present technology, causing the camera sync component 306 to generate an appropriate electrical signal is performed using a start of frame sync (FSIN).

[0131] Referring again to FIG. 3, after modifying the scan phases of the camera system 302 preceding the lidar timestamp Tl ₂ in the common time reference 510, the processor 110 is configured to transmit the modified scan phases, i.e. adjusted schedule, to the timing component 306 of the camera system 302 for operating the first sensor 230, thereby modifying its predetermined camera scan rate. Thus, the camera system 302 must generate at least one subsequent snapshot of the image data 310 (corresponding to the camera timestamp t ₇ ), and the lidar system 304 generates mapping data 312 of objects located in the predetermined area 402 at the same future time, based on a common reference 510 in time.

[0132] In some non-limiting embodiments of the present technology, processor 110 may further be configured to repeat the synchronization procedure described above with reference to FIG. 4-8, with some periodicity. For example, processor 110 may be configured to repeat the timing procedure for each successive rotation cycle of second sensor 240, i. e. synchronization of at least one successive of the camera timestamps 508 with each successive of the lidar timestamps 608. In other words, synchronization of image data 310 and map data 312 may be performed substantially continuously during vehicle 220 operation, i. in each subsequent future cycle of rotation of the second sensor 240.

[0133] In other non-limiting embodiments of the present technology, the processor 110 may be configured to repeat the synchronization procedure after a predetermined number of rotation cycles of the second sensor 240.

[0134] In some non-limiting embodiments of the present technology, such a specific time offset 702 is used for one rotation cycle of the second sensor 240. In other non-limiting embodiments of the present technology, such a specific time offset 702 is used for a predetermined number of future rotation cycles of the second sensor 240. Accordingly, processor 110 may be configured, in a subsequent rotation cycle of second sensor 240, to iteratively recalculate time offset 702 in real time, during operation of vehicle 220, following the above description with respect to FIG. 6-8.

[0135] Thus, in non-limiting embodiments of the present technology, after causing the camera system 302 to provide image data 310 at a common time reference 510 in a given scan phase (e.g., in the scan phase corresponding to timestamp t ₁ of camera timestamps 508), and compute time offset 702, processor 110 is configured to use it for at least a predetermined number of subsequent scan phases of camera system 302 (e.g., five (5) subsequent scan phases as illustrated in the embodiments of FIGS. 7 and 8) for synchronization in only one subsequent rotation cycle of the second sensor 240 (eg, in the rotation cycle corresponding to the lidar timestamp Tl ₂ of the lidar timestamps 608).

[0136] In other non-limiting embodiments, processor 110 is configured to use time offset 702 for an additional predetermined number of scan phases of camera system 302 to synchronize within more than one subsequent rotation cycle of second sensor 240.

[0137] Given the architecture and examples provided above, a data synchronization method for operating an SDV (eg, vehicle 220) can be implemented. With reference now to FIG. 9 illustrates a flow diagram of a method 900 according to non-limiting embodiments of the present technology. Method 900 is performed by processor 110. Processor 110 may be part of electronic device 210, for example.

[0138] In non-limiting embodiments of the present technology, processor 110 is communicatively coupled to a camera system and a lidar system (e.g., camera system 302 and lidar system 304 illustrated in FIG. 3) mounted on vehicle 220. Camera system 302 includes including the first sensor 230 and the camera synchronization component 306 . The lidar system 304 includes a second sensor 240 and a lidar synchronization component 308.

[0139] The camera system 302 is configured to capture, by way of the first sensor 230, image data 310 of a predetermined area 402 of the surrounding area 250 of the vehicle 220. points in time at a predetermined camera scan rate. In this regard, in some non-limiting embodiments of the present technology, the first sensor 230 of the camera system 302 has a field of view that coincides with the predefined area 402 when the camera system 302 is installed on the vehicle 220.

[0140] As described in detail with reference to FIG. 4 and 5, the camera system 302 is configured to operate in scan phases whose duration indicates a predetermined camera scan rate. Each scan phase has a capture phase and an operational pause phase (eg, scan phase 502 includes a capture phase 504 and an operational pause phase 506). In the capture phase 504, the camera system 302 is configured to generate a given snapshot of the image data 310. In the pause phase 506, the camera system 302 is idle (ie, not taking snapshots) in order to prepare to take a subsequent snapshot. Both the capture phase 504 and the idle phase 506 may be predetermined by a specific configuration of the camera system 302. In some non-limiting embodiments of the present technology, the idle phase 506 may be modified (eg, increased).

[0141] The lidar system 304 is configured to generate, by means of the second sensor 240, map data of the surrounding area 250 of the vehicle 220. The map data of the surrounding area 250 of the vehicle 220 comprises data points generated by the second sensor 240. The data points represent objects located in the surrounding area 250 of the vehicle 220 at various points in time. The map data of the surrounding area 250 includes the map data 312 of the predefined area 402.

[0142] In some non-limiting embodiments of the present technology, the lidar system 304 may be a rotational lidar system. In this regard, the lidar system 304 is configured to generate mapping data of the surrounding area 250 of the vehicle 220 in rotation cycles, with each rotation cycle corresponding to a full 360 degree rotation of the second sensor 240 about its central vertical axis, for example, in direction 414, as illustrated. in fig. 4. In this regard, the lidar system 304 is configured, at each angular position of the second sensor 240, to generate cartographic data of objects located in the corresponding area of the surrounding area 250 of the vehicle 220.

[0143] As described with reference to FIG. 4 and 6, in non-limiting embodiments of the present technology, the lidar system 304 is configured to generate mapping data 312 of objects located in a predefined area 402 of the surrounding area 250 when the second sensor 240 is at the first angular position 408.

STEP 902 - CAUSING, THROUGH THE ELECTRONIC DEVICE, THE CAMERA SYSTEM, AND THE LIDAR SYSTEM, TO PROVIDE IMAGE DATA AND MAP DATA TO THE ELECTRONIC DEVICE AT A COMMON TIME ORIGIN

[0144] In step 902, the processor 110 is configured to cause the camera system 302 and the lidar system 304 to provide image data 310 of a predefined area 402 and map data 312 of objects located in the predefined area in a common time reference, for example, in a common reference 510 in time.

[0145] To this end, first, the processor 110 is configured to receive an absolute time indicator (eg, an absolute time indicator 320 received from the server 215). As detailed above with reference to FIG. 3, the absolute time indicator 320 may be an astronomical time indicator.

[0146] Second, as detailed above with reference to FIG. 3 and 5, the synchronization component 306 of the camera system 302 is configured to register data indicating the start and end of the shooting phase 504 of the scanning phase 502 and to transmit this data, via the communication line 314, to the processor 110. Additionally, the processor 110 is configured, based on received data, generate camera timestamps 508 in a common time reference 510 for subsequent scan phases. Finally, processor 110 is configured to transmit, via link 314, data indicative of generated camera timestamps 508 back to camera sync component 306 .

[0147] In some non-limiting embodiments of the present technology, each of the camera timestamps 508 may be calculated, by the processor, as the midpoint of the shooting phase 504, as detailed above with reference to FIG. 5.

[0148] With respect to the lidar system 304, first, the lidar timing component 308 is configured to record data indicative of a plurality of angular positions of the second sensor 240 and corresponding times in one rotation cycle (e.g., in rotation cycle 604 as illustrated in FIG. 6). Second, as detailed with reference to FIG. 3, 4, and 6, the lidar timing component 308 is configured to transmit this data, via the link 316, to the processor 110. In response, the processor 110, based on the transmitted data, is configured to generate a plurality of lidar timestamps 608 in a common reference 510 by time. Each of the plurality of lidar timestamps 608 corresponds to a future point in time when the second sensor 240 should be at the first angular position 408 in a subsequent rotation cycle of the second sensor 240. In other words, each of the plurality of lidar timestamps 608 corresponds to the point in time when the second sensor 240 generates map data 312 of objects located in a predetermined area 402 of the surrounding area 250. Finally, the processor 110, via the link 316, is configured to transmit a plurality of lidar timestamps 608 to the lidar synchronization component 308 .

[0149] In non-limiting embodiments of the present technology, link 314 and link 316 are based on the PTP protocol described above with reference to FIG. 3.

STEP 904 - ELECTRONICALLY DEFINING FIRST TIME STAMP FOR MAP DATA

[0150] At step 904, the processor is configured to determine the next lidar timestamp from the plurality of lidar timestamps 608 by which processor 110 is to synchronize capture of image data 310 and capture of map data 312. To this end, processor 110 is configured to receive, from the lidar synchronization component 308, the first angular position indicator 408 of the second sensor 240.

[0151] In some non-limiting embodiments of the present technology, processor 110 is configured to determine a subsequent lidar timestamp as the immediately following lidar timestamp from a plurality of lidar timestamps 608, such as the lidar timestamp Tl ₂ in the embodiments of FIG. 7.

[0152] In other non-limiting embodiments of the present technology, processor 110 is configured to use a timestamping algorithm to determine a lidar timestamp to synchronize image data 310 and map data 312.

STEP 906 - ELECTRONICALLY DETERMINING A SECOND TIME STAMP FOR IMAGE DATA

[0153] At 906, the processor is configured to determine the next camera timestamp from the camera timestamps 508. In some non-limiting embodiments of the present technology, processor 110 determines the subsequent camera timestamp as the subsequent camera timestamp from the camera timestamps 508 preceding the lidar timestamp Tl ₂ in a common time reference 510, which in the embodiments of FIG. 7 is the timestamp t ₇ of the camera.

STEP 908 - ELECTRONICALLY DEFINING THE TIME OFFSET BETWEEN THE FIRST TIME STAMP AND THE SECOND TIME STAMP

[0154] In step 908, after determining the lidar timestamp Tl ₂ and the camera timestamp t ₇ for synchronizing image data 310 and map data 312, processor 110 is further configured to calculate a time offset between them, which is roughly illustrated by time offset 702 and described above with reference to FIG. 7.

[0155] In non-limiting embodiments of the present technology, time offset 702 indicates the difference between times, in a common time reference 510, appropriately corresponding to the lidar timestamp Tl ₂ and the camera timestamp t ₇ .

STAGE 910 - the use of temporary displacement through an electronic device in order to initiate a delay between subsequent instant images formed by means of a chamber system, so that (I) at least one subsequent instant picture is formed through a chamber system, and (II) subsequent data points relatively previously defined SECTIONS ARE FORMED BY THE LIDAR SYSTEM AT THE IDENTICAL FUTURE TIME AT THE COMMON ORIGIN OF TIME

[0156] At step 910, the processor is configured to use the time offset 702 to initiate a delay between subsequent snapshots of the image data 310 such that the lidar timestamp Tl ₂ and the camera timestamp t ₇ must match at a future time, and therefore, the capture of image data 310 and the capture of map data 312 are synchronized by this future point in time, thereby generating an adjusted schedule for the operation of the first sensor 230.

[0157] In non-limiting embodiments of the present technology, processor 110 is configured to utilize time offset 702 by increasing idle phases of operation from corresponding scan phases of first sensor 230, thereby modifying a predetermined camera scan rate.

[0158] In some non-limiting embodiments of the present technology, processor 110 is configured to extend the pause phase of only one corresponding subsequent scan phase. In these embodiments, the processor 110 is configured to initiate a delay between only two subsequent survey phases of the respective scan phases of the first sensor 230 preceding the lidar timestamp Tl ₂ in the common time reference 510.

[0159] In other non-limiting embodiments of the present technology, the processor 110 is configured to divide the time offset 702 equally between subsequent scan phases of the first sensor 230 preceding the lidar timestamp Tl ₂ in the common time reference 510, and further equally increase each corresponding pause phase in work. In these embodiments, the processor 110 is configured to initiate an equal delay between each two successive survey phases of the respective scan phases of the first sensor 230 preceding the lidar timestamp Tl ₂ in the common time reference 510.

[0160] In this regard, in those non-limiting embodiments of the present technology, in which the rotation cycle of the second sensor 240 is equal to a predetermined number-multiplier of the scan phases of the first sensor 230, the processor 110 is configured to use the time offset 702 by evenly distributing it between the predetermined a certain multiplier of the subsequent scan phases of the first sensor 230 preceding the lidar timestamp Tl ₂ in the common time reference 510.

[0161] In non-limiting embodiments of the present technology, processor 110 is further configured to transmit such modified scan phases, i. the adjusted schedule for the operation of the first sensor 230, to the camera synchronization component 306 .

[0162] In non-limiting embodiments of the present technology, processor 110 is configured to perform the synchronization procedure described above, including using time offset 702, during operation of vehicle 220, i. in real time, for example, while the vehicle 220 is moving.

[0163] In non-limiting embodiments of the present technology, processor 110 may be configured to recalculate time offset 702, thereby repeating the synchronization procedure described above.

[0164] In some embodiments of the present technology, the processor 110 is configured to repeat the synchronization procedure before each subsequent lidar timestamp from the plurality of lidar timestamps 608, i. for each subsequent rotation cycle of the second sensor 240. Thus, it can be said that the processor 110 can be configured to synchronize the capture of image data 310 and the capture of map data 312 substantially continuously during the operation of the vehicle 220.

[0165] In other non-limiting embodiments of the present technology, the processor 110 is configured to repeat the synchronization procedure after a predetermined number of subsequent rotation cycles of the second sensor 240.

[0166] Thus, in non-limiting embodiments of the present technology, after causing the camera system 302 to provide image data 310 at a common time reference 510 in a given scan phase (e.g., in the scan phase corresponding to timestamp t ₁ of camera timestamps 508), and calculate the time offset 702, the processor 110 is configured to use it for at least a predetermined number of subsequent scanning phases of the camera system 302 (e.g., five (5), in the embodiments of Figs. 7 and 8) to synchronize in only one subsequent cycle rotation of the second sensor 240 (eg, in a rotation cycle corresponding to the lidar timestamp Tl ₂ of the lidar timestamps 608).

[0167] In other non-limiting embodiments, processor 110 is configured to use time offset 702 for an additional predetermined number of scan phases of camera system 302 to synchronize, using such defined time offset 702, within more than one subsequent rotation cycle of second sensor 240.

[0168] It should be clearly understood that not all technical effects mentioned in this document should be used in every embodiment of the present technology.

[0169] Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The above description is intended to be exemplary and not limiting. As such, the scope of the present technology may be limited solely by the scope of the appended claims.

Claims (64)

1. A computerized data synchronization method for operating an unmanned vehicle (SDV), wherein the data is generated by a camera system and a lidar system, the camera system and the lidar system being communicatively coupled to an electronic device, where the data includes: (i) image data of a camera system installed on the SDV to generate image data about a predetermined area of the SDV's surroundings, wherein the camera data is in the form of a sequence of snapshots generated by the camera system at a camera scan rate, moreover, the sequence of snapshots represents a predetermined area at various points in time, (ii) mapping data from a lidar system installed on the SDV to generate map data about the vicinity of the SDV, including a predetermined area, wherein the mapping data is in the form of data points generated by the lidar system, moreover, the data points represent objects located in different parts of the SDV neighborhood at different points in time, wherein the method is carried out by means of an electronic device, the method comprising the steps of: causing, by means of the electronic device, the camera system and the lidar system to provide image data and mapping data to the electronic device at a common time reference, moreover, the common time reference is an indicator of the absolute time during which the data is generated; determining by means of the electronic device a first timestamp for the map data, wherein the first timestamp corresponds to a given point in time in a common time origin during which the lidar system generates data points representing objects located in a predetermined area; determining by means of the electronic device a second timestamp for the image data, wherein the second timestamps correspond to another given point in time at the common time origin during which the camera system generates a given snapshot of the predetermined area; determining by means of an electronic device a time offset between the first timestamp and the second timestamp, moreover, the time offset indicates the difference between a given point in time and another given point in time in the common origin of time; use by means of an electronic device a time offset in order to initiate a delay between successive snapshots generated by the camera system, moreover, the camera scan rate is adjusted by dividing the duration of the time offset between several subsequent pictures to add an appropriate time delay to them, so that (i) at least one subsequent snapshot is formed by the camera system, and (ii) subsequent data points relative to a predetermined area are formed by the lidar system at an identical future time point at a common time origin. 2. The method of claim. 1, in which the use of a temporary offset is performed during the operation of the SDV. 3. The method of claim 1, wherein data synchronization for SDV operation is performed substantially continuously during SDV operation. 4. The method of claim 1, wherein causing the camera system to provide image data is performed during the first iteration of the camera system operation, and the use is performed for a subsequent camera system operation. 5. The method of claim 4, wherein the subsequent operation is a predetermined number of subsequent iterations. 6. The method of claim 5, wherein the method of claim 1 is repeated after a predetermined number of subsequent iterations for an additional number of subsequent iterations. 7. The method of claim 1, wherein the camera system has a field of view that matches a predetermined portion of the SDV's neighborhood when mounted on the SDV. 8. The method of claim 1, wherein the prompting includes using the PTP protocol (Precision Time Protocol) by the electronic device to transmit an absolute time indicator to the camera system and the lidar system. 9. The method of claim 1, wherein the absolute time indicator is an astronomical time indicator. 10. The method of claim 1, wherein the lidar system is a rotational lidar system, such that the lidar system generates data points representing objects located in a first region of the SDV neighborhood at a first point in time, and wherein the first region is associated with an angular position of the lidar system at the first moment of time. 11. The method of claim 10, wherein the lidar system generates data points representing objects located in the second region of the SDV neighborhood at the second point in time, and wherein the second region is associated with the angular position of the lidar system at the second point in time, moreover, the angular position of the lidar system differs at the first moment of time and at the second moment of time, while the first section differs from the second section. 12. The method of claim 10, wherein determining the first timestamp comprises the steps of: receiving by means of an electronic device an indicator of a predetermined angular position of the lidar system, so that when the lidar is at a predefined angular position, the lidar generates data points about objects located in the predefined area, determining by means of an electronic device the first timestamp as the point in time in the common time reference at which the lidar system is in a predetermined angular position. 13. The method of claim 12, wherein the determination of the first timestamp is performed by the electronic device using a timestamp determination algorithm. 14. The method of claim. 1, in which this snapshot is formed by the camera system during the time interval for forming a snapshot, while the second timestamp is a point in time in a common time reference that corresponds to a midpoint in the time interval for forming snapshot. 15. The method of claim 1, wherein using the time offset electronically to initiate a delay between successive snapshots generated by the camera system comprises: causing the camera system to add a snapshot time delay after a given future snapshot. snapshot. 16. The method of claim 15, wherein the snapshot time delay is a set of snapshot time delays added after respective future snapshots that are equal in time to the duration of the time offset. 17. The method of claim 16 wherein the lidar system operates in rotation cycles indicating the lidar scan rate and the camera system operates in scan phases indicative of the camera scan rate, and wherein the rotation cycle of the lidar system is equal to a predetermined phase multiplier. scanning the camera system, wherein the plurality of snapshot time delays are distributed over a number of future snapshots corresponding to a predetermined multiplier of the camera system scan phases. 18. An electronic device containing: CPU; a machine-readable non-volatile memory carrier containing data, wherein the data is generated by a camera system and a lidar system connected in communication, via a communication interface, to the processor, where the data includes: (i) image data of a camera system installed on the SDV to generate image data about a predetermined area of the SDV's surroundings, wherein the camera data is in the form of a sequence of snapshots generated by the camera system at a camera scan rate, moreover, the sequence of snapshots represents a predetermined area at various points in time, (ii) mapping data from a lidar system installed on the SDV to generate mapping data regarding the vicinity of the SDV, including a predetermined area, wherein the mapping data is in the form of data points generated by the lidar system, moreover, the data points represent objects located in different parts of the SDV neighborhood at different points in time, wherein the electronic device is mounted on an unmanned vehicle (SDV); wherein the processor is configured to: cause the camera system and the lidar system to provide image data and mapping data at a common time reference, moreover, the common origin of time is an indicator of the absolute time during which the data is generated; determine the first timestamp for map data, wherein the first timestamp corresponds to a given point in time in a common time origin during which the lidar system generates data points representing objects located in a predetermined area; determine a second timestamp for the image data, wherein the second timestamps correspond to another given point in time at the common time origin during which the camera system generates a given snapshot of the predetermined area; determine by means of an electronic device the time offset between the first timestamp and the second timestamp, moreover, the time offset indicates the difference between a given point in time and another given point in time in the common origin of time; use a time offset to initiate a delay between successive snapshots generated by the camera system, adjust the camera scan rate by dividing the duration of the time shift between several subsequent shots to add an appropriate time delay to them, so that (i) at least one subsequent snapshot is generated by the camera system, and (ii) subsequent data points about the predetermined area are generated by the lidar system at the same future time point at the common time origin. 19. The electronic device of claim 18, wherein the processor is further configured to use the time offset during operation of the SDV. 20. The electronic device of claim 18, wherein the processor is further configured to cause the camera system to provide image data during the first iteration of the camera system operation, is further configured to use the time offset for subsequent camera system operation.

Images