KR20180071962A - Multiscale weighted matching and sensor fusion for dynamic vision sensor tracking - Google Patents

Abstract

A dynamic vision sensor (DVS) pose estimation system comprises a sensor fusion-based DVS, a conversion estimator, an inertia measurement unit (IMU), and a camera pose estimator. The DVS detects DVS events and forms frames based on the number of DVS events. The conversion estimator estimates 3D conversion of a DVS camera based on an estimated depth and performs matching of confidence level values in a camera projection model to make at least one of the DVS events, detected during the first frame, correspond to a DVS event detected during the second frame. The IMU detects inertia movements of the DVS with respect to world coordinates between the first frame and second frame. The camera pose estimator combines information from a change of pose of the camera projection model between the first frame and second frame based on the estimated change and detected inertia movements of the DVS.

Description

[0001] MULTISCALE WEIGHTED MATCHING AND SENSOR FUSION FOR DYNAMIC VISION SENSOR TRACKING [0002]

The present invention relates to a dynamic vision sensor (DVS), and more particularly, to an apparatus and method for estimating a pose of a DVS.

에 의해 나타날 수 있다. 또는, 0이 아닌 픽셀은 그 위치에서 나타나는 이벤트들의 수 또는 최신 이벤트의 도달 시간에 의해 나타날 수 있다.The output of the DVS changes based on the event of the brightness sensed by the camera. In general, the output of the DVS is an event stream, with each event associated with a particular state, i. E., An event location within a binary state that represents a change in the amount or < / RTI > A certain number of DVS events are sampled to form an image in which pixel locations containing one or more events are set to non-zero and other pixel locations are all set to zero. The value of each non-zero pixel may be determined by different techniques. For example, each non-zero pixel may contain a time stamp, pixel coordinates, and the most recent event state change (i. E., A change in the amount of brightness is +1, a change in brightness is -1) vector

Lt; / RTI > Alternatively, a non-zero pixel may be indicated by the number of events appearing at that location or the arrival time of the latest event.

Since the conventional DVS is an asynchronous sensor with no time integration, DVS frames have to be formed based on a specific sampling time or frame integration time, so that changes between temporally adjacent frames can be compared with the estimated camera motion. The major difficulties associated with DVS camera motion or tracking are: (1) features within each DVS frame may be sparse and highly deformable, so that feature-based image matching may be difficult and may impair motion estimation accuracy And (2) due to the lack of extraction of key features, the corresponding landmarks can not be used through the movement of the DVS. Thus, it is difficult to contrast the current estimates of camera motion or pose, and it may be difficult to refer to landmarks to reduce sensor motion estimation variance.

An embodiment of the present invention can provide a DVS, a 3D conversion estimator, an inertial measurement unit, and a camera pose estimator based on sensor fusion.

The DVS can detect DVS events and form frames based on the number of accumulated events. The transform estimator estimates the 3D transform based on the matching of the estimated depth and the confidence level values in the camera model so that at least one of the plurality of DVS events detected during the first frame is detected during the second frame following the first frame It is possible to correspond to the DVS event. The inertial measurement unit may detect inertial movements of the DVS for world coordinates between the first and second frames. The camera pose estimator may combine information from a change in the pose of the DVS camera between the first frame and the second frame based on the estimated transform and detected inertial movements. In one embodiment, the camera pose estimator may use inertial motions of the detected DVS to correct for an estimated change in the pose of the transformation model between the first frame and the second frame based on the estimated transformation. In one embodiment, the transform estimator may estimate the depth, match the camera projection model for the first and second frames for a plurality of frame integration times, and the transform estimator may also estimate The first estimate of the change in the camera pose between the first frame and the second frame corresponding to the first frame integration time based on the estimation of the change in the camera pose between the first frame and the second frame . In one embodiment, the transform estimator

는 인덱스이고,

는 검출된 DVS 이벤트이고,

는 DVS 이벤트(

)에 대한 신뢰 수준 값(스칼라) 이고,

는 프레임 이고,

는 프레임 인덱스이고,

는 DVS에 대한 카메라 프로젝션 모델이고,

은 DVS에 대한 카메라 프로젝션 모델의 역수 이고,

는 카메라 프로젝션 모델(

) 내의 검출된 이벤트(

)에 대응하는 벡터고,

는 세계 좌표에서 상기 DVS 이벤트(

)의 (3D) 깊이이고,

는 프레임

및 프레임

사이의 세계 좌표들에 기초한 복수의 가능한 벡터 변환들 중 하나이다.Lt; RTI ID = 0.0 > 1, < / RTI >

Is an index,

Is a detected DVS event,

DVS event (

(Scalar), < / RTI >

Is a frame,

Is a frame index,

Is a camera projection model for DVS,

Is the reciprocal of the camera projection model for DVS,

Is a camera projection model

The detected event (

), ≪ / RTI >

Lt; RTI ID = 0.0 > DVS < / RTI &

(3D) depth,

Frame

And frame

Lt; / RTI > is one of a plurality of possible vector transformations based on world coordinates between.

One embodiment includes detecting DVS events; At least one of the plurality of DVS events observed during the first frame matches the DVS event detected during the second frame by matching the confidence level values in the camera projection model based on the estimated depth of the camera, And the second frame follows the first frame; Detecting inertial movements of the DVS for world coordinates between the first and second frames; And estimating a change in the camera pose between the first frame and the second frame based on the estimated transform and the inertial movements of the detected DVS.

One embodiment provides a DVS pose estimation system that may include a DVS, a transform estimator, and a camera pose estimator pose estimator based on a multi-scale time scheme. The DVS can detect DVS events within the camera projection model for the DVS camera. The transform estimator estimates a transformation of the camera based on the matching of the confidence level values of the camera projection model and the estimated depth to determine that at least one of the plurality of DVS events detected during the first frame is the second A DVS event detected during the frame, and a second frame following the first frame for each frame integration time. The camera pose estimator may estimate the change in the pose of the camera between the first frame and the second frame for at least two of the plurality of frame integration times, When the time is longer than the second frame integration time, for the first integration time, based on the estimation of the change in the pose of the camera between the first frame and the second frame for the second frame integration time, It is possible to correct the estimation of the change in the pose of the camera between frames.

According to an embodiment of the present invention, an apparatus and method for accurately estimating a pose of a DVS can be provided.

)이 일 실시 예에서 결정될 수 있는 방법을 도시한다.
도 4는 본 명세서에서 개시된 주제에 따른, DVS 카메라 포즈를 추정하는 예시적인 프로세스를 도시한다.
도 5는 본 명세서에서 개시된 주제에 따른, DVS 포즈 추정 시스템을 포함하는 하나 이상의 집적 회로들(칩들)을 포함하는 전자 장치를 도시한다.
도 6은 본 명세서에서 개시된 주제에 따른, DVS 포즈 추정 시스템을 포함할 수 있는 메모리 시스템을 도시한다.Figure 1 shows a block diagram of a DVS pose estimation system according to the subject matter disclosed herein.
FIG. 2 illustrates two exemplary simultaneous frame integration times for the same DVS event stream or the same inertial measurement section output data stream, according to the subject matter disclosed herein.
Figure 3 shows the confidence level value

) Can be determined in one embodiment.
4 illustrates an exemplary process for estimating a DVS camera pose, in accordance with the subject matter disclosed herein.
FIG. 5 illustrates an electronic device comprising one or more integrated circuits (chips) including a DVS pose estimation system, according to the subject matter disclosed herein.
Figure 6 illustrates a memory system that may include a DVS pose estimation system, in accordance with the subject matter disclosed herein.

In the following text, the various detailed descriptions are presented to provide an understanding of the text. It will be appreciated, however, that the spirit of the invention disclosed without this detailed description can be readily implemented by those skilled in the art. In other instances, well-known methods, procedures, structures, and circuits have not been described so as not to obscure the text.

Reference in the specification to "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic associated with the embodiment may be included in at least one embodiment of the text. That is, the terms " in one embodiment "or" in one embodiment "or" according to one embodiment "or similar expressions used in various places in the text do not require reference to the same embodiment . Moreover, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word " exemplary " means " serving as an example, instance, or illustration. Alternatively, in accordance with what is stated in the text, the singular terms include plural forms, and the plural terms may include singular forms. The various figures, including the construction drawings, are mentioned, illustrated and not quantified in the text for purposes of illustration only. For example, the order of some components may be exaggerated for other components for clarity. Also, where properly considered, reference numerals may be repeated among the figures to indicate corresponding and / or like elements.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular form " a " is intended to include plural forms unless the context clearly dictates otherwise. When used in this specification, terms such as " comprising, " " comprising, " / Or the components specify presence, but do not preclude the addition or presence of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Terms such as " first "and" second "are used as labels for the configurations described above and are not intended to be in any particular order (e.g., spatial, temporal, logical, etc.) unless otherwise defined. Moreover, the same reference numbers may be used throughout the two or more drawings to refer to parts, structures, blocks, circuits, units, or modules having the same or similar function. It is to be understood, however, that such use is merely for brevity's sake of simplicity and convenience, and that the organization and structural details of such arrangements and units are not intended to be identical in all embodiments or commonly referenced parts / Is one means for designating certain embodiments of the invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. These commonly used predefined terms are to be interpreted as having a meaning consistent with their meaning in the context of this specification and / or the related art, and are to be interpreted as being idealized or overly formal, unless explicitly defined herein It should not be interpreted.

As used herein, the term " module " refers to any combination of software, firmware, and / or hardware configured to provide the functions described herein in connection with the module. As applied to any of the implementations described herein, the term " software " may be embodied as a software package, code, and / or a set of instructions or instructions. As applied to any implementation described herein, the term " hardware " is intended to encompass all types of circuitry, such as, by way of example, singly or in any combination, implemented by wired circuitry, programmable circuitry, state machine circuitry, and / Firmware that stores instructions. The modules may be implemented as firmware and / or hardware that constitutes part of a larger system, such as, but not limited to, software, integrated circuit (IC) and system on chip .

The subject matter disclosed herein relates to a dynamic vision sensor that provides a pose estimation of a DVS camera based on matching of a depth map of a camera with an adaptive sampling time and a confidence map of temporal correlation between detected events in a camera projection model (DVS) pose estimation system. Based on the confidence map, weighted image matching is performed between the two frames to estimate the pose and position of the DVS. Enhanced image matching based on a confidence map provides greater matching accuracy over a larger number of frames than conventional matching techniques. That is, the benchmark tracking system based on the matching of local features is not appropriate, and there may be a lack of reference to the global map, because the local features are rare and highly variant in DVS frames, resulting in estimation variances. The subject matter disclosed herein uses multiscale temporal resolution techniques to maximize the likelihood of estimated camera poses and to reduce estimation variance. To further prevent system divergence and time integration deviations, inertial sensor sensors can be integrated with DVS to optimize system accuracy.

The subject matter disclosed herein also provides a DVS tracking system that provides sensor fusion for DVS camera tracking and multi-scale highlight matching between frames. In one embodiment, a multi-scale matching technique is used to reduce DVS pose estimation errors that may be caused by high feature / event changes across DVS frames. To avoid system divergence, reduce time integration deviations, and optimize tracking accuracy, IMU sensor data may be included in the DVS pose estimation.

In one embodiment, the enhanced image matching technique based on the image representation of the confidence map leads to a higher matching accuracy that provides better DVS image recognition and DVS slams (Simultaneous Localization and Mapping; SLAM). In addition, the multi-scale image matching technique reduces the DVS pose estimation error. The convergence of the DVS pose estimation system and the IMU sensor further avoids system divergence, reduces time integration deviations, and optimizes tracking accuracy.

1 shows a block diagram of a DVS pose estimation system 100 according to the subject matter disclosed herein. The system 100 includes a DVS 101, a frame shaper 102, a transform estimator 103, a camera pose estimator 104, an inertial movement unit (IMU) 105, and an integrator 106 . It should be understood that the various blocks shown in FIG. 1 may be implemented as physical hardware modules, software modules, or a combination (firmware) of physical hardware modules and software modules.

에 의해 모델링 될 수 있다. DVS(101)은 DVS 이벤트 데이터 스트림(107)을 프레임 셰이퍼(102)로 출력한다. DVS(101) 데이터 스트림(107)은, 각각의 이벤트가 특정한 상태, 즉 이미지 센서 어레이 내의 이벤트 위치 및 휘도의 양 또는 음의 변화를 나타내는 이진 상태와 관련된 이벤트들의 스트림을 포함한다.The DVS 101 detects DVS events in the field of view of the DVS 101. The field of view of the DVS 101 is the camera projection model for the DVS 101

Lt; / RTI > The DVS 101 outputs the DVS event data stream 107 to the frame shaper 102. The DVS 101 data stream 107 includes a stream of events associated with a binary state in which each event indicates a particular state, i.e., an event location within the image sensor array and a change in the amount or the brightness of the luminance.

The frame shaper 102 receives frame selection information 108 that sets the frame integration time applied to the DVS event data stream 107. As used herein, the term " frame integration time " refers to a selectable time period during which DVS events are gathered and combined together to form a frame. For example, in one embodiment a frame integration time of 50 ms may be selected and the DVS events are combined for the duration of each 50 ms and combined to be in the same frame. In another embodiment, a frame integration time of 100 ms may be selected and the DVS events are combined for the duration of each 100 ms and combined to be in the same frame. It is to be understood that 50 ms or 100 ms and other frame integration times are possible. Also, in one embodiment, at least two frame integration times may be selected at the same time.

Figure 2 illustrates two exemplary simultaneous frame integration times for the same DVS event stream or the same IMU output data stream, according to the subject matter disclosed herein. 2, the frame integration time of 50 ms and the frame integration time of 100 ms can be selected at the same time, and the DVS events (and / or the IMU output data) can be synchronized with the frames corresponding to the two frame integration times Can be combined at the same time. In another embodiment, two or more concurrent frame integration times may be selected.

) 내에서 컨피던스 맵을 생성한다. 프레임(

)내의 이벤트(

)가 물리적으로 이전 프레임(

) 내의 이벤트(

)에 대응하는 신뢰도를 나타내는, 이전 프레임(

)의 각각의 검출된 DVS 이벤트에 대해 프레임(

) 내의 각각의 검출된 DVS 이벤트에 대한 신뢰 수준 값(

)이 결정된다.The frame shaper 102 outputs frames of DVS events to the transform estimator 103 for each selected frame integration time. The transform estimator 103 estimates the transformation of the DVS events from one frame to the next for each selected frame integration time. As part of the estimation process, the transform estimator 103 estimates the camera projection model for the DVS 101 based on the temporal correlation between the detected DVS events from one frame to another

) To generate a confidence map. frame(

) Within the

) Physically moves to the previous frame (

) Within the

≪ / RTI > representing the reliability of the previous frame < RTI ID = 0.0 &

For each detected DVS event of frame < RTI ID = 0.0 >

) ≪ / RTI > for each detected DVS event

) Is determined.

)이 어떻게 결정될 수 있는지를 도시한다. 현재 프레임(

) 내의 이벤트(

)에 대해,FIG. 3 is a graph illustrating a confidence level value

) Can be determined. Current frame (

) Within the

)About,

)의 시간 윈도(TW) 내의 이벤트(

)의 인접 영역에 나타나는 이벤트들의 수 이고, M은 이전 프레임(

)에서 신뢰된 것으로 결정된 이전 이벤트들의 수 이고, α는 상수 이고, 시간 간격에 의존하고, 프레임 의존적일 수 있다.

이 이미 결정된 문턱 값 이상일 경우, 이벤트(

)는 신뢰 수준 값(

)을 갖는 신뢰된 이벤트라고 결정된다.In the above equation, N is the current frame (

) In the time window TW

), M is the number of events appearing in the previous frame (

), ≪ / RTI > is a constant, may be time interval dependent, and may be frame dependent.

Is equal to or greater than the predetermined threshold value, the event (

) Is the confidence level value

≪ / RTI >

In one embodiment, the transform estimator 103 estimates the transformation of detected events in a pair of temporally successive frames, based on the following equation:

는 인덱스이고,

는 검출된 DVS 이벤트이고,

는 DVS 이벤트(

)에 대한 신뢰 수준 값(스칼라) 이고,

는 프레임 이고,

는 프레임 인덱스이고,

는 DVS에 대한 카메라 프로젝션 모델이고,

은 DVS에 대한 카메라 프로젝션 모델의 역수 이고,

는 카메라 프로젝션 모델(

) 내의 검출된 이벤트(

)에 대응하는 벡터고,

는 세계 좌표에서 DVS 이벤트(

)의 (3D) 깊이이고,

는 프레임

및 프레임

사이의 세계 좌표들에 기초한 복수의 가능한 벡터 변환들 중 하나 이다. 변환 모델은 깊이(

) 및 카메라 프로젝션 모델(

) 내 신뢰 이벤트들(

)의 매칭에 기초하여 카메라의 3D 변환(즉,

)을 추정한다. 카메라 프로젝션 모델은 프레임의 위치들에 3D 세계 포인트들을 투사하기 위한 변환 매트릭스를 포함하지만, 카메라 프로젝션 모델의 변환 매트릭스는 카메라 캘리브레이션(calibration) 시 미리 고정될 수 있고, 카메라 동작 내내 바뀌지 않을 것이다. 그러므로, 카메라 프로젝션 변환은, 한 프레임으로부터 다음 프레임으로 움직이는 카메라에 대해 추정된 변환과는 상이하다.In the above equation,

Is an index,

Is a detected DVS event,

DVS event (

(Scalar), < / RTI >

Is a frame,

Is a frame index,

Is a camera projection model for DVS,

Is the reciprocal of the camera projection model for DVS,

Is a camera projection model

The detected event (

), ≪ / RTI >

Is the DVS event (

(3D) depth,

Frame

And frame

Lt; / RTI > is one of a plurality of possible vector transforms based on world coordinates between. The transformation model is a

) And camera projection model (

) My Trust Events (

) Of the camera based on the matching of the camera (i.e.,

). The camera projection model includes a transformation matrix for projecting 3D world points at locations of the frame, but the transformation matrix of the camera projection model may be fixed in advance during camera calibration and will not change throughout the camera motion. Therefore, the camera projection transformation is different from the estimated transformation for the camera moving from one frame to the next.

The transform estimator 103 estimates the transform for each temporally contiguous pair of frames for each event stream having a different frame aggregation time. That is, if only one frame integration time is selected, the transform estimator 103 estimates the transform for each pair of temporally successive frames in one event stream received by the transform estimator 103. If more than one frame integration times are selected, the transform estimator 103 estimates the transform for each pair of temporally consecutive frames for each event stream corresponding to the selected frame integration time.

) 및 이전 프레임(

)은 높은 신뢰도 수준을 갖는 것으로 결정된 DVS 이벤트(

)들을 포함하는 것으로 도시된다. 즉, 프레임(

) 내의 DVS 이벤트들(

,

및

)은, 프레임(

) 내의 DVS 이벤트들(

,

및

)에 대해 각각 높은 신뢰 수준을 갖는다. 변환(

)은 프레임(

) 및 프레임(

) 사이에서 결정된다. 변환(

)은 이벤트들을 세계 좌표들로 맵핑하는데 사용되고, 이 후 세계 좌표들에 기초하여 카메라 포즈(

,

및

) 추정하는데 사용된다. The conversion estimator 103 outputs the estimated conversion (s) to the camera pose estimator 104. In one embodiment, the camera pose estimator 104 uses the estimated transform (s) to estimate the camera pose being adjusted for the map on world coordinates, instead of adjusting the camera pose based on the previous frame . If more than one frame integration times are selected, the camera pose estimator 104 estimates a camera pose for each event stream having a different frame integration time. 4 illustrates an exemplary process for estimating a DVS camera pose, in accordance with the subject matter disclosed herein. 4,

) And the previous frame

) Is a DVS event (< RTI ID = 0.0 >

As shown in FIG. That is,

) ≪ / RTI >

,

And

) ≪ / RTI >

) ≪ / RTI >

,

And

), Respectively. conversion(

) Is a frame (

) And frame (

). conversion(

) Is used to map the events to world coordinates, and then the camera pose (

,

And

).

Referring again to FIG. 1, the IMU 105 is physically connected to the DVS 101 to allow the IMU 105 to detect inertial movements of the DVS 101. The output of the IMU 105 is integrated by the integrator 106 synchronized with the frame integration time (s) using the same frame integration time (s) and used by the frame shaper 102. When more than one frame integration time is selected by the frame selection information 108, the integrator 106 concurrently consolidates the output of the IMU 105 for each selected frame integration time (see, e.g., Figure 2) .

In one embodiment, the camera pose estimator 104 receives the estimated transforms from the turnaround estimator 103 and receives the output from the integrator 106, and the camera pose estimator 104 receives from the integrated IMU 105 Lt; RTI ID = 0.0 > DVS. ≪ / RTI > In one embodiment, the camera pose estimator 105 receives camera pose estimates 105 from the integrator 106 to estimate the position and / or motion of the DVS based on the received estimated transform and the output received from the integrator 106 The output can also be used.

In another embodiment, the camera pose estimator 104 may correct or update estimated pose using two estimated transforms that were generated based on two different frame integration times. That is, the camera pose estimator 104 uses the estimated transform based on the relatively short frame integration time, such as 50 ms, to correct or update the estimated pose based on the transform based on the relatively long frame integration time, such as 100 ms . For example, the camera pose estimator 104 may modify or update every second estimated pose based on a 100 ms frame integration time using every fourth estimated pose based on a 50 ms integration time (e.g., , See dotted line 201 in Fig. 2). In another embodiment, the camera pose estimator 104 may use the received transforms and the output received from the integrator 106 based on a relatively short frame integration time such as 50 ms, (E.g., see FIG. 2) the estimated pose, estimated position, and / or estimated motion of the DVS from the output received from the integrator 106 based on the long frame integration time.

FIG. 5 illustrates an electronic device 500 that includes one or more integrated circuits (chips) that includes a DVS pose estimation system, in accordance with the subject matter disclosed herein. The electronic device 500 may be used in a computing device, a personal digital assistant (PDA), a laptop computer, a mobile computer, a web tablet, a cordless telephone, a cell phone, a smart phone, a digital sound player, or a wired or wireless electronic device, It does not. The electronic device 500 includes an input / output device 520, a memory (not shown) such as, but not limited to, a controller 510, a keypad, a key boat, a display or a touch screen display, 530 and an air interface 540. [ For example, the controller 510 may include at least one microprocessor, at least one digital signal processor, at least one microcontroller, and the like. The memory 530 may be configured to store the instruction code used by the controller 510 or user data. Various system components, including electronic device 500 and electronic device 500, may include a DVS pose estimation system according to the subject matter disclosed herein. The electronic device 500 may use a wireless interface 540 configured to transmit data to or receive data from a wireless communication network using an RF signal. For example, the wireless interface 540 may include an antenna, a wireless transceiver, and the like. The electronic device 500 may be any one or more of the following systems: CDMA, GSM, NADC, E-TDMA, WCDMA, , CDMA2000, Wi-Fi, Muni Wi-Fi, Bluetooth, Enhanced Digital Cordless Communication (DECT), Wireless Universal Serial Bus (Wireless USB), Flash low-latency access IEEE 802.20, General Packet Radio System (GPRS), iBrust, WiBro, WiMAX, WIMAX-Advanced, Universal Mobile Telecommunication Service Time Division Multiplexing (UMTS-TDD) Such as but not limited to Duplex, High Speed Packet Access (HSPA), Evolution Data Optimized (EVDO), Long Term Evolution-Advanced (LTE-A) Used by the communication interface protocol of .

FIG. 6 illustrates a memory system 600 that may include a DVS pose estimation system, in accordance with the subject matter disclosed herein. The memory system 600 may include a memory device 610 and a memory controller 620 that store a large amount of data. The memory controller 620 controls the memory device 610 to read the data stored in the memory device 610 or write data to the memory device 610 in response to a read / write request of the host 630. The memory controller 620 may include an address mapping table for mapping an address provided from the host 630 (e.g., a mobile device or a computer system) to the physical address of the memory device 610. The memory device 610 may include one or more semiconductor devices including a DS pose estimation system according to the subject matter disclosed herein.

As will be appreciated by one of ordinary skill in the art, the advanced concepts disclosed herein may be modified and varied over a wide range of applications. Accordingly, the scope of the claimed subject matter is not limited by any of the specific illustrative teachings set forth above, but is defined by the following claims.

Claims (20)

A DVS for detecting DVS (Dynamic Vision Sensor) events and forming frames based on the accumulated DVS events;
Estimating a transformation of the camera of the DVS based on the estimated depth and matching the confidence level values in the camera projection model such that at least one of the plurality of events detected during the first frame corresponds to the DVS event detected during the second frame , The second frame being a transform estimator following the first frame;
An Inertial Measurement Unit (IMU) for detecting inertial movements of the DVS with respect to world coordinates between the first and second frames; And
And a camera pose estimator for combining information from a change in a pose of the DVS camera between the first frame and the second frame based on the estimated information and the detected inertial movements of the DVS A dynamic vision sensor pose estimating system. The method according to claim 1,
Wherein the camera pose estimator comprises:
Using the detected inertial movements of the DVS to correct the estimated change in the pose of the transformation model between the first frame and the second frame based on the estimated transformation, . The method according to claim 1,
Wherein the conversion estimator comprises:
Estimating the conversion of the DVS camera for the first and second frames during a plurality of frame integration times,
Wherein the camera pose estimator calculates the camera pose estimator based on an estimate of the change in the pose of the DVS camera between the first frame and the second frame corresponding to a second frame integration time, Corrects a first estimate of a change in a pose of the DVS between one frame and the second frame, and wherein the first frame integration time is longer than the second frame integration time. The method according to claim 1,
Wherein the transform estimator estimates the transform of the DVS camera for the first and second frames during a plurality of frame aggregation times,
Wherein the IMU detects inertial movements of the DVS for world coordinates between first and second frames during each of the plurality of frame integration times. The method according to claim 1,
Wherein the conversion estimator comprises:

Estimates the transform based on the transform,

Is an index,

Is a detected DVS event,

DVS event (

(Scalar), < / RTI >

Is a frame,

Is a frame index,

Is a camera projection model for DVS,

Is the reciprocal of the camera projection model for DVS,

Is a camera projection model

The detected event (

), ≪ / RTI >

Lt; RTI ID = 0.0 > DVS < / RTI &

(3D) depth,

Frame

And frame

Wherein the dynamic vision sensor pose estimator is one of a plurality of possible vector transforms based on world coordinates between the input vectors. 6. The method of claim 5,
Wherein the camera pose estimator is operative to determine a camera pose of the DVS based on the estimated transition of the camera projection model, A dynamic vision sensor pose estimation system using inertial movements. 6. The method of claim 5,
Wherein the transform estimator estimates the transform of the DVS camera for the first and second frames for a plurality of frame integration times,
Wherein the camera pose estimator is configured to estimate the camera pose based on an estimate of the change in the DVS camera pose between the first frame and the second frame corresponding to a second integration time, Corrects an estimate of the change in the DVS camera pose between the second frame, and wherein the first integration time is longer than the second integration time. 6. The method of claim 5,
Wherein the transform estimator estimates the transform of the DVS camera for the first and second frames during a plurality of frame integration times,
Wherein the IMU detects inertial movements of the DVS for the world coordinates between the first and second frames during each of the plurality of frame integration times. Detecting DVS events and forming frames from the accumulated DVS events;
Estimate the transform of the camera based on the estimated depth and match the confidence level values in the camera projection model such that at least one of the plurality of DVS events observed during the first frame corresponds to the DVS event detected during the second frame, The second frame coming after the first frame;
Detecting inertial movements of the DVS for world coordinates between the first and second frames; And
Estimating a change in a pose of the camera between the first frame and the second frame based on the estimated transform and the detected inertial movements of the DVS, How to estimate. 10. The method of claim 9,
Further comprising correcting the estimated change in the pose of the camera between the first frame and the second frame generated based on the estimated transformation using the detected inertial movements of the DVS Way. 10. The method of claim 9,
Wherein estimating the transform of the camera for the first and second frames comprises estimating the transform of the camera during a plurality of frame integration times,
The first frame corresponding to the first frame integration time and the second frame corresponding to the first frame integration time based on the estimation of the change in the pose of the camera between the first frame and the second frame corresponding to the second frame integration time Correcting an estimate of the change in a pose of the camera at the first frame integration time and the first frame integration time being longer than the second frame integration time. 10. The method of claim 9,
Wherein estimating the transform of the camera comprises estimating the transform of the camera for the first and second frames during a plurality of frame integration times,
Wherein detecting the inertial movements of the DVS comprises detecting inertial movements of the DVS for the world coordinates between the first and second frames during each of the plurality of frame integration times Way. 10. The method of claim 9,
Wherein the step of estimating the transform comprises:

And estimating the transform based on the transform,

Is an index,

Is a detected DVS event,

DVS event (

(Scalar), < / RTI >

Is a frame,

Is a frame index,

Is a camera projection model for DVS,

Is the reciprocal of the camera projection model for DVS,

Is a camera projection model

The detected event (

), ≪ / RTI >

Lt; RTI ID = 0.0 > DVS < / RTI &

(3D) depth,

Frame

And frame

And one of a plurality of possible vector transformations based on the world coordinates between. 14. The method of claim 13,
Further comprising correcting the estimated change in the pose of the camera between the first frame and the second frame based on the estimated transform using the detected inertial motions of the DVS. 14. The method of claim 13,
Wherein estimating the transform of the camera comprises estimating the transform of the camera for the first and second frames during a plurality of frame integration times,
Based on an estimation of the change in the pose of the camera between the first frame and the second frame corresponding to a second integration time, Correcting an estimate of the change in a pose of the camera, wherein the first integration time is longer than the second integration time. 14. The method of claim 13,
Wherein estimating the transform of the camera projection model comprises estimating the transform of the camera for the first and second frames during a plurality of frame integration times,
Wherein detecting the inertial movements of the DVS comprises detecting inertial movements of the DVS for the world coordinates between the first and second frames during each of the plurality of frame integration times. A DVS for detecting DVS events and forming frames based on the number of accumulated events;
Estimating a transformation of the camera of the DVS based on the estimated depth and determining at least one of a plurality of DVS events detected during the first frame to correspond to a DVS event detected during a second frame for at least two frame integration times. A confidence estimator for comparing confidence level values in the model, the second frame being for each frame integration time following the first frame; And
And a camera pose estimator for estimating a change in a pose of the camera between the first frame and the second frame for at least two of the plurality of frame integration times based on the estimated transformation, Further comprising, for a first frame integration time, a first frame integration time based on an estimation of the change in the pose of the camera between the first frame and the second frame, Correcting an estimate of the change in the pose of the camera and the first frame integration time being longer than the second frame integration time. 18. The method of claim 17,
Further comprising an inertial measurement unit (IMU) for detecting inertial movements of the DVS for world coordinates between the first and second frames during each of at least two frame integration times,
Wherein the camera pose estimator is further configured to determine a pose of the camera between the first frame and the second frame for at least two of the plurality of frame integration times based on the estimated motion and the detected inertial motions of the DVS. And estimates the change in the dynamic vision sensor pose. 19. The method of claim 18,
Wherein the camera pose estimator is further operative to determine, for each frame integration time using the detected inertial movements of the DVS, the difference between the first frame and the second frame generated based on the estimated transformation, A dynamic vision sensor pose estimation system for correcting an estimated change. 18. The method of claim 17,
Wherein the conversion estimator comprises:

Estimates the transform for each frame aggregation time based on the frame aggregation time,

Is an index,

Is a detected DVS event,

DVS event (

(Scalar), < / RTI >

Is a frame,

Is a frame index,

Is a camera projection model for DVS,

Is the reciprocal of the camera projection model for DVS,

Is a camera projection model

The detected event (

), ≪ / RTI >

Lt; RTI ID = 0.0 > DVS < / RTI &

(3D) depth,

Frame

And frame

Wherein the dynamic vision sensor pose estimator is one of a plurality of possible vector transforms based on world coordinates between the input vectors.

Images