JP2016500169A - Annotation method and apparatus - Google Patents

Abstract

To solve or alleviate the problems of existing augmented reality systems. An annotation method includes capturing (100) data representing a light field with a plenoptic image capture device (4); and matching (101) the captured data with corresponding reference data. Retrieving (102) an annotation associated with an element of the reference data; and rendering (103) a view generated from the captured data and including at least one annotation. [Selection] Figure 3

Description

  The present invention relates to an annotation method for adding an annotation to data corresponding to a scene.

  Due to the rapid progress in the development of handheld portable devices such as smartphones, palmtop computers, portable media players, personal digital assistants (PDA) devices, it has been proposed to include new features and applications involving image processing. Yes. In such applications, i.e. image annotation or captioning, the user points the portable device towards a scene, e.g. a landscape, a building, a poster or a painting in a museum, and the display is superimposed information relating to that scene. Together with the image. Such information may include, for example, names of mountains and settlements, names of people, historical information of buildings, commercial information such as advertisements, and menus of restaurants, for example. An example of such a system is described in EP 1246080 and EP 2207113.

  The annotation information can be supplied to the portable device by a server that is in the wireless communication network. Here, the corresponding functional configuration of the communication network with server and portable device is called an annotation system.

  WO 05/114476 describes a mobile image based information retrieval system including a mobile phone and a remote recognition server. In this system, an image taken by a mobile phone camera is transmitted to a remote server, where a recognition process is performed. This leads to the need for high bandwidth to transmit images as well as the delay to calculate annotations in the server and transmit them back to the mobile phone.

  Many annotation systems and methods include comparing a set of reference images stored in a database with images acquired by an annotation device. Since the actual viewing angle and illumination conditions are different compared to the images stored in the database, the comparison algorithm must remove the effects of these parameters.

  A further higher performance image annotation technique uses a 3D reference model. In many cases, this involves a registration process, that is, a process of spatially transforming the captured (or target) image to align with the reference 3D model. For example, in the case of a building, the 3D model of the object is stored in the reference database along with the details to be annotated. The 2D image acquired by the portable device is registered with this model, and if a match can be found, the object is recognized and the corresponding annotation is superimposed on the 2D image.

  The image annotation model based on the 3D model has an advantage that the dependence on the viewing angle is less than that of the 2D model. A single 3D model can be used as a reference for matching multiple different 2D images captured at different angles from different locations. However, building a collection of 3D models is a difficult and cumbersome process. This usually requires a 3D or stereo camera. Moreover, the process of registering 2D captured images and 3D models is time consuming.

  Accordingly, it is an object of the present invention to solve or at least mitigate the above mentioned problems of existing augmented reality systems.

According to the present invention, these objectives are:
-Capturing data representing a light field with a plenoptic capture device;
-Executing program code for matching the captured data with the corresponding reference data;
-Executing program code for retrieving annotations associated with an element of the reference data;
-Executing program code for rendering a view generated from the captured data and including at least one annotation;
Achieved through a method comprising:

The present invention also provides
A device for capturing and annotating data corresponding to a scene,
A plenoptic camera for capturing data representing a light field;
-With a processor;
-With a display;
-Program code for causing the processor to retrieve annotations associated with an element of data captured by the camera when the program code is executed and generated from the captured data; Program code for rendering a view including at least one annotation on the display;
Achieved through an apparatus comprising:

The present invention is also an apparatus for determining an annotation comprising:
-With a processor;
-With the store;
-Program code, when the program code is executed, causing the processor to receive data representing a light field, match the data with one reference data in the store, and bind to the reference data Program code for determining an annotation and causing the remote device to transmit the annotation;
There is also provided an apparatus comprising:

  Plenoptic cameras are known per se and are commercially available at low prices. Unlike conventional cameras that only capture a 2D projection of a scene on a sensor, a plenoptic camera provides data representing the light field to this pixel / sub-image as well as the intensity of light on each pixel. Data representing a matrix that also indicates the direction of light reaching or at least the intensity of light reaching each single sub-image from various directions.

  Thus, the plenoptic sensor generates more data containing information about the light reaching each sub-image than the conventional 2D image data generated by a conventional 2D image sensor.

  The data generated by the plenoptic sensor includes information about a scene that is not directly available from a conventional 3D sensor or a stereo camera. Thus, because more and different information is available, the matching process of reference data and captured data is more reliable than conventional methods of matching 2D images with 2D or 3D models. It can be intuitively understood that having more information about the captured scene is beneficial in improving recognition performance and improving registration quality.

  The data and model matching provided by the plenoptic camera is also more robust than the 3D model and 2D or 3D captured image data matching.

  Matching data representing a light field and captured by a plenoptic sensor may include projecting the light field data onto a 2D image and matching the 2D image with a 2D or 3D reference model. This process results in an increased likelihood of matching because different projections (eg, corresponding to different focal points that can be selected during rendering of the plenoptic image) are possible. However, additional resources are required to calculate this or these projections, and information about the captured scene is lost during the interaction, resulting in reduced matching accuracy and speed.

  Thus, in one embodiment, the data captured by the plenoptic sensor is matched with reference data representing a reference light field. Advantageously, this matching is performed without projecting any captured light field data onto the 2D image and / or without projecting any reference light field data onto the 2D image. Thus, matching is not only based on lightness at each point in the scene, but also in the plenoptic domain, without loss of information due to conversion onto 2D or 3D images, and capture. This is also based on the direction on the ray in the generated data and the reference data.

  The method may include registering the captured light field data onto the reference light field data. This registration process is aimed at finding the geometric relationship between the captured light field data and the various annotations to be displayed. For example, for map data, the ultimate goal of the registration process is to find where the light field captured by the plenoptic sensor is in the reference map and later overlay map annotations in the correct location. There is to do. By performing this registration process entirely within the plenoptic space, all the information present in the data representing the light field is used to produce a more accurate annotation of the scene.

  The method may include matching the captured light field data with reference light field data generated by the plenoptic sensor at different locations. The method may include matching the captured light field data with reference light field data generated by the plenoptic sensor at different distances. The method includes matching the captured light field data with reference light field data generated by different types of plenoptic sensors or having a different number of pixels in each sub-image. Good.

  By registering the captured light field data with the reference light field data, in the captured light field data to properly and accurately register with the more complete information present in the reference light field data. All of the information that exists in the system will be utilized, so that scene annotation can be performed appropriately and accurately.

  The invention can be better understood by using the description of the embodiments provided as an example and shown in the figures.

FIG. 2 schematically illustrates a plenoptic camera that captures data representing a light field of a scene with an object at a first distance. FIG. 2 schematically illustrates a plenoptic camera that captures data representing a light field of a scene with an object at a second distance. FIG. 6 schematically illustrates a plenoptic camera that captures data representing a light field of a scene with an object at a third distance. FIG. 1 schematically illustrates a system that includes various device elements that together implement the present invention. FIG. 3 is a block diagram of a method for capturing data representing a light field and rendering an annotated 2D image. It is a block diagram of the local registration method using a global model. FIG. 6 is a block diagram of a global registration method in plenoptic space.

  Unlike conventional cameras that only capture a 2D projection of a scene on a sensor, plenoptic sensors aim to capture the complete light field that exists in a given scene. A complete light field may contain seven parameters for each pixel: three for position, two for direction, one for wavelength, and one for time.

  A plenoptic sensor is a data representing a so-called plenoptic light field, i.e. a matrix, from which at least four of these parameters are hit, i.e. each pixel of the plenoptic sensor. Data representing a matrix from which the 2D position and 2D direction of the ray to be calculated can be calculated. We may call this data “light field data”.

  As of today, at least two companies have proposed plenoptic sensors capable of recording such plenoptic light fields. That is, Lytro and Raytrix. Although the two cameras of these companies are slightly different in design, the main idea is to decompose different directions of light that are expected to hit a single photosite (or pixel) in a standard camera sensor. ) For this purpose, as shown in FIG. 1, an array of microlenses 20 is placed behind the main lens 1 in place of a conventional camera sensor. The image sensor 21 is moved backward.

  Thus, the microlens 20 redirects the light according to its angle of incidence, and the redirected light reaches different pixels 210 of the sensor 21. The amount of light measured by each of the N × M pixels 210 that make up the sub-image depends on the direction of the light beam that hits the microlens 20 in front of the sub-image.

  1-3 show a simple one-dimensional sensor containing n = 9 sub-images, each sub-image having one row of N × M pixels (or photosites) 210, in this example N is equal to 3 and M is equal to 1. Many plenoptic sensors have more sub-images and more pixels for each sub-image, for example 9 × 9 pixels, and N × M = 81 different light on the microlens 20 It enables discrimination between orientations. Assuming all objects in the scene are in focus, each sub-image thus contains a patch of lightness values representing the amount of light coming from different directions on that sub-image.

  In this structure, the array of microlenses 20 is positioned on the image plane formed by the main lens 1 of the camera, and the sensor 21 is positioned at a distance f from the microlens, where f is the focal length of the microlens. It is. While this design allows for high angular resolution, the spatial resolution is relatively low (the number of effective pixels per rendered image is equal to the number of microlenses). This problem is addressed by other plenoptic cameras where the microlens is focused on the main lens image plane, thus creating a gap between the microlens and the image plane. Such a design comes at the price of reduced angular resolution.

  As can be seen from FIGS. 1 to 3, the plenoptic light field corresponding to the scene having a single point 3 in this example depends on the distance from the point 3 to the main lens 1. In FIG. 1, all light beams from this object reach the same microlens 20, so that all pixels in the sub-image corresponding to this microlens record the first positive light intensity while others This results in a plenoptic light field in which all other pixels corresponding to the other lens record different, null light intensities. In FIG. 2, where the object 3 is closer to the lens 1, some of the light beam from point 3 is tied to another sub-image, ie two microlenses adjacent to the previously hit microlens. Reach the sub-image pixel. In FIG. 3 where the object 3 is at a greater distance from the lens 1, the different pixels in which some of the light beams from point 3 are associated with two microlenses adjacent to the previously hit microlens. To reach. Therefore, the digital data 22 transmitted by the sensor 21 depends on the distance to the object 3.

  Thus, the plenoptic sensor 21 has, for each sub-image corresponding to the microlens 20, a set of N × M values representing the amount of light coming from various directions on the lens above this sub-image. The light field data 22 including is delivered (delivered). For a given focused object point, each pixel of the sub-image corresponds to a ray intensity measure that hits the sensor at a constant angle of incidence.

  There are many advantages to knowing the direction of the rays. Performing refocusing (changing the focused object in the scene) or changing the camera viewpoint, among other tasks, by carefully rearranging the rays Can do.

  FIG. 4 schematically shows a block diagram of an annotation system embodying the present invention. The system includes a user device 4, such as a handheld device, a smartphone, a tablet, a camera, glasses, goggles and the like. Device 4 includes a plenoptic camera 41 such as the camera shown in FIGS. 1-3 for capturing data representing a light field on scene 3, and a processor such as microprocessor 400 with suitable program code. A communication module 401 such as a cellular interface and / or WIFI for connecting the remote server 5 such as a cloud server and the device 4 on a network such as the Internet 6. The server 5 includes a storage 50 with a database, such as a SQL database, a set of XML documents, a set of light field data images, etc., for storing a reference light field data collection and / or one or more global models. A processor 51 including a microprocessor with computer code for causing the microprocessor to perform the operations required in the annotation method. Annotations and corresponding positions can also be stored in the storage 50 along with the reference light field data.

  The program code executed by the user device 4 may include, for example, application software or app that can be downloaded and installed in the user device 4 by the user. The program code may also include a part of the operating code of the user device 4. The program code may also include code embedded in a web page or executed in a browser, including, for example, Java, JavaScript, HTML5 code, and the like. The program code may be stored as a computer program product in a tangible apparatus readable medium such as flash memory, a hard disk, or any type of permanent or semi-permanent memory.

  Program code is stored in the user device 4 to cause the microprocessor to send captured data sets corresponding to the light field, or at least some of the features of these data sets, to the remote server 5. And executed by the microprocessor 400. The program code is arranged to send this light field data in a “plenoptic format”, i.e. without losing information about the direction of the rays. The program code also causes the microprocessor 400 to receive annotations associated with the annotated data in the light field format, or the annotated image, or previously transmitted light field data from the server 5 and is annotated and captured. You can also render a view corresponding to the data.

  In one embodiment, the program code in the user device 4 is also used to identify a local feature present in the captured data and calculate a description of this local feature, eg, the program code is sent to the remote server 5 It also includes a module for computing a binary vector that can be sent.

Program code executed by the microprocessor in server 5 may include an executable program or other code that causes server 5 to perform at least some of the following tasks:
-Receiving data representing light fields from the device;
-Retrieve the model and / or multiple reference data in a plenoptic format;
-Matching data received from a user device with a part of the model or one of a plurality of reference data, respectively;
-Determining the annotation associated with the model or each associated with one of the reference data;
-Sending annotations, annotated images or annotated data corresponding to the received data to the device.

  In various embodiments, instead of sending a capture data set to a remote server for matching with reference data in the server, this match is stored locally in a set of locally stored reference data or user devices. And can be done locally.

  Various possible embodiments of methods that can be implemented using the systems, devices, and arrangements of FIGS. 1-4 will now be described.

A. Multiple independent reference data sets based on plenoptic local features In one embodiment, a collection of known and previously captured reference data sets representing light fields in the storage 50 of the server 5, e.g. A collection of reference data previously captured with a tick camera or converted from a 3D model is available. In this case, the matching data must be recognized from the set of reference data before proper registration is possible. Registration with the matching reference data is only performed after that.

A possible sequence of steps used in this embodiment is shown in FIG. This includes the following:
Step 100: The light field to be annotated is captured by the plenoptic camera 41 in the user device 4 or retrieved from any possible source of light field data. Although a 2D projection of the captured plenoptic light field may be displayed on the 2D display 40 of the user device 4, the data is preferably as light field data, i.e. for the direction of the incident ray on each sub-image. It is memorized without losing information.
Step 101: Process if the plenoptic camera used to capture the reference data is not of the same type as the plenoptic camera used to capture the light field data to be annotated May include a step 101 of converting or resampling either data into the format of the other data. For example, different plenoptic cameras may generate light field data having a different number of pixels in each sub-image, or sample the light field in different ways. This conversion may be performed within the user device 4 and / or the remote server 5.
Step 102: Detection of local features in the captured data. As described below, for example, by following a DPF (depth plenoptic feature) algorithm or by using disparity information contained within a light field, Alternatively, the detection can be performed by expressing the light field with an epipolar volume. Other detection methods and other types of local features may be used. The type of local features used and the detection method may depend on the scene, location, user choice, etc.
Step 103: A description of these local features found in the capture data. Depending on the type of local feature detected during the previous step, for example, as described below, a binary vector or other descriptor well adapted by the description of the disparity or local feature point in the epipolar volume It is contemplated that different types of descriptors can be used, including (descriptor) and the like. The detection and description of local features is advantageously performed by a suitable software module in the user device 4 that only needs to send these short descriptions to the server 5. It is possible to send complete light field data to the server 5 as well, and the server 5 will then detect and describe local features, but this will result in less efficient use of available bandwidth.
Step 104: Recognition of captured data based on the described local features. This can be done in various ways. In one embodiment, local features are quantized (step 1040) and then the quantized features are used to generate reference data having the same (or nearly the same) quantized feature set during step 1041. You can search. The reference data may be retrieved from the user device and / or from the remote storage 50 in the remote server 5. The pre-filtering of the reference data is based on various filtering criteria, such as the location of the user device 4 predetermined from a satellite or terrestrial location system, the signal received from the scene, the user's selection, etc. May be done. The reference data may include data representing a 2D image, a 3D model, or preferably a light field. This step may be performed by suitable program code in the server 5, but local recognition in the user device 4 is also possible if the number of reference data is not too large.
The quantization step 1040 allows the system to be more easily scaled when the number of known criteria increases.
Step 106: Matching the detected local features in the captured data with the local features in the reference data identified during the previous step. Local features in the reference data are detected and described in a previous step when the collection 50 is constructed. This step may be performed by suitable program code in the server 5 but may also be performed in the user device 4.
Step 107: Find a geometric transformation that maps the local features detected from the captured data to the matching reference data. This step is called “registration”. Transformation includes warping, scaling, translation or homography of the captured data using rotation. If multiple reference images are available, this step may include determining the reference data with the best registration quality. Registration may take place in the user device 4, in the remote server 5, or partially in the user device and in the remote server.
In one embodiment, the result of the registration process also indicates the total position of the user device 4 that captures the scene in relation to the information to be displayed as an “augmented layer”. Camera position and orientation may be identified by six parameters: three for position and three for that orientation. This step may be performed by suitable program code in the server 5 but may also be performed in the user device 4.
Step 108: Retrieve at least one annotation associated with the reference data in the collection 50, as well as the location or feature of the image to which this annotation is to be associated.
Step 109: Render a view, such as a 2D or 3D image, on the display 40 of the user device 4 based on the captured data with at least one of the annotations retrieved during step 108.

B. Global Reference Data Set Based on Plenoptic Local Features Method A described above uses a collection of reference data representing different light fields and reference data that matches the reference data with the highest accuracy or confidence based on local features. Depends on the process to decide.

  Here, a global method that uses a global model of a scene without depending on the availability of a collection of reference light field data will be described. This method still uses local features for matching and registration of the captured data with this model. This type of method is useful, for example, for outdoor location, but could be used for other augmented reality applications where a model of the whole scene, such as buildings, museums, malls, etc. is available.

  The global model may consist of a cloud of local features computed over a light field dataset captured with one or more plenoptic cameras. For example, a model of a city or reference scene may be constructed by aggregating a large set of light field data captured by various cameras. Local features are detected and described in these various data pieces. These described features are then assigned to specific physical locations within the global coordinate system. Finally, the model is thus composed of a cloud of local features, each representing a specific physical location in the global coordinate system. In the case of a city, the coordinate system may be that used in, for example, GPS (WGS84), and it is believed that all features may represent a particular point / local area within that coordinate system.

  Alternatively, the model is not composed of plenoptic local features extracted from plenoptic samples. For example, a 3D model of a city can be obtained while the query is a plenoptic sample. In that case, the possibility of rendering synthetic light field data from the 3D model is considered. Another possibility is that a geometric transformation that maps the input plenoptic image on the 3D model using a mutual information measure between the two data modalities is related to the mutual information measure. May apply a minimization process that is optimized in

To match new capture data captured by the plenoptic camera 41 in the user device 4 against this cloud of local plenoptic features, the following approach illustrated with FIG. May be used:
Step 100: Capture or retrieve data representing a light field to be annotated.
Step 101: Resample data if necessary.
Steps 102-103: Detect and describe local features in the captured data representing the light field.
Step 110: Match the detected local features with local features of the global model 1101, such as a model stored in the database 50, for example. This matching can be speeded up by accelerating the search by binning features together. Based on the preceding information 1102 (GPS information, user input, etc.), a pruning step 1100 may be performed to speed up matching. At this time, matching is performed only on a subset of local features corresponding to the preceding information. Locality sensitive hashing may be used, in which case a set of hash functions is computed for feature descriptors to create clusters based on different hash values. The set of hash functions is selected such that two descriptors that are close to each other in the descriptor space produce the same hash value.
Step 111: Calculate a geometric transformation that projects the local features detected in the capture data with matching local features in the global model. This is a registration step. The output of this step is the pose estimation of the camera 41. Thus, it is possible to know where the camera that captures the capture data is in relation to the model coordinate system.
Step 108: The annotation is then retrieved. Annotations are usually position-dependent and are themselves registered in the model coordinate system.
Step 109: An annotated image is rendered.

  Again, the use of plenoptic information improves the robustness of the matching and registration process, especially under different lighting conditions, image deformation, etc.

C. Global registration based on light field data using a global model Here, a further registration method based on global registration using a global model is described. Like previous method B, this method can be used when a known global model of a given scene is available. For example, in the case of a city, we can load a 3D model of that city that has a-priori information that we are in a given city and is therefore already available. The registration process sends out the position of the camera that captured the light field data in relation to the model coordinate system.

As an example, an exemplary method based on global registration may include the following steps, illustrated using FIG.
Step 152: During step 152, the global model of the scene or environment in which the user is currently located is stored in the memory of the user device 4 such as the user's smartphone or tablet or navigation system including the plenoptic sensor 2, for example. Loaded. The model loaded from the storage 50 may depend on, for example, the user's location determined using GPS, user selection, automatic scene analysis, other a priori known information, and the like.
Step 100: The light field to be annotated is captured by the camera 41 of the user device 4. Although the 2D projection of the captured plenoptic light field may be displayed on the 2D display 40 of the user device 4, the data is preferably as light field data, i.e. for the direction of the incident ray on each pixel. The information is stored without loss.
Step 101: The process may include an additional step of converting or resampling the captured data to facilitate or expedite the matching and recognition process, for example if the models have different formats. . For example, different plenoptic cameras may generate data having a different number of pixels in each sub-image, or sample the light field in different ways. This conversion can take place in the user device 4 or in the remote server 5.
Step 150: The initial position may be estimated based on, for example, GPS, user input information, or other similar prior information.
Step 151: The captured data is registered in relation to the model. In the output, there are six complete degrees of freedom for the position of the camera in relation to the model. If the model is loaded into the user device 4, registration can be performed by a processor in this device.
Step 108: A set of annotations associated with a location around the calculated location of the device 4 or what should be visible from this location is retrieved from the model.
Step 109: A view, such as a 2D or 3D image, is rendered on the display 40 of the user device 4 based on the captured data with at least one of the annotations retrieved during the previous step.

The registration step 151 of the global registration method described above is preferably given an estimate of the camera position and uses an objective function to model the plenoptic light field sample (in the above case). Calculates the error to be projected onto the city model. Use this objective function (also known as a cost function) to apply an iterative optimization process in such a way that the camera position estimate is refined and improved to minimize projection errors be able to. This optimization process can be broken down into the following steps:
1. Obtain / calculate an initial estimate of the position of the user device. This can be done, for example, in the case of a smartphone including a plenoptic camera, by using a smartphone GPS, an accelerometer and a compass to calculate the position and orientation of the device. This initial set value is set as the current set value.
2. Calculate the projection of the input plenoptic sample into the model. A projection error is calculated using the objective function (step 1510).
3. Given the error and objective function, calculate the next camera position estimate (step 1511) and set it as the current estimate.
4). If the error is larger than the specific threshold value, the process returns to step 2; otherwise, the process proceeds to step 5.
5. The current estimate is the optimized position of the user device and corresponds to the actual position of the device in relation to the model.

  Since we are using data representing a light field, we adjust the objective function used in step 1510, thus using all the information that the objective function exists in the data set, and using standard 2D The registration can be made more robust than when the image is used.

  An objective function tailored specifically for plenoptic input samples can be derived to make the registration more robust to all kinds of transformations and lighting conditions. If a plenoptic model is not available, one possible approach is to generate a plenoptic synthetic sample from the 3D model. This sample can be generated by simulating a virtual plenoptic camera and performing a ray-tracing process on different 3D model points. It is believed that each point of the 3D model can be represented using 3D coordinates and physical properties such as reflectance or transparency. The scene light source may be similarly described for the purpose of obtaining a realistic 3D scene. In the absence of a scene light source, lighting can be viewed as ambient and thus equally affecting each object in the scene. The raytracing method then involves reconstructing the ray path in space to simulate the actual ray traveling in the scene. In the presence of light sources, light rays are traced starting from these light sources and propagated onto the objects in the scene. When ambient lighting is considered, the rays are generated directly from the physical points of the 3D model. Reflection, refraction, scattering or dispersion are some of the optical effects that can be simulated by ray tracing to ensure good realism of scene rendering.

  A virtual plenoptic camera can be placed in the virtual scene to simulate a light field hitting the plenoptic camera sensor. All rays that enter the main lens of the camera can then be virtually projected onto a virtual sensor to create plenoptic reference data that is responsive to the 3D model.

  After retrieving this plenoptic reference data, the camera viewpoint can be determined such that the correlation between the light intensities in the reference data and the captured data is maximal. Other objective functions could be used to determine the most likely viewpoint of the camera in the model.

Local Feature Detection and Description Both methods A and B are specific for data that is truly informative, ie data whose entropy is high compared to other areas of space. It aims to reduce the registration space for local features only. Moreover, the mutual information or relative entropy between the two local features aims to be low so that they can be easily differentiated from each other if the two local features represent two different areas. ing. The final desired properties of these features are the same, given two views of the same scene, whatever the transformation between these two views (geometric transformation, exposure changes, etc.) The feature can be detected.

  According to one aspect, the type of local feature used for registration and recognition is selected depending on the type of scene. For example, a natural panoramic view does not use the same features as a city at the street level. In the former case, a horizontal line can be used as the feature, and in the latter case, the appropriate feature is the point where multiple different types of depths intersect.

  WO 2012/084362, the contents of which are incorporated herein by reference, describes an augmented reality method in which the algorithm depends on the scene. However, this document does not suggest adapting the type of local features used for registration to the type of scene. A method similar to that described in WO 2012/084362 can be used to determine the type of local feature to be used depending on the scene type determined from eg device location, image analysis, user selection, received signal, etc. It can be used in the apparatus and method described herein to determine.

First example of local feature: Depth Plenoptic Features (DPF)
In one embodiment, the local features used for capture data registration include a planar intersection.

  Images of manufactured objects, such as photographs or mechanical parts in an urban environment, for example, are often numerous artificial structures that are usually very regular in terms of geometry and usually have a low degree of texture including. Within these areas, the points where multiple planes intersect are typically considered to represent corners in 3D. Thus, in such an artificial scene, feature points can be defined as areas where a minimum number of planes intersect.

  This type of feature detection can be done efficiently and accurately by taking advantage of all the information present in the captured data in light field format.

  In the data sent by the plenoptic sensor in the plenoptic camera 41 (FIG. 4), different pixels of the sub-image arrive at different angles of incidence on the microlens 20, i.e. at different distances. Corresponds to the light beam coming from the object. Accordingly, an area where an object is focused in different focused planes can be easily detected as sub-images in which a plurality of adjacent pixels have the same or substantially the same value.

  Thus, pixel sets taken from different sub-images are retrieved in order to produce focused images at different depths without the depth field calculations or other computationally intensive tasks. Depth plenoptic features can be defined as areas where physical points at different depths exist simultaneously, and these features can be used to register captured data with reference data.

  Consider stacks with different projections of the light field at different focal lengths. If one image of this stack is taken, the object that is in focus will be less focused on the previous image. The same is true for the next image. Therefore, it is possible to calculate a 3D gradient on this stack. Areas with a high gradient magnitude correspond to objects / pixels with a high focus level. This low gradient magnitude area corresponds to objects present at different depths that can be detected and used as high entropy features for registering captured data. Thus, combining this in-focus detection technology with the ability of plenoptic cameras to provide different focused information for the same physical area results in repeatable features with a high level of information provision. .

  Thus, this local feature detection method includes, for example, an area in the data that corresponds to a plane that is substantially parallel to the line of sight, the stack has a low 3D gradient, and corresponds to the same object at different depths. Detection may be included. The method may also include detecting areas in the data that correspond to a plane that is substantially orthogonal to the line of sight and for which adjacent pixels have similar values. The local feature detection method may include detecting an intersection between a plane substantially perpendicular to the line of sight and a plane substantially parallel to the line of sight.

  More generally, local feature detection is an area in a plenoptic light field where pixels corresponding to a particular depth have a predetermined relationship with pixels of the same sub-image at different depths. Detection can be included. For example, high entropy or high frequency in the depth direction (parallel to the line of sight) may also be considered a useful feature for registration.

Second example of local feature: local feature based on disparities In one embodiment, the local feature used to identify the captured plenoptic light field includes disparity information contained within the light field. Is used.

  The disparity of a physical point is the displacement between two projections of the physical point with respect to one plane. In a typical visual system, the parallax is calculated as corresponding to the difference in position for the same physical point projected from two different views on the same image plane.

  The displacement of a point projection from two different views is related to the depth of that point in relation to the plane on which the point is projected. A point at a certain distance from the camera plane has a higher parallax (displacement) value than a point further away from the plane. That is, the closer the object is to the plane, the greater the parallax value. As a result, the depth is inversely related to the parallax value.

  Since the capture of the plenoptic light field includes information on the position and direction of the rays coming from the physical point, it is possible to extract different rays corresponding to different views coming from the same physical point. At this time, parallax and depth information can be calculated using the sub-image pixels associated with these rays.

  At this time, depth information can be combined with local features to improve the robustness of identification and matching.

  In one embodiment, the depth information can be used as an average to cluster points into objects that exist at a particular depth. This embodiment is particularly advantageous for manufactured objects or urban scenes that often contain a significant number of artificial structures that are geometrically regular. In fact, planes are frequent in such artificial environments. At this time, the cluster represents a plane at a specific depth orthogonal to the line of sight of the camera.

  With clusters, matching is more robust. In fact, instead of just constraining a single local feature, multiple groups of features can be matched together. Matching these clusters is more restrictive than using only local features and therefore produces better results.

  Keypoint clustering also has the advantage of discarding meaninglessly separated features that do not belong to any cluster. Thus, the number of features needed to match a scene is reduced, and as a result is more adapted to systems that need to match large annotations or many captured images.

Third example of local feature: Epipolar Volume form
In one embodiment, epipolar volumes and more specifically lines called epipolar lines within these volumes are used to detect meaningful and stable local feature points. Epipolar lines can be combined with other feature detectors, such as Harris affine feature region detectors. Representing a plenoptic light field sample as an epipolar volume form is extremely interesting because it simplifies and speeds up many analyzes of plenoptic volumes. An epipolar volume is created by stacking images together when the camera movement between the two images is simply horizontal translation. The following conclusions can be obtained by analyzing these volumes. That is, a line that exists on these volumes may represent a single physical point. Therefore, the slope of this line also defines the depth of this point.

  Thus, in one embodiment, local features in the light field data are determined and projected in epipolar volume space. Within this space, two short lines are filtered to remove non-stable features, while points are clustered into lines and only a single local feature point per line is retained ( retain). At the output, a stable local feature set is obtained as detected under different viewpoints.

Local Feature Description: Binary Plenoptic Feature Descriptor Local feature description (eg, in step 103 of FIG. 5) could be done in binary form. The Hamming distance can be used to compare two features together to see if they are similar, so not only the descriptor size of each feature is significantly reduced, but also the comparison Speed up. In fact, the Hamming distance can be calculated efficiently using a dedicated vector instruction that calculates the distance for multiple bytes at once.

  The DPF features described above can be described using descriptors that leverage information from a gradient operator. A faster method is to perform a pixel value comparison to describe the detected features. This can be viewed as a simplified version of the gradient operator. These comparisons of pixel values are made around previously detected feature points so that the desired repeatability and informability of such descriptors can be maintained. The result of a single comparison corresponds to 1 bit long information. Performing multiple comparisons results in a bit-string descriptor, where each bit corresponds to a specific single comparison.

  This principle of binarized descriptors can be used in plenoptic space by taking advantage of all the information of plenoptic light field data to obtain plenoptic binary descriptors. . If the image is generated by a standard pinhole camera, the pixel value comparison will correspond to a comparison of the visual information of the image. In the case of a plenoptic camera, the comparison is done in different dimensions to maximize descriptor entropy.

  As seen earlier, the plenoptic image is composed of a plurality of sub-images. A single sub-image contains multiple representations of the same physical point under different viewpoints. Therefore, this information redundancy is effectively used in plenoptic binary descriptors. When this plenoptic binary descriptor is combined with the aforementioned DPF detector, the focal stack utilized by the detector can also be used as a source of comparison points. Thus, the plenoptic binary descriptor contains both information about different views of the area and information about different depths of this feature area.

  At this time, the plenoptic binary descriptor is calculated by selecting one set of comparison point pairs. One part of these pairs corresponds to the location of pixel values taken from the sub-image around the feature point area detected by the DPF detector. The other part corresponds to points around the feature point area but at different depths in the focal stack of the DPF detector. This set of pairs is selected only once and the same is used for all descriptor calculations.

  There are various strategies for selecting this set of comparison points. The first is a strategy of selecting randomly within a desired space, which can be either a focus stack or a sub-image. This works well with high reliability, while learning the best set and maximizing the inter-distance between different features while minimizing the intra-distance between the same features It is also possible to use machine learning. For medium size feature areas, a search based on the best comparison point greedy algorithm is performed to maximize the variance while minimizing the correlation of descriptors.

In order to compute a binary descriptor representing a given feature area, the following procedure can be applied:
1. For each comparison point pair, it is determined whether the rendered grayscale pixel value at the first comparison point is less than at the other point.
2. If the comparison is true, a binary “1” is added to the descriptor (which is initially empty), otherwise a binary “0” is added.
3. The procedure is repeated for each comparison point to create a binary string descriptor.

  Using these techniques, the thus-determined binary descriptor of the captured data can be compared with the binary descriptor of the reference plenoptic light field. This comparison may be based on a Hamming distance to determine their relative distance in this plenoptic feature space.

From registration to augmented scene After registration with any one of the methods described above, the plenoptic camera 41 in the user device 4 is relative to the registered reference scene. The position and orientation are known. Reference data corresponding to the captured data is similarly known and associated with an annotation set for various elements or features of the data in the reference database. Annotations may consist of text, images, video, audio, manipulation or enhancement of existing features, 3D objects, and the like. These depend on the scene and view context to be annotated.

  The final expanded (annotated) image is then rendered. For example, it is possible to generate a 2D image (still image or video) showing a captured landscape with a mountain name or other annotation superimposed on the image. Alternatively, in an urban environment, directions to the nearest store and amenity can be displayed on the image.

  In one embodiment, rendering of the view (in-focus object, camera perspective) occurs prior to the incorporation of the annotation. Thus, since the poses for a given rendered view as well as the position of the annotations in the model are known, they can be projected into the view selected to render.

Augmented Reality (AR) Plenoptic Rendering and Application The capture of a scene in plenoptic space hears a new potential door for augmented reality rendering. In fact, the position and direction of rays that hit the sensor in the plenoptic camera can, among other features, retrieve depth information, refocus after image capture, or change the user's viewpoint. This information can be leveraged to further improve scene rendering and provide a new experience to the user. The following paragraphs describe some possible advanced rendering capabilities.

  In fact, one particular advantage of augmented reality is the fact that the user can interact with the elements of the image rendered by the process, for example by clicking on features of interest to obtain some relevant additional information. is connected with. This interaction is very advantageous because the user will interact directly with the object, whether real or virtual, instead of being passive.

  For example, it is often desirable to tell the user that any particular object in the rendered image is interactive and associated with the annotation, so that the user can click it, for example . One way to solve this problem is to display a notification, such as a text box with an arrow pointing to the object. However, if multiple interactive objects are part of the captured scene, there must be many notifications telling the user what the interactive elements are.

  The plenoptic space allows new interactive elements that provide a better experience for the user. As described above, the data captured by the plenoptic sensor has the ability to be rendered as a 2D image with different focal lengths even after the data is captured. Also, the refocusing process can be calculated independently for each local part of the data and does not necessarily consider the data as a whole. In other words, this means that certain objects in one image can be focused even if they do not belong to the same depth in the scene. ing.

  Thus, annotated objects or annotated image features can be rendered in focus while the other elements of the scene are blurred. In this way, the user can immediately notice what is annotated or interactive in the image and what is not.

  As an example, an interactive augmented reality manual or video tutorial can be envisaged, where the different knobs or parts of the printer contain useful instructions that are displayed in augmented reality based on user selections. A 2D annotated image can be rendered from the plenoptic light field, showing the printer, and the rest of the image is blurred while all its interactive knobs or parts are in focus To. Thus, the user is presented with an interactive part of the printer, who can click to access the annotation. Similarly, the user may change the depth of focus if he wants a focused view of other elements.

  Changing the plenoptic camera viewpoint offers the possibility to render each point of a scene as a partial 3D element. Rays originating from the scene are captured from one location rather than all locations around the object, so 3D reconstruction is only partial. However, this partial 3D reconstruction allows to render objects in the scene with swinging / jittering motion. These objects appear as 3D objects that pop out of the image as seen from a particular direction. Again, this effect can be calculated locally for the selected object in the scene. Thus, the interactive elements of one scene are displayed as moving objects and can thus draw the user's attention while the other objects remain stationary. At this time, the user can click on these swinging elements to trigger the display of annotation contents.

  The various operations of the methods described above are performed by any suitable means capable of performing these operations, such as various hardware and / or software component (s), circuits, and / or module (s). It's okay. In general, any of the operations described in this application may be performed by corresponding functional means capable of performing them. Various means, logic blocks, and modules include circuits, application specific integrated circuits (ASICs), or general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate array signals (FPGAs). Or various other including, but not limited to, programmable logic devices (PLDs), individual gate or transistor logic, individual hardware components, or any combination thereof designed to perform the functions described herein Hardware and / or software component (s) and / or module (s) may be included. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may be implemented as a combination of computing devices, such as a DSP and microprocessor, multiple microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. A server may be implemented as a single machine, as a set of machines, as a virtual server, or as a cloud server.

  As used herein, the expression “light field data” is calculated from a 3D model as if it were generated by a plenoptic camera or captured by a plenoptic camera, and the light of one scene It means any data describing a field image, ie an image in which not only the lightness and color of light but also the direction of this light is stored. A 2D or 3D projection rendered from such a plenoptic light field image is not considered a plenoptic light field image because the direction of this light has been lost.

  As used herein, the expression “plenoptic space” is a multidimensional space in which the light field can be described, ie, the amount of light reaching the sensor or light in all directions in the space. It may mean a multidimensional space in which the function to be described can be described. The plenoptic space can be described by at least two parameters for the position of each sub-image and at least one additional parameter for the direction of light reaching this sub-image. In many cases, the plenoptic space consists of two parameters for the position of each sub-image, two parameters for the direction of light on this sub-image, at least one parameter for the wavelength, and possibly time. Is described by one parameter (for video).

  The term “annotation” as used herein includes, for example, text, still images, video images, logos, image layers, sound and / or other elements that can be superimposed or added to the image. Encompasses a variety of possible elements.

  As used herein, the term “pixel” may refer to a single monochrome photosite or a plurality of adjacent photosites for detecting light in different colors. For example, three neighbors for detecting red, green, and blue light can form a single pixel.

  As used herein, the term “determining” encompasses a variety of actions. For example, the term “determining” includes calculating, computing, processing, deriving, investigating, looking up (eg, table, Including referencing within a database or another data structure), ascertaining, estimating, and the like. The term “determining” also includes receiving (eg, receiving information), accessing (eg, accessing data in a memory) and the like. In addition, the term “determine” may include resolving, selecting, selecting, setting, and the like.

  Capturing an image of a scene involves the use of a digital camera to measure the light intensity reaching the camera image sensor. Capturing light field data may involve the use of plenoptic cameras, or it may involve generating light field data from 3D models or other descriptions of scenes and light sources. .

  The expression “render view”, eg “render a 2D view from light field data” encompasses actions to calculate or generate an image, eg to calculate a 2D image from information contained in the light field data. . The expression “project a view”, eg “project a 2D view based on light field data”, may be used to assert the fact that multiple different views may be rendered.

  The method or algorithm steps described in connection with this disclosure may be implemented in the form of direct hardware, in the form of software modules executed by a processor, or in a combination of the two. A software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, and the like. A software module may include a single instruction or many instructions and may be distributed across different code segments, across different programs, and across multiple storage media. A software module can be an executable program, a single part used in a complete program, a routine or library, a plurality of interconnected programs, an “apps” that is executed by many smartphones, tablets or computers. ”, A widget, a flash application, a part of HTML code, or the like. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The database may be implemented as a SQL database, an XML document set, a semantic database, or any structured data collection that includes a set of information available on an IP network, or any other suitable structure.

  Thus, certain aspects may include a computer program product for performing the operations presented herein. For example, such computer program products may include a computer-readable medium having instructions stored (and / or encoded) for performing the operations described herein. It can be executed by one or more processors. For certain aspects, the computer program product may include packaging material.

  It should be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

DESCRIPTION OF SYMBOLS 1 Main lens 4 User device 5 Server 6 Network 20 Micro lens 21 Image sensor 40 Display 41 Plenoptic camera 50 Storage 51 Processor 210 Pixel 400 Processor 401 Communication module

EP 1246080 specification European Patent Application No. 2207113 International Publication No. 05/114476 International Publication No. 2012/084362

Claims (24)

-Capturing (100) data representing a light field with a plenoptic camera (41) in the device (4);
-Executing (101) program code for matching the captured data with the reference data;
-Executing (102) program code for retrieving annotations associated with an element of the reference data;
-Executing (103) program code for rendering a view generated from the captured data and including at least one annotation;
Annotation method including   The method of claim 1, wherein the reference data defines a reference light field.   The method of claim 2, comprising generating the reference data from a 3D model of the scene.   The method of claim 2, wherein the matching step includes matching the captured data with a data piece of a plurality of reference data pieces representing different right views.   The method of any one of the preceding claims, comprising detecting (102) local features in the captured data.   The local feature detection step (102) includes detecting areas where pixels at a first depth have a predetermined relationship with pixels at different depths. Method.   The method of claim 5, wherein the local feature detection step (102) includes detecting parallax in the captured data.   The method of claim 5, wherein the local feature detection step (102) includes calculating an epipolar volume or line.   The method according to any one of claims 5 to 8, comprising the step (1011) of describing the local features.   The method of claim 9, wherein the local feature is described by a descriptor in binary form.   The method of claim 10, comprising calculating a Hamming distance between the descriptors.   The method of any one of claims 5 to 11, comprising matching (106) the local features in the captured data with local features in the reference data.   13. A method according to any one of claims 5 to 12, comprising registering (107) the reference data and the plenoptic data using the local features.   14. A method according to any one of claims 5 to 13, comprising detecting a type of scene and determining a type of local feature to be detected in the captured data in accordance with the scene type.   15. Selecting one or a limited number of reference data pieces prior to the matching depending on the position of the device (4), selections made by a user or received signals. The method as described in any one of.   The method according to claim 1, wherein the reference data includes a global model of a scene.   17. The method of claim 16, comprising minimizing (1510) a cost function representing a projection error of the captured data onto the reference data.   18. The step (109) of rendering a view includes rendering a 2D view derived from captured data and overlaying an annotation on the 2D view. the method of.   The step (109) of rendering the 2D view from the captured data is focused on the annotated object or feature of the annotated image while the rest of the scene is blurred. 18. The method of claim 17, comprising displaying the annotated object or annotated image features in a conforming fashion. A device (4) for capturing and annotating data corresponding to a scene,
A plenoptic camera (41) for capturing (100) data representing a light field;
A processor (400);
-A display (40);
-Program code for causing the processor to retrieve annotations associated with an element of the data captured by the camera when the program code is executed and generated from the captured data Program code for rendering a view on the display (40) comprising at least one annotation;
Including the device.   The apparatus of claim 20, wherein the program code is further arranged to cause the processor (400) to detect local features present in captured data when the program code is executed.   21. The apparatus of claim 20, wherein the program code is further arranged to describe each detected local feature with a binary vector.   A computer program product comprising a tangible device readable medium for causing the device to perform the method of any one of claims 1-19. An apparatus (5) for determining an annotation,
A processor (51);
-Store (50);
A program code that, when the program code is executed, causes the processor to receive data representing a light field, matches the data with one reference data in the store, and is associated with the reference data; Program code for determining the annotation and sending the annotation to the remote device (4);
Including the device.