EP3314474A1

EP3314474A1 - Interactive device with three-dimensional display

Info

Publication number: EP3314474A1
Application number: EP16742340.9A
Authority: EP
Inventors: Fabrice Bernard; Ludovic PERRINE; Jean-Marc Oury; Bruno Heintz
Original assignee: F4
Current assignee: F4
Priority date: 2015-06-24
Filing date: 2016-06-22
Publication date: 2018-05-02
Also published as: WO2016207551A1; US20180322143A1; FR3038090A1; FR3038090B1

Abstract

An interactive device comprises a user interface (8) for defining display requests comprising at least location data and display data, a network interface arranged to issue a request for web page data corresponding to a display request, a memory (4) for receiving corresponding web page data, and a web interpreter (10) for displaying a web page from web page data. The web interpreter (10) comprises at least a WebGL interpreter and a 3D motor arranged to calculate, for given web page data comprising map data and object data, face data of at least one object associated with the object data, to group together said face data in face block data and transmit the face block data to the WebGL interpreter for the calculation of three-dimensional display data of said at least one object while maintaining a correspondence between at least some of the objects associated with the object data and the face data that corresponds thereto in the face block data. The web interpreter (10) is arranged to allow web page data to be displayed based on the map data and three-dimensional display data of the objects.

Description

Interactive device with three-dimensional display

The invention relates to web devices, and in particular devices for interaction. More particularly, the invention relates to the field of mapping on the Internet.

Internet mapping has long been limited to the reproduction of paper maps as images. This was imprecise and heavily bandwidth-consuming, and posed relevance issues, as the images did not provide a simple means of scaling or customization.

Important players in the new technologies have changed this situation by offering maps on the Internet that are lighter, scalable and adaptive, and allow the user to quickly navigate a given map. For note, like the invention, these solutions are all of "pure Web" type, that is to say that a Web client (a browser or a smartphone application) interacts with a server to obtain the data, but does not store anything locally, except cache. All data comes from the server, in real time except for cache data. This is crucial because, given the bandwidth currently available, the constraints on what is possible to obtain in terms of visual effects are very strong if one wishes to maintain a fluid display.

These same actors added layers of information on the maps to add some information, such as the location of certain businesses or monuments, and so on. However, it is literally about layers in the graphic sense of the term: the nature of the interaction is limited to the display of a text, the clickable area allowing this interaction is limited to a small area chosen arbitrarily , and can not for example concern an entire building, or a park in a city. Efforts have also been made to bring some realism by adding three-dimensional views for some buildings. But again, these are purely graphical elements, devoid of interaction. The invention improves the situation. For this purpose, the invention proposes an interactive device, comprising a user interface for defining display requests comprising at least location data and display data, a network interface arranged to send a request for data of web pages. in correspondence with a display request, a memory for receiving corresponding web page data, and a web interpreter for displaying a web page from web page data, which web interpreter includes at least one WebGL interpreter and an engine 3D arrangement for computing, for data web page data including mapping data and object data, face data of at least one object associated with the object data, for grouping said face data into data of face blocks and transmitting the face block data to the WebGL interpreter for computing three-dimensional display data of the at least one object by keeping a correspondence between at least some of the objects associated with the object data and the corresponding face data in the face block data, the web interpreter being arranged to allow display of web page data to from the mapping data and the three-dimensional display data of the at least one object.

This device is advantageous because it allows to have a representation of a map in three dimensions, with as many objects in three dimensions as necessary, each object being individually clickable and allowing an unknown interaction until then. In addition, since the face data is calculated on the client side, it becomes possible to transmit a large quantity of objects in three dimensions, which makes it possible to model all the buildings of a city, but also the trees and other road details. , as well as networks that achieve a degree of detail and interactivity never achieved while maintaining a fluid display.

According to various variants, the device may have one or more of the following characteristics:

the memory stores only data of type cache, the display data comprises zoom data, and the mapping data define a geographical environment substantially centered on a location designated by the location data, and the extent of which depends on the zoom data,

the object data includes building data including building location data, roof shape data, and height data,

the object data comprises terrain data including location data, and altitude data,

the 3D engine calculates the face data by triangulation from the object data and zoom data,

the web interpreter is arranged to determine an object data identifier of an object designated by a user via the user interface from the correspondence between at least some of the objects associated with the object data and the data of the user. corresponding faces in the face block data.

the code of the 3D engine is received as display data at the device initialization and / or during updates,

the 3D engine arranges the data of blocks of faces in an object of the Float32Array type, and

the device comprises material resources dedicated to the display.

The invention also relates to an interactive display method comprising the following operations:

at. define a display request comprising at least location data and display data,

b. issue a request for web page data in correspondence with a display request

vs. receive corresponding web page data including mapping data and object data, and transmit them to a web interpreter including a 3D engine and a WebGL interpreter

d. calculating with the 3D engine face data of at least one object associated with the object data, and grouping these face data in face block data, e. transmitting the faceblock data to the WebGL interpreter, maintaining a correspondence between at least some of the objects associated with the object data and the corresponding face data in the faceblock data, for computing data three-dimensional display of said at least one object, f. calling the web interpreter with the mapping data and the three-dimensional display data of the at least one object to display web page data.

Other features and advantages of the invention will appear better on reading the following description, taken from examples given for illustrative and non-limiting purposes, taken from the drawings in which:

FIG. 1 represents a schematic diagram of a device according to the invention, in its operating environment,

FIG. 2 represents a schematic diagram of an operating loop of the device of FIG. 1, and

FIG. 3 represents a schematic diagram of an updating operation of FIG. 2.

The drawings and the description below contain, for the most part, elements of a certain character. They can therefore not only serve to better understand the present invention, but also contribute to its definition, if any.

This description is likely to involve elements likely to be protected by copyright and / or copyright. The rights holder has no objection to the identical reproduction by anyone of this patent document or its description, as it appears in the official records. For the rest, he reserves his rights in full.

FIG. 1 represents a generic diagram of an interactive device 2 according to the invention in its environment. The example described here finds a particularly interesting application in the field of on-line mapping. However, the invention finds its application in any situation involving a three-dimensional display involving interactivity with the user in a context "any server", or "any flow (streaming English)", that is to say that all the data comes from a server, the device does not store anything locally (including the JavaScript code of the 3D engine in the example described here) apart from the cache. This application is particularly suitable for web use, but may be used as an application for smartphones for example, in a format other than a web page. In this case, the application may store the code data of the 3D engine for example, but will still not aim to store other data than cache, that is to say, the stored data is not persistent.

The device 2 comprises a memory 4, a display 6, a user interface 8, a network interface (not shown) and a web interpreter 10.

The memory 4 can be any means of storing information: flash memory, RAM memory, a hard disk, a connection to a remote storage or in the cloud, etc. In the context of the invention, the memory 4 is used to store the data for processing. As mentioned above, the data of the memory 4 are not intended to be persistent, except to serve as cache whose validity period is programmed and limited. Indeed, the invention is a real-time application where all the data are obtained from the server, without prior local storage.

The display 6 can be any conventional display of the screen type, video projector, projection glasses, etc. The user interface 8 allows the user of the device 2 to choose the display parameters of the card and the location he wants to view.

The display parameters may include, but are not limited to, latitude and longitude, zoom level, viewing angle, orientation. These parameters are classic in this type of application. The invention also proposes other display parameters, called "customization" parameters. Indeed, in known applications, it is possible to activate or deactivate the display of certain elements. However, this only involves displaying or not displaying the corresponding information layers. It is not a question of allowing to personalize the display of a restricted set of objects, for example. Moreover, it is not possible to speak of "objects" in the applications of the prior art. They are simple cartographies, possibly equipped with displays in three dimensions, but not individualizable. Conversely, the invention makes it possible to make each element of the map independent, as a computer object.

The web interpreter 10 can be realized by any browser implementing HTML, JavaScript and WebGL. As will be seen in the following, the web interpreter 10 includes a WebGL interpreter and a 3D engine (not shown). In the example described here, the user accesses the map through a web page. This web page contains both HTML code, JavaScript code and WebGL code. The JavaScript code defines the 3D engine, which receives object data to be represented on the map portion required by the user, and transforms that object data into face data. The face data is transmitted to the WebGL interpreter which interacts with the hardware resources of the device 2 to calculate the data to be displayed as such.

As mentioned above, all the data is obtained from a server 12 which comprises a memory 14 and a motor 16. The memory 14 is similar to the memory 4, except that it stores all the information useful for the operation of the device 2 (unlike the memory 4 which stores almost exclusively cache).

The engine 16 is an element that receives the requests for data from Web pages of the device 2. As seen above, from the data received by the user interface 8, the device 2 defines a request for displaying the mapping from location data and display data. These data together define a geographical area. Based on this geographic area, the device 2 issues a web page data request to the server 12, and the engine 16 selects in the memory 14 the mapping data as well as the data of the objects that correspond to this query.

In the example described here, the mapping data of the memory 14 are stored in the form of tiles ("tiles" in English) in several levels of detail. Thus, each tile breaks down into four sub-tiles when the zoom level increases, and vice versa, and so on. In the example described here, the mapping data contains for each tile the list of objects associated with it. Alternatively, the object data could contain data indicating the tiles to which they are attached.

In the example described here, the identification of the tiles is implicit, ie the device 2 can determine which tile data to request from the server 12 via the web page data request. Thus, the latter is a pure resource request, and the server 12 does not develop any "intelligence". Alternatively, the web page data request could be more abstract and contain the location data and the display data, and the server 12 would load via the engine 16 to determine the relevant tile data.

The data of the objects comprise, in the example described here, data enabling the three-dimensional display of these objects to be defined in an abstract manner. Thus, for a building, the object data includes data defining the footprint, the shape of its roof, and building height data. A building can have several heights. Thus, when the 3D engine calculates the face data of a building, it proceeds by "raising" an outline corresponding to the footprint of the building on the ground to the height of the corresponding object data, and calculating the triangles defining each faces of the object thus defined. Another way of looking at this is to consider that a building is "extruded" through its footprint on its height. For a soil or a relief, it is proceeded of similar way: starting from the tiles of the data of cartography, a mesh of the ground by triangulation is realized. This mesh can be used to represent reliefs by means of object data of the relief data type, which includes location data enabling to indicate the tile to which they correspond, and altitude data to indicate the height of the relief. When relief data is received, the vertices of the tile triangles corresponding to the corresponding location data are raised to the height designated by the corresponding altitude data. Objects can also be immaterial, such as a bus or metro line, or a water or electricity distribution network.

Advantageously, but optionally, depending on the web page data request, the engine 16 may select the object data based on a value indicating their size. Thus, if the zoom is very far, the engine 16 only returns details whose size is relevant given the resolution of the reproduced card. And as soon as a user zooms in, the engine 16 will send object data whose size has made them relevant given the desired resolution. This has the advantage of controlling the download and processing load on the device side, but also improving the visual experience: as a user zooms in, the world becomes clearer.

FIG. 2 represents a schematic diagram of an operating loop of the device of FIG. 1.

In an operation 200, the device 2 initializes the display of the map. For this, the location data and the display parameters are obtained from the user interface 8, and the device 2 implements the network interface to issue a display request to the server 12 with these data. In response, the server 12 transmits data including the code of the 3D engine, as well as the mapping data and the corresponding object data. In subsequent exchanges, the 3D engine code is no longer transmitted.

In an operation 220, the device 2 receives the mapping data and the object data, as well as the code of the 3D engine. On the basis of these data, the web interpreter 10 displays, for example, the mapping data first, which gives the user a quick view of the "basemap". At the same time, the 3D engine processes the object data to calculate the corresponding face data. The face data of each object is grouped and entered in a face data block by means of a 32-bit array ("Float32Array") type of memory block used by the WebGL interface. hereinafter Buffer32. The Buffer32 is a classic WebGL tool used to send a 3D display data calculation request to a hardware resource from face data defined in the Buffer32. The WebGL interpreter transmits the Buffer32 containing all the data from the face to the hardware resource of the device 2 for computing the data for the three-dimensional display of the object data. Each Buffer32 can contain data related to 65536 triangle vertices. When a Buffer32 is "full", the 3D engine instantiates another one, while preserving the correspondences between the data of each Buffer32 and the objects to which they correspond. Alternatively, Buffer32 could be realized with another type of array of values. The face data may include data defining vertices, faces, or texture mapping information on the triangles.

Simultaneously, the web interpreter 10 maintains a correspondence between the grouped front data for each object of the Buffer32 and the identifiers of the corresponding objects. Thus, if a user wants to select an object, he clicks with a mouse on a location on the screen (or other means of naming a location on the screen as the event "onMouseOver" JavaScript). In response, the WebGL interpreter issues a ray picking request, which returns the face in the Buffer32 that has been designated by the click, and, based on this match, the Web interpreter 10 is able to know which object has been clicked or designated, and to offer the user a menu of options for making specific requests to this object or to a class of objects corresponding to the server 12, or display this object in a particular way. Alternatively, other methods than ray picking could be used to determine the face that is under the mouse (eg the single-point rendering of the scene with colorization of the faces based on their identifier and identification of the face from the color of this point).

This characteristic is fundamental for two reasons: - The concatenation of a considerable set of front data in some Buffer32 makes it possible to display a large quantity of objects without losing fluidity. This was impossible before the invention; and

Maintaining the correspondence between the objects and the corresponding face data in the face data block makes the objects truly interactive.

Before, the so-called "objects" were point points, or three-dimensional non-clickable images. In the case of points, it was difficult to click, and the clickable area was unrelated to the object concerned. Moreover, because of the associated "overlapping data layer" approach, the idea was not to make the objects interactive, but to add a layer of simplistic information.

On the contrary, the invention makes it possible to produce a new type of card in which the interaction is possible and intuitive: if one wants to have information on a metro line or a building, or on the vegetation of a street, just click anywhere on the corresponding object.

In the existing maps, most of the details were omitted for the sole reason that they represented a mass of information that could not be processed in a block without unduly slowing down the download and / or display. Thanks to the transition to the "object" context, the details are themselves objects. It is therefore possible to choose the priority of displaying the objects, the most precise details, or "less useful" that can be displayed last, or not be displayed at all.

In addition, transforming the map from a collection of juxtaposed basemap tiles to a world of individual objects generates a significant amount of new unimaginable applications previously. For example, it becomes possible to realize real estate or geolocation applications that provide fast and relevant information. In the latter case, one can imagine that a click on a hospital will know the specialties that are practiced, or that a click on a particular building will indicate what special care is and / or the chambers of the services concerned, to click on a particular bench to indicate that it has been vandalized etc. And the other surrounding details can be prioritized in a lower way to preserve the user experience by providing the information that interests him in priority. It thus becomes possible to introduce a huge amount of "layers of information" in a simple way in cards, where previously impossible or inconvenient due to the use of "layers" superimposed and not directly related to objects individualized.

Finally, in an operation 240, the device 2 updates the display 6 according to the user inputs via the user interface 8. Thus, if the user changes the zoom level, moves the card or interacts with the one objects, the web interpreter 10 calls the server 12 to obtain new data and calculates the modified data accordingly.

FIG. 3 represents an exemplary implementation of the operation 240. In an operation 300, the web interpreter 10 determines the location data associated with the center of the card designated by the user interface 8. Then, the interpreter Web 10 determines the display settings such as viewing angle and zoom to determine a new area to be represented by the display.

In an operation 320, the web interpreter 10 determines the list of tiles corresponding to the new view determined in the operation 300 and asks the server only those that it does not already have. In response, the server 12 returns the list of corresponding tiles. Alternatively, the server 12 can keep a list of tiles and objects already transmitted and / or considered current, and directly transmit new relevant data.

Finally, in an operation 340, the web interpreter 10 carries out the same operations as for the operation 220 for the newly obtained data, and updates, if necessary, the data already known and displayed.

For example, if a zoom level has been changed, finer information about the relief is received. Updating the altitude of a building already displayed can be performed by simply updating the elevation in the definition of building face data in the face data block. Thus, for objects already displayed, the update is as light as possible and allows the construction of a 3D map with data arriving only sequentially and in disorder.

All the optimizations presented above are made possible by the fact that the invention implements an "object" context described above, whereas this was not possible previously.

Claims

claims

An interactive device, comprising a user interface (8) for defining display requests including at least location data and display data, a network interface arranged to issue a web page data query corresponding to a display request, a memory (4) for receiving corresponding web page data, and a web interpreter (10) for displaying a web page from web page data, which web interpreter (10) comprises at least one WebGL interpreter and a 3D engine arranged to compute, for data web page data including mapping data and object data, face data of at least one object associated with the object data, to group these data of faces in face block data and transmitting the face block data to the WebGL interpreter for calculating three-dimensional display data of the at least one object in storage a correspondence between at least some of the objects associated with the object data and the corresponding face data in the face block data, the web interpreter (10) being arranged to enable display of web page data; from the mapping data and the three-dimensional display data of the at least one object.

2. Device according to claim 1, wherein the memory stores only cache type data.

The apparatus of claim 1 or 2, wherein the display data comprises zoom data, and the mapping data defines a geographic environment substantially centered on a location designated by the location data, and whose extent depends on zoom data.

Device according to one of the preceding claims, wherein the object data comprises building data comprising data of building location, roof shape data, and height data.

Apparatus according to one of the preceding claims, wherein the object data comprises terrain data including location data, and altitude data.

The apparatus of claim 3 and one of claims 4 and 5, wherein the 3D engine calculates the face data by triangulation from the object data and zoom data.

Apparatus according to one of the preceding claims, wherein the web interpreter (10) is arranged to determine an object data identifier of a user-designated object via the user interface (8) from the correspondence between at least some of the objects associated with the object data and the corresponding face data in the face block data.

8. Device according to one of the preceding claims, wherein the code of the 3D engine is received as display data at the initialization of the device (2) and / or during updates.

9. Device according to one of the preceding claims, wherein the 3D engine arranges the data of blocks of faces in an object of type Float32Array.

An interactive display method comprising the following operations:

b. issue a request for web page data in correspondence with a display request

vs. receive corresponding web page data including mapping data and object data, and transmit them to a web interpreter including a 3D engine and a WebGL interpreter calculating with the 3D engine face data of at least one object associated with the object data, and grouping these face data in face block data,

transmitting the faceblock data to the WebGL interpreter, maintaining a correspondence between at least some of the objects associated with the object data and the corresponding face data in the faceblock data, for computing data displaying the at least one object in three dimensions, calling the web interpreter with the mapping data and the three-dimensional display data of the at least one object to display web page data.