WO2013122536A1 - Ajout de données d'augmentation à une vue en perspective 3d - Google Patents

Ajout de données d'augmentation à une vue en perspective 3d Download PDF

Info

Publication number
WO2013122536A1
WO2013122536A1 PCT/SE2013/050121 SE2013050121W WO2013122536A1 WO 2013122536 A1 WO2013122536 A1 WO 2013122536A1 SE 2013050121 W SE2013050121 W SE 2013050121W WO 2013122536 A1 WO2013122536 A1 WO 2013122536A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
perspective
camera device
browser
transform
Prior art date
Application number
PCT/SE2013/050121
Other languages
English (en)
Inventor
Oskar LINDE
Anders Bond
Petter Ivmark
Original Assignee
13Th Lab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 13Th Lab filed Critical 13Th Lab
Publication of WO2013122536A1 publication Critical patent/WO2013122536A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation

Definitions

  • the present invention relates generally to camera devices and more particularly to a method for adding augmentation data to a 3D perspective view of an object in front of a camera device of a mobile camera arrangement.
  • Most mobile devices today are equipped with a camera allowing real time viewing of events in front of the camera on a display of the mobile device. Viewing of events on the display from a 3D angle is desirable, and adding of augmentation data in a correct 3D perspective to the picture displayed on the display device is also desirable.
  • augmentation data is to be displayed in a 3D perspective on a display of a mobile communication arrangement in relation to an object in front of a camera device in the mobile communication arrangement
  • An object of the present invention is to provide a method of adding augmentation data in a 3D perspective to a view in relation to an object in front of a camera device of a mobile communication arrangement, without a developer having to consider movement of the camera device relative the object when programming the augmentation data.
  • a method is provided allowing e.g. standard HTML and JavaScript code and content related to a certain event or a physical object in front of a camera device to be correctly rendered on top of or relative to that object as the camera device is being moved around by a user, without the developer of the HTML content having to understand or adapt the rendering of the content himself as its perspective on the screen is shifted to reflect the movement of the camera device relative to the object.
  • the method comprises the steps of: determining a perspective value of the camera device of the mobile camera arrangement; receiving frames of image data of the object from the camera device of the mobile camera arrangement; determining a camera position of the camera device for a frame of the received frames; determining a transform for the frame of the received frames, based on the camera position; and in a browser displaying augmentation data in a 3D perspective view of the frame of the received frames, wherein the augmentation data is resized by the transform in relation to the object it is possible to in e.g. HTML add augmentation data in a correct 3D perspective to a view of an object in front of a camera device of a mobile camera arrangement without knowledge of movement of the camera device relative the object, thus allowing a developer to render traditional HTML and JavaScript content on top of reality.
  • Fig. 1 is a schematic illustration of a mobile camera arrangement according to an embodiment of the present invention.
  • the method is exemplified for iOS and Web Kit, and comprises the following steps.
  • a perspective value of the camera device of the mobile camera arrangement is determined, preferably based on at least the focal length of the camera in video pixels, the video width in pixels and the width of the displayed video in screen pixels.
  • the perspective value is calculated by first dividing the focal length of the camera measured in video pixels by the video width in video pixels, thereafter multiplying the result by the displayed video stream width, measured along the same axis, in screen pixels.
  • the perspective value is thereafter preferably set to a perspective node of an HTML page e.g. by calling a JavaScript method.
  • the perspective node is preferably an HTML div named e.g. viper.world node and the perspective value is given as a parameter to the CSS attribute "-webkit-perspective".
  • the cameras optical centre is re-calculated to screen pixels in the web view and given as parameter to the CSS attribute "-webkit-perspective_origin" of the perspective node.
  • a scene node is preferably provided in an HTML div named e.g. viper.scene_node.
  • the scene node is preferably given the CSS property "-webkit-transform-style: preserve-3d", wherein any content inside it is transformed with a scene transform.
  • the perspective node is preferably directly within the ⁇ body> tag of the HTML document, and the scene node is preferably located within the perspective node.
  • the perspective value is here described as a calculation in pixels. This can of course be calculated in other forms such as in mm.
  • Frames of image data of the object are received from the camera device of the mobile camera arrangement.
  • a camera position of the camera device is determined for a frame of the received frames, preferably for each frame of the received frames.
  • the camera position relative to a 3D scene in the real world, or a subset of the real world, such as a surface or object, is preferably estimated from SLAM (Simultaneous Localization And Mapping), image or surface recognition, object recognition or other localization techniques, which are suitable for mobile device processing.
  • the camera position calculation is preferably performed in the native layer of the application using e.g. C/C++ or Objective-C.
  • a transform for the current frame of the received frames is thereafter determined based on the estimated camera position.
  • the transform is preferably also performed in the native layer, and comprises identification of how the object is positioned and moves relative to a 3D coordinate system, i.e. being a coordinate system transform.
  • the transform is preferably given as a parameter to CSS attribute "-webkit- transform: matrix3d(. . .)" of the scene node.
  • the transform is preferably determined for each frame of the received frames. Setting of CSS attributes of nodes in the HTML page from the native part of the application are e.g. done through an evaluation of a JavaScript method such as a native-to-JavaScript call (stringByEvaluatingJavaScriptFromString) .
  • Augmentation data in this case augmentation HTML and JavaScript data, is in a browser displayed to a 3D perspective view of the frame of the received frames, wherein the augmentation data is resized by the transform in relation to the object.
  • Images used to initiate a 3D perspective view may specify actual (real world) dimensions. In such a case the scale of a scene is calculated to correctly match reality.
  • an HTML page scale is used, which preferably is taken into account when calculating a scene node 3D transform matrix.
  • a default value of the scale may be 1000, meaning that an image sized 0.5 x
  • a method of detecting and tracking an event in front of the camera device is preferably utilized in combination with the method of adding augmentation data as described above.
  • a possibility is provided for a JavaScript program to be able to receive notifications of specific events that happen in front of the camera of a mobile device, such as a specific picture or object being held up in front of the camera and thus in real-time adapt the content of the web page to what the camera device is pointed at, or based on certain events that happen in front of it.
  • This is a great improvement in the possibility of customization of the browser content and the ability to make that content dynamic over the traditional methods of using GPS or Wi-Fi location to adapt the web page's content to the camera device's surroundings.
  • a dedicated application program e.g.
  • an iOS or Android application for each use case that may be conceived (e.g. one application for recognizing articles in news papers, another application for recognizing museum art), or creating static content in bespoke/proprietary content formats that need to be published to specific content hosts and cannot be updated in real time by a developer, giving the developer no way to run/ execute code client side as the application is used and thus dynamically react to events as can be done with a client side application and is also the case with a normal web page on the Internet.
  • developers can create normal web pages, running client side JavaScript code, that they host themselves and a user can simply load a new page (and thus the developer's JavaScript program) from different places on the Internet for different use cases just like on the rest of the Internet.
  • the method offers a solution for JavaScript programs, running in a built-in browser of a mobile camera arrangement, to be notified of certain events that happen in front of a camera device thereof and for the JavaScript program embedded in a web page loaded in the built-in browser to react to these events in real time.
  • An example is a web page for a museum that adapts and presents information about a certain work of art automatically as the device's camera is pointed towards it.
  • the certain events can e.g. be generic, such as "faces”, which would then notify the JavaScript program that a face was found in front of the camera, preferably along with other features such as the orientation, scale and position of that face in the image or in 3D space.
  • Certain events could e.g. also be known images or objects found in front of the camera that were provided by the JavaScript program at runtime, when the page was loaded in the browser, such as a set of known images, e.g. a specific set of paintings, that the JavaScript program of the web page would like to get notified about as they appear in front of the camera device.
  • the JavaScript run-time environment often has certain APIs (Application Program Interfaces) offered by the device manufacturer to be able to communicate with hardware and sensors in the mobile device.
  • APIs Application Program Interfaces
  • An example of this is that a JavaScript program can often ask the mobile device about the mobile device's GPS position via a JavaScript API exposed by the manufacturer.
  • JavaScript APIs are however often limited to what the manufacturer has chosen to offer access to and not easily extensible to other use cases than those pre-defined by the manufacturer.
  • the mobile camera arrangement 1 is typically a smart phone having a built-in camera device 2.
  • the mobile camera arrangement is typically configured to allow a custom application program 3 to be installed therein, allowing communication through an API having a set of native instructions for communication to other parts of the mobile camera device.
  • the mobile camera arrangement 1 is exemplified by an iPhone, running iOS.
  • the custom application program 3 is installed in the iPhone creating functions for capturing events in front of the camera of the iPhone and exposing these functions via a set of APIs.
  • the custom application program 3 is when run configured to embed the iOS operating system's built-in browser 4 as a so called web view and allow a standard webpage and JavaScript program therein to use these additionally exposed APIs in order to request and be notified of events in front of the camera of the iPhone.
  • the method for allowing a JavaScript program in a web page to be notified comprises the steps of: from a JavaScript program running in the system's built-in browser requesting notifications of a certain event happening in front of the camera device of the mobile camera arrangement; analysing data from the camera device of the mobile camera arrangement; detecting the certain event from the analysed data; and notifying the JavaScript program in the built-in browser of the detected certain event.
  • the certain event is preferably requested from the webpage running in the built-in browser embedded by the custom application program.
  • the step of analyzing data and the step of detecting the certain event are preferably executed by the custom application running in the mobile camera arrangement.
  • the data from the camera device can e.g. be one or more camera images, or a stream of camera images.
  • the JavaScript program is preferably notified of each such detected position.
  • the camera images are preferably collected from the camera device. After the camera images have been collected, the images are analyzed in a known way by image, object, scene or other pattern or 2D or 3D recognition searching for a match to the certain event.
  • the data from the camera device can also e.g. be image descriptors or a video stream.
  • the step of requesting a certain event preferably comprises sending the request for the certain event through the custom API exposed by the custom application program.
  • Installation of the custom application program thus creates a set of APIs that the webpage and JavaScript program can use to register for certain events for which it would like to get notified about.
  • registerForCameraEvent("face", myFaceEventCallback) While on many systems, such as in this case the iOS, the built-in browser in a mobile camera arrangement can only communicate to and from the native application layer of the mobile camera arrangement using the pre-defined APIs provided by the manufacturer himself (for example a camera API or a location/GPS API), the present invention utilizes a way of circumventing this restriction to allow the custom application program to communicate with, receive from, and send data to, the built- in browser provided by the manufacturer.
  • the application program uses a previously defined custom URL scheme with a specific format that the native application is aware of (in this example using window.location or a hidden iframe to initiate the request) to start monitoring a certain event.
  • the custom application program can intercept this request in a callback before it is loaded and processed by the built-in browser.
  • the request from the web page is then terminated, causing the browser not to load a new page. If it's not a match, the URL request goes through uninterrupted.
  • the invention uses a way to execute a JavaScript command from the native application.
  • the native application program can use a method included in the built-in browser API (such as UIWebView:stringByEvaluatingJavaScriptFromString in iOS).
  • the application program will then send a JavaScript command as a string, which will be executed by the built-in browser and a result returned to the JavaScript program in the browser.
  • Another alternative way to achieve the same functionality is to let the native part of the application (in this case the Objective-C part of the application) set up a web server on e.g. the local host address 127.0.0.1 of the device and let the JavaScript part of the application (in this case the embedded web view) communicate with the native part of the application using HTTP requests or web sockets to this web server on the same device.
  • the native part of the application in this case the Objective-C part of the application
  • the JavaScript part of the application in this case the embedded web view
  • the method steps of the present invention are preferably performed by computer program code of a computer program for the mobile camera arrangement.
  • the computer program can preferably be stored on a computer readable means forming a computer program product.
  • the method for allowing JavaScript in a mobile web browser is adapted for the mobile camera arrangement instead being an Android phone.
  • the native application has created a set of Java functions that unlike iOS can be exposed directly to the JavaScript program in the browser, thus letting the JavaScript program directly do pre-defined native function calls from the browser in order to request notification of certain events.
  • the custom application program on Android can also utilize the Java-based APIs that unlike iOS can be exposed directly to the JavaScript program in the browser, allowing the native application to do a so called call-back to the JavaScript program when an event occurs in front of the camera.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)

Abstract

La présente invention porte sur un procédé d'ajout de données d'augmentation à une vue en perspective 3D d'un objet se trouvant devant un dispositif de caméra (2) d'un agencement de caméra mobile (1), le procédé comprenant les étapes consistant à : déterminer une valeur de perspective du dispositif de caméra de l'agencement de caméra mobile; recevoir des trames de données d'image de l'objet en provenance du dispositif de caméra de l'agencement de caméra mobile; déterminer une position de caméra du dispositif de caméra pour une trame des trames reçues; déterminer une transformation pour la trame des tramess reçues, en fonction de la position de caméra; et afficher, dans un navigateur (4), des données d'augmentation dans une vue en perspective 3D de la trame des trames reçues, les données d'augmentation étant redimensionnées par la transformation par rapport à l'objet.
PCT/SE2013/050121 2012-02-14 2013-02-13 Ajout de données d'augmentation à une vue en perspective 3d WO2013122536A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE1250117-7 2012-02-14
SE1250117 2012-02-14

Publications (1)

Publication Number Publication Date
WO2013122536A1 true WO2013122536A1 (fr) 2013-08-22

Family

ID=48984532

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SE2013/050121 WO2013122536A1 (fr) 2012-02-14 2013-02-13 Ajout de données d'augmentation à une vue en perspective 3d

Country Status (1)

Country Link
WO (1) WO2013122536A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9575564B2 (en) 2014-06-17 2017-02-21 Chief Architect Inc. Virtual model navigation methods and apparatus
US9589354B2 (en) 2014-06-17 2017-03-07 Chief Architect Inc. Virtual model viewing methods and apparatus
US9595130B2 (en) 2014-06-17 2017-03-14 Chief Architect Inc. Virtual model navigation methods and apparatus
US10681183B2 (en) 2014-05-28 2020-06-09 Alexander Hertel Platform for constructing and consuming realm and object featured clouds
US10724864B2 (en) 2014-06-17 2020-07-28 Chief Architect Inc. Step detection methods and apparatus

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090102859A1 (en) * 2007-10-18 2009-04-23 Yahoo! Inc. User augmented reality for camera-enabled mobile devices
US20090285444A1 (en) * 2008-05-15 2009-11-19 Ricoh Co., Ltd. Web-Based Content Detection in Images, Extraction and Recognition
US20100257252A1 (en) * 2009-04-01 2010-10-07 Microsoft Corporation Augmented Reality Cloud Computing
US20110164163A1 (en) * 2010-01-05 2011-07-07 Apple Inc. Synchronized, interactive augmented reality displays for multifunction devices

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090102859A1 (en) * 2007-10-18 2009-04-23 Yahoo! Inc. User augmented reality for camera-enabled mobile devices
US20090285444A1 (en) * 2008-05-15 2009-11-19 Ricoh Co., Ltd. Web-Based Content Detection in Images, Extraction and Recognition
US20100257252A1 (en) * 2009-04-01 2010-10-07 Microsoft Corporation Augmented Reality Cloud Computing
US20110164163A1 (en) * 2010-01-05 2011-07-07 Apple Inc. Synchronized, interactive augmented reality displays for multifunction devices

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
BLAIR MACLNTYRE ET AL.: "The Argon AR Web Browser and standards-based AR application environment", MIXED AND AUGMENTED REALITY (ISMAR), 2011 10TH IEEE INTERNATIONAL SYMPOSIUM, 26 October 2011 (2011-10-26) *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10681183B2 (en) 2014-05-28 2020-06-09 Alexander Hertel Platform for constructing and consuming realm and object featured clouds
US11368557B2 (en) 2014-05-28 2022-06-21 Alexander Hertel Platform for constructing and consuming realm and object feature clouds
US11729245B2 (en) 2014-05-28 2023-08-15 Alexander Hertel Platform for constructing and consuming realm and object feature clouds
US9575564B2 (en) 2014-06-17 2017-02-21 Chief Architect Inc. Virtual model navigation methods and apparatus
US9589354B2 (en) 2014-06-17 2017-03-07 Chief Architect Inc. Virtual model viewing methods and apparatus
US9595130B2 (en) 2014-06-17 2017-03-14 Chief Architect Inc. Virtual model navigation methods and apparatus
US10724864B2 (en) 2014-06-17 2020-07-28 Chief Architect Inc. Step detection methods and apparatus

Similar Documents

Publication Publication Date Title
US10114480B2 (en) Screen content display method and system
US11321925B2 (en) Mixed-reality system, program, method, and portable terminal device
TWI671677B (zh) 交互處理方法及裝置
WO2015143956A1 (fr) Procédé et appareil de blocage d'une publicité dans une page web
CN104036011B (zh) 网页元素的显示方法以及浏览器装置
EP2909998B1 (fr) Pages internet ayant des fonctions matérielles incorporées
CN107943547B (zh) 多服务集成方法、装置、智能终端、服务器和操作系统
US8490117B1 (en) Bridging script engines
US20150279310A1 (en) Information processing terminal, display terminal, and display method
WO2013048329A2 (fr) Procédé de capture d'événements devant un dispositif de caméra
WO2013122536A1 (fr) Ajout de données d'augmentation à une vue en perspective 3d
US20180189083A1 (en) Method and device for operating target application on corresponding equipment
EP3748500A2 (fr) Modèles de services de composants d'inversion de contrôle destinés à des environnements virtuels
US20140201619A1 (en) Method And Device For Implementing Local Operation On Web Page
CN107798064B (zh) 页面处理方法、电子设备及计算机可读存储介质
CN109040339B (zh) 基于ajax的跨域请求处理方法、装置及设备
EP3016012A1 (fr) Procédé et dispositif de marquage terminal
US20130081010A1 (en) Template and server content download using protocol handlers
CN105786476B (zh) 移动客户端与服务器之间的数据处理方法及系统
CN105260383B (zh) 一种用于展现网页图像信息的处理方法及电子设备
CN106462911B (zh) 提供云流服务的方法及其装置和系统
CN106919260B (zh) 网页操作方法和装置
US11758016B2 (en) Hosted application as web widget toolkit
CN113822978B (zh) 电子地图渲染方法及装置
WO2019085654A1 (fr) Procédé et dispositif de traitement de données de pages, terminal client, et serveur

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13749588

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20-11-2014)

122 Ep: pct application non-entry in european phase

Ref document number: 13749588

Country of ref document: EP

Kind code of ref document: A1