WO2016095176A1

WO2016095176A1 - Interacting with a perspective view

Info

Publication number: WO2016095176A1
Application number: PCT/CN2014/094276
Authority: WO
Inventors: Descheng LI
Original assignee: Nokia Technologies Oy; Navteq (Shanghai) Trading Co., Ltd.
Priority date: 2014-12-18
Filing date: 2014-12-18
Publication date: 2016-06-23

Abstract

According to an example aspect of the present invention, there is provided an apparatus, comprising a receiver configured to receive camera sensor data, and at least one processing core configured to determine, based on the camera sensor data, at least one of a first rotation increment and a first translation increment, to obtain, based at least in part on the at least one of the first rotation increment and the translation increment, an updated perspective view, and to cause displaying of the updated perspective view on a display.

Description

INTERACTING WITH A PERSPECTIVE VIEW

FIELD OF INVENTION

The present invention relates to browsing， navigating and/or operating a perspective view， such as for example a street view.

BACKGROUND OF INVENTION

Maps printed on paper typically present a projection of map data in a bird’s-eye view， with streets and buildings depicted as if viewed from above. These maps resemble simplified satellite photographs from the point of view of their perspective. An advantage of this perspective is that navigating streets using this perspective is simple， since all streets are rendered on the map as if from above， and map users can mentally rotate the projection to enable making choices as to which way to turn.

Maps viewed on dynamic displays capable of changing the information displayed can be viewed in different perspectives. For example， a digitally displayed map may be presented so that a far edge from the point of view of a user points ahead of the user， so that the user needn’t mentally rotate the map when using it. To enable this， a compass may be employed to determine the orientation of a device， and the map， presented in the bird’s-eye view， is rotated to facilitate use by the user.

Dynamic displays are currently in wide use by consumers in their devices， such as for example mobile phones， smartphones， tablet computers and electronic readers. Map software may be installed in such devices to enable map use in addition to other uses the devices may have. Purpose-built personal map devices are also available， where mapping is the main or sole purpose of the device.

Digital maps may offer a user a selection of views， for example， the user may select a satellite image view or a traditional map view. A satellite image view has the advantage of providing a more realistic depiction of trees， buildings and other long-term features in a landscape. An advantage of a traditional map view is that it removes cluttering elements present in a satellite view. Examples of cluttering elements include clouds and other transient content， such as cars and billboards that may have changed before the user views the image.

In addition to the bird’s-eye， or satellite， perspective some maps now enable a street view， by which it is meant a view from a street of the street. Capturing a street view may comprise driving a car outfitted with large-angle cameras along streets and capturing a large number of images. This enables an immersive perspective， where a user may pre-view a walk or drive along a planned route and see the buildings along the route from the same， or a similar， perspective as he will later see them when actually moving along the route.

SUMMARY OF THE INVENTION

The invention is defined by the features of the independent claims. Some specific embodiments are defined in the dependent claims.

According to a first aspect of the present invention， there is provided an apparatus， comprising a receiver configured to receive camera sensor data， and at least one processing core configured to determine， based on the camera sensor data， at least one of a first rotation increment and a first translation increment， to obtain， based at least in part on the at least one of the first rotation increment and the translation increment， an updated perspective view， and to cause displaying of the updated perspective view on a display.

Various embodiments of the first aspect may comprise at least one feature from the following bulleted list：

·the camera sensor data is derived from at least one camera sensor comprised in the apparatus

·the display is comprised in the apparatus

·at least one of the at least one camera sensor is comprised in a back-facing camera of the apparatus

·at least one of the at least one camera sensor is comprised in a front-facing camera of the apparatus

·the at least one processing core is configured to obtain the updated perspective view from a remote server

·the at least one processing core is configured to obtain the updated perspective view from a memory comprised in the apparatus

·the first rotation increment comprises a yaw angle increment of the apparatus

·the at least one processing core is configured to determine， based on the camera sensor data， a second rotation increment and to obtain， based at least in part on the second rotation increment， the updated perspective view

·the second rotation increment comprises a pitch angle increment of the apparatus

·the at least one processing core is configured to determine， based on the camera sensor data， a zoom increment and to obtain， based at least in part on the zoom increment， the updated perspective view

·the at least one processing core is configured to determine the first rotation increment at least in part based on running an edge detection algorithm on the camera sensor data

·the perspective view comprises a perspective street view.

According to a second aspect of the present invention， there is provided a method， comprising receiving， in an apparatus， camera sensor data， determining， based on the camera sensor data， at least one of a first rotation increment and a first translation increment， obtaining， based at least in part on the at least one of the first rotation increment and a first translation increment， an updated perspective view， and causing displaying of the updated perspective view on a display.

Various embodiments of the second aspect may comprise at least one feature from the preceding bulleted list laid out in connection with the first aspect.

According to a third aspect of the present invention， there is provided an apparatus， comprising means for receiving， in an apparatus， camera sensor data， means for determining， based on the camera sensor data， at least one of a first rotation increment and a first translation increment， means for obtaining， based at least in part on the at least one of the first rotation increment and the first translation increment， an updated perspective view， and means for causing displaying of the updated perspective view on a display.

According to a fourth aspect of the present invention， there is provided a non-transitory computer readable medium having stored thereon a set of computer readable instructions that， when executed by at least one processor， cause an apparatus to at least receive， in an apparatus， camera sensor data， determine， based on the camera sensor data， at least one of a first rotation increment and a first translation increment， obtain， based at least in part on the at least one of the first rotation increment and the first translation increment， an updated perspective view， and cause displaying of the updated perspective view on a display.

According to a fourth aspect of the present invention， there is provided a system， comprising a user device configured to determine， based on camera sensor data， at least one of a first rotation increment and a first translation increment and to transmit a request， based at least in part on the at least one of the first rotation increment and the first translation increment， for perspective view data， and a server device， configured to receive the request， and to responsively provide the perspective view data to the user device.

According to a fourth aspect of the present invention， there is provided a computer program configured to cause a method in accordance with the second aspect to be performed.

Industrial Applicability

At least some embodiments of the present invention find industrial application in providing a more efficient man-machine interface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGURE 1 illustrates an example system capable of supporting at least some embodiments of the present invention；

FIGURE 2 illustrates an example use case in accordance with at least some embodiments of the present invention；

FIGURE 3 illustrates an example apparatus capable of supporting at least some embodiments of the present invention；

FIGURE 4 illustrates signalling in accordance with at least some embodiments of the present invention；

FIGURE 5 is a first flow chart of a first method in accordance with at least some embodiments of the present invention；

FIGURE 6 is a second flow chart of a second method in accordance with at least some embodiments of the present invention， and

FIGURE 7 illustrates angle naming conventions.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Using camera sensor data to cause navigation， such as rotation and translation， of a street view， or generally a perspective view， renders using a perspective view more effective on a small device. In particular， navigational user interface elements can be eliminated from the view， thus enabling unobstructed display of a larger fraction of the perspective view data.

FIGURE 1 illustrates an example system capable of supporting at least some embodiments of the present invention. The system of FIGURE 1 comprises a perspective view provider 120， which may comprise， for example， a street view provider. Perspective view provider 120 will hereinafter be referred to as server 120 for brevity. Likewise the term street view will be used hereinafter， however it is to be understood that in some embodiments of the invention， another kind of perspective view is used instead of a street view. For example， a three-dimensional medical imaging perspective view may be used depicting， for example， a shape of a tumour or arterial blockages to enable immersive observation of their features. Server 120 is in communication with network 130 via connection 123. Connection 123 may comprise， for example， a wire-line connection， such as for example a coaxial Ethernet connection or a fibre-optic connection.

Network 130 may comprise， for example， the Internet and/or at least one internet service provider， ISP， corporate network. Network 130 may be based on internet protocol， IP， switching， for example. IP may comprise version 4， IPv4， or version 6， IPv6， or a combination thereof， for example.

Base station 140 is in communication with network 130 via connection 134. Connection 134 may comprise， for example， a wire-line connection， such as for example a coaxial Ethernet connection or a fibre-optic connection. Base station 140 may comprise， for example， a cellular base station configured to operate in accordance with at least one cellular radio access technology， RAT. Examples of such technologies include wideband code division multiple access， WCDMA， and long term evolution， LTE.

Device 110 may comprise， for example， a mobile phone， smartphone， tablet computer， phablet device or other suitable electronic device with a display， internal memory and at least one processing core or processor. In the system of FIGURE 1， device 110 is in communication with base station 140 via cellular radio link 141. Cellular radio link 141 may be arranged in accordance with a cellular RAT that both base station 140 and device 110 can support， for example WCDMA or LTE.

Access node 150 is in communication with network 130 via connection 135. Connection 135 may comprise， for example， a wire-line connection， such as for example a coaxial Ethernet connection or a fibre-optic connection. Access node 150 may comprise， for example， a wireless access point configured to operate in accordance with at least one non-cellular RAT. Examples of such technologies include wireless local area network， WLAN， and worldwide interoperability for microwave access， WiMAX.

Laptop 110’ may comprise， for example an ultrabook， portable workstation or other suitable laptop device with a display， internal memory and at least one processing core or processor. In the system of FIGURE 1， laptop 110’ is in communication with access node 150 via non-cellular radio link 151. Non-cellular radio link 151 may be arranged in accordance with a non. cellular RAT that both access node 150 and laptop 110’ can support， for example WLAN or WiMAX. In some embodiments， device 110 is enabled to connect to access node 150 using the non-cellular RAT. In some embodiments， laptop 110’ is absent.

Device 110 may be configured to determine its location， for example by using an inbuilt satellite positioning receiver. Examples of suitable satellite positioning systems include global positioning system， GPS， Galileo and GLONASS systems. In some embodiments， more than one satellite positioning system is used to improve accuracy. Device 110 may also determine its orientation， for example from the satellite positioning system or from an inbuilt compass function comprised in device 110. In addition to， or alternatively to， a satellite positioning system， device 110 may employ a cellular network positioning function to obtain its location， for example.

Laptop 110’ may likewise be configured to determine its location， for example using similar methods as device 110. Where radio link 151 is short range， laptop 110’ may use radio link 151 to facilitate determining the location of laptop 110’ .

Server 120 may store a dataset of street views， which may be obtained， for example， by driving camera cars along streets in at least one city. A street view may comprise a single field of vision from a single point. The field of vision may be limited in terms of pitch angle and yaw angle， as in street view 100 illustrated in FIGURE 1， or a street view may comprise a spherical view from a single point to all directions， depending on the embodiment. By pitch angle it is meant an angle from the horizontal， for example the ground， corresponding to the vertical extent in street view 100. By yaw angle it is meant an angle from a radial axis extending from an observer， corresponding to the horizontal extent in street view 100. The yaw angle may alternatively be considered a heading， defined in relation to true north， for example.

Device 110 and/or laptop 110’ may request a street view from server 120， for example by transmitting a request to server 120. The request may be routed via network 130. The request may comprise a location estimate. Responsive to the request， server 120 may provide， again via network 130 or via another route， a street view relating to the location estimate. In other words， server 120 may be configured to provide， upon request， a street view of the location identified in the location estimate. The provided street view may be a spherical street view. In some embodiments， the provided street view is a spherical street view which lacks a solid angle section corresponding to a skyward patch and/or a solid angle section corresponding to a downward patch. For example， the spherical street view may lack data for pitch angles exceeding a first angle and/or below a second angle. A pitch angle of zero may be defined as a horizontal direction parallel to the ground at a height of 1,5 meters， for example. In some embodiments， the request lacks a location estimate， but comprises an explicit or implicit request to localize the device sending the request. Server 120 may then responsively determine the location of the device， and transmit the street view concerning that location to the device.

Once in possession of the street view received from server 120， device 110 or laptop 110’ may provide on a screen a display of the street view， for example， a section of the street view selected based on a determination as to a current orientation of device 110 or laptop 110’ . In other words， the user may be presented with a view that corresponds to what he sees around his device with his own eyes. The view may be augmented with additional information， such as navigation instructions， for example. Alternatively to this， the request sent to the server may already comprise the current orientation， and server 120 may provide the correct section of the entire spherical street view for display. An advantage of this is that a smaller quantity of data needs to be transmitted， whereas an advantage of transmitting the entire spherical view is that the user is enabled to pan the view without requesting， and waiting for， additional sections of the spherical view from server 120.

Alternatively to requesting a street view of a current location， device 110 and/or laptop 110’ may request and receive from server 120 a street view of a different location. The requested location may correspond to a location the user intends to visit， for example. Responsively， device 110 or laptop 110’ may present a section of the street view received from server 120， which however need not correspond to a present location and/or orientation of device 110 or laptop 110’ .

Navigating a street view， or perspective view in general， comprises changing the viewing angle and/or zooming. In terms of a currently displayed view， an identified rotation increment causes the displayed view to change. For example， a yaw rotation increment of 5 degrees would cause the section of the spherical street view that is displayed on the screen to shift by 5 degrees sideways， bringing some previously undisplayed content on-screen and shifting some previously displayed content off-screen. Rotation in this regard loosely resembles a human turning his head to view a different section of his surroundings. Likewise a rotation increment concerning the pitch angle would cause the section of the spherical street view that is displayed on the screen to shift upward or downward along the spherical street view， bringing some previously undisplayed content on-screen and shifting some previously displayed content off-screen. In general， by increment it is meant a change， which may be an increase or a decrease in absolute value.

Zooming may comprise either zooming in the spherical street view to inspect a smaller detail comprised in the view. An operation to change a location where the spherical view is captured from， known also as a translation， may trigger a new request to server 120 for a street view at the new location. A translation thus corresponds to moving in the immersive street view， for example along a road. A specific translation may be considered a translation increment in the sense that the translation increment corresponds to the extent to which the location where the street view is captured from changes.

According to various embodiments of the present invention， navigating the street view is accomplished with the aid of at least one camera comprised in device 110 or laptop 110’ . For example， in terms of device 110， it may comprise a back-facing camera. A back-facing camera may be disposed in device 110 on an opposite side of the device with respect to its display. When the user is holding device 110 in front of himself， the back-facing camera may have an unobstructed view into the surroundings of device 110.

Device 110 may be configured to switch the back-facing camera on， and to detect objects and/or patterns in camera sensor data produced by the back-facing camera. In case the user pans device 110， meaning he turns it sideways， meaning he changes the yaw angle of the orientation of the device， this can be determined from observing how sensor data from the back-facing camera changes. In case the turn is to the right， objects in the back-facing view appear to shift to the left in the image. As a response to determining a change in the yaw angle， the street view displayed on the display of the device may be navigated， in detail by changing the yaw angle of the view into the spherical street view by the same， or a similar， yaw angle increment. Thus the user may experience the view in the display as a window into the spherical street view that he can pan by turning device 110. In case the device is tipped upward or downward， a change in the pitch angle of the orientation of the device may be determined in a similar manner. When the device is tipped downward， objects and patterns in the view of the back-facing camera appear to move upward. When the device is tipped upward， objects and patterns in the view of the back-facing camera appear to move downward.

Laptop 110’ may likewise have a back-facing camera， and/or a front-facing camera， known as a webcam. A front-facing camera in laptop 110’ or device 110 may be used to determine an angle increment much like the back camera can， although the user may be present in the field of view of the front-facing camera. In this case， device 110 or laptop 110’ may be configured to identify from the view a background， and to determine the angle increment based on the background only. In some embodiments， device 110 or laptop 110’ is configured to use both a back-facing camera and a front-facing camera. In detail， an angle increment may be obtained independently using sensor data from the back-facing camera and from the front-facing camera. A final angle increment， obtained as an average of the two independently obtained angle increments， may then be used to navigate the street view displayed on the screen.

In case device 110 or laptop 110’ is moved forward， objects in the field of view of the back-facing camera seem to become enlarged. This may be detected， again from the sensor data of the back-facing camera， and used to navigate the street view by causing a translation of viewpoint in the immersive view. For example， the user may walk along a street， and a new street view may be requested from server 120 that better matches the new location of the user in the immersive view.

In case device 110 or laptop 110’ is moved backward， objects in the field of view of the back-facing camera seem to become smaller. This may be detected， again from the sensor data of the back-facing camera， and used to navigate the street view by causing a translation of viewpoint in the immersive view. For example， the user may walk along a street， and a new street view may be requested from server 120 that better matches the new location of the user. A more usual use case could be a turn followed by walking forward， since people rarely walk backward.

In some embodiments， street view data may be cached， at least in part， in device 110 or laptop 110’ ， to thereby reduce the need for requesting updated street view data from server 120. For example， device 110 may request a plurality of spherical street views at a go to anticipate directions where the displayed view may be translated to.

FIGURE 2 illustrates an example use case in accordance with at least some embodiments of the present invention. Like numbering denotes like features as in FIGURE 1. In FIGURE 2， street view 200 is a section of a spherical street view， and section 240 represents a section of the spherical street view currently displayed on the screen of device 110. View 210 represents a processed version of sensor data incoming from a camera sensor of a back camera of device 110. Comprised in view 210 is object 220， which may be detected， for example， at least in part by employing an edge detection algorithm to sensor data from the back camera. Examples of suitable objects comprise desks， plants and equipment.

In FIGURE 2， object 220 is determined to move toward the right in accordance with arrow 221. This is interpreted as a tilting of device 110 along the yaw axis， wherefore section 240 is panned to the left along the spherical street view， to display content previously not displayed to the left of section 240.

In some embodiments， more than one object or pattern is determined in view 210， and a tilt of device 110 is determined as a response to a plurality of them moving in the same direction. This has the effect that a single object， which may be a car， cannot trigger navigation of the street view by moving on its own， independently of any tilting of device 110.

FIGURE 3 illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is device 300， which may comprise， for example， a device 110 of FIGURE 1 or FIGURE 2， or laptop 110’ of FIGURE 1， for example. Comprised in device 300 is processor 310， which may comprise， for example， a single-or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor 310 may comprise a Qualcomm Snapdragon 800 processor， for example. Processor 310 may comprise more than one processor. A processing core may comprise， for example， a Cortex-A8 processing core manufactured by ARM Holdings or a Brisbane processing core produced by Advanced Micro Devices Corporation. Processor 310 may comprise at least one Qualcomm Snapdragon and/or Intel Atom processor. Processor 310 may comprise at least one application-specific integrated circuit， ASIC. Processor 310 may comprise at least one field-programmable gate array， FPGA. Processor 310 may be means for performing method steps in device 300. Processor 310 may be configured， at least in part by computer instructions， to perform actions.

Device 300 may comprise memory 320. Memory 320 may comprise random-access memory and/or permanent memory. Memory 320 may comprise at least one RAM chip. Memory 320 may comprise magnetic， optical and/or holographic memory， for example. Memory 320 may be at least in part accessible to processor 310. Memory 320 may be means for storing information. Memory 320 may comprise computer instructions that processor 310 is configured to execute. When computer instructions configured to cause processor 310 to perform certain actions are stored in memory 320， and device 300 overall is configured to run under the direction of processor 310 using computer instructions from memory 320， processor 310 and/or its at least one processing core may be considered to be configured to perform said certain actions. Memory 320 may be at least in part comprised in processor 310.

Device 300 may comprise a transmitter 330. Device 300 may comprise a receiver 340. Transmitter 330 and receiver 340 may be configured to transmit and receive， respectively， information in accordance with at least one cellular or non-cellular standard. Transmitter 330 may comprise more than one transmitter. Receiver 340 may comprise more than one receiver. Transmitter 330 and/or receiver 340 may be configured to operate in accordance with global system for mobile communication， GSM， wideband code division multiple access， WCDMA， long term evolution， LTE， IS-95， wireless local area network， WLAN， Ethernet and/or worldwide interoperability for microwave access， WiMAX， standards， for example.

Device 300 may comprise a near-field communication， NFC， transceiver 350. NFC transceiver 350 may support at least one NFC technology， such as NFC， Bluetooth， Wibree or similar technologies.

Device 300 may comprise user interface， UI， 360. UI 360 may comprise at least one of a display， a keyboard， a touchscreen， a vibrator arranged to signal to a user by causing device 300 to vibrate， a speaker and a microphone. A user may be able to operate device 300 via UI 360， for example to accept incoming telephone calls， to originate telephone calls or video calls， to browse the Intemet， to manage digital files stored in memory 320 or on a cloud accessible via transmitter 330 and receiver 340， or via NFC transceiver 350， and/or to initiate programs， such as for example a perspective view display program.

Device 300 may comprise or be arranged to accept a user identity module 370. User identity module 370 may comprise， for example， a subscriber identity module， SIM， card installable in device 300. A user identity module 370 may comprise information identifying a subscription of a user of device 300. A user identity module 370 may comprise cryptographic information usable to verify the identity of a user of device 300 and/or to facilitate encryption of communicated information and billing of the user of device 300 for communication effected via device 300.

Processor 310 may be furnished with a transmitter arranged to output information from processor 310， via electrical leads internal to device 300， to other devices comprised in device 300. Such a transmitter may comprise a serial bus transmitter arranged to，for example， output information via at least one electrical lead to memory 320 for storage therein. Alternatively to a serial bus， the transmitter may comprise a parallel bus transmitter. Likewise processor 310 may comprise a receiver arranged to receive information in processor 310， via electrical leads internal to device 300， from other devices comprised in device 300. Such a receiver may comprise a serial bus receiver arranged to， for example， receive information via at least one electrical lead from receiver 340 for processing in processor 310. Alternatively to a serial bus， the receiver may comprise a parallel bus receiver.

Device 300 may comprise further devices not illustrated in FIGURE 3. For example， where device 300 comprises a smartphone， it may comprise at least one digital camera. Some devices 300 may comprise a back-facing camera and a front-facing camera， wherein the back-facing camera may be intended for digital photography and the front-facing camera for video telephony. Device 300 may comprise a fingerprint sensor arranged to authenticate， at least in part， a user of device 300. In some embodiments， device 300 lacks at least one device described above. For example， some devices 300 may lack a NFC transceiver 350 and/or user identity module 370.

Processor 310， memory 320， transmitter 330， receiver 340， NFC transceiver 350， UI 360 and/or user identity module 370 may be interconnected by electrical leads internal to device 300 in a multitude of different ways. For example， each of the aforementioned devices may be separately connected to a master bus internal to device 300， to allow for the devices to exchange information. However， as the skilled person will appreciate， this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

FIGURE 4 illustrates signalling in accordance with at least some embodiments of the present invention. The vertical axes， from left to right， correspond to device 110 and server 120 of FIGURE 1. Time advances from the top toward the bottom.

In phase 410， a street view display is initialized in device 110. For example， a street view program or application may be loaded into execution memory and/or a location of device 110 may be determined， for example based on a satellite navigation receiver and/or a cellular location feature. In phase 420， device 110 requests from server 120 a street view， wherein the request may comprise the determined location of device 110 or a request for server 120 to localize device 110. The location may be expressed as a combination of longitude and latitude， for example. Optionally， the request may also comprise information concerning an orientation of device 110. An orientation may be expressed as a combination of pitch and yaw angles， or alternatively simply as the yaw angle.

In phase 430， server 120 provides， responsively， the street view requested in phase 420. The street view may be a spherical street view， for example. In case the request of phase 420 comprised the information concerning the orientation， the street view provided in phase 430 may be a sectional street view， less than the spherical street view， enabling displaying a section of the spherical street view into the current orientation of device 110. In case only a yaw angle， or information allowing the yaw angle to be determined， is provided， the section may initially be displayed at a default pitch angle.

In phase 440， device 110 renders to the user the section of the street view. The rendered section may be a section of a received spherical street view， the received sectional street view， or， for example， a section of a received section of the spherical street view. In phase 450， device 110 determines， based on camera sensor data， a first rotation increment. The camera sensor data may originate in a back-facing and/or a front-facing camera， as described above. If device 110 can navigate the displayed street view based on street view data it has received in phase 430， it does so. This may be the case， for example， where device 110 received a spherical street view in phase 430. This may also be the case， for example， where device 110 received a half of a spherical street view in phase 430， and the rotation increment is small enough so that the displayed section may be updated without requesting updated street view data from server 120.

In phase 460， device 110 requests updated street view data from server 120. For example， this may occur as a response to the user navigating， by tilting device 110， in the street view to such an extent that device 110 needs mode street view data from server 120. Alternatively， or additionally， this may occur as a response to the user navigating in the street view by translating to a different location in the street view， necessitating a new spherical street view， or section of a new spherical street view， to be obtained in device 110. In phase 470， server 120 responsively provides the updated street view data to device 110. For example， the user may indicate a movement along a street in the displayed street view to necessitate street view data from a different location than what was originally displayed.

FIGURE 5 is a first flow chart of a first method in accordance with at least some embodiments of the present invention. The phases of the method may be performed in device 110 or laptop 110’ ， for example. Phase 510 comprises initializing the street view， or more generally perspective view， system. Phase 520 comprises running the background， or back-facing， camera. Phase 530 comprises detecting and tracking features in camera sensor data from the background camera. Phase 540 comprises computing local parameters， which may comprise， for example， determining a rotation increment. Phase 550 comprises adjusting parameters， such as for example navigating a street view on a screen. Phase 560 comprises acquiring street view data， such as for example updated street view data. Phase 570 comprises displaying a current， or updated， street view， which may be based， for example， on the street view data acquired in phase 560. After phase 570， processing returns to phase 520.

In general， a street view may be in specific modes. For example， in a first rotation scheme， only the yaw angle is modified， with the pitch angle being kept constant. In a second rotation scheme， the yaw angle is kept constant and the pitch angle is modified. Angles may be viewing angles， modified based on camera sensor data， as described above. A street view may alternatively be in a fixed scheme， wherein pitch and yaw angles are fixed， and translation is enabled. In this case， moving the device further from， and toward， the user， would cause a translation of the point from which the street view is being observed. In some cases， a scale may be determined wherein a relatively small movement of the device may cause a much larger movement in the point from which the street view is being observed. This may enable， for example， navigating a street or city in the street view while the user himself doesn’t move. In a free mode， translation and increments in pitch and yaw may all be enabled， for example， based on camera sensor data as described herein.

FIGURE 6 is a second flow chart of a second method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may take place in device 110， laptop 110’ or in a control device configured to control the functioning thereof， for example.

Phase 610 comprises receiving， in an apparatus， camera sensor data. Phase 620 comprises determining， based on the camera sensor data， a first rotation increment. The first rotation increment may be an increment in pitch and/or yaw angle， for example. In addition to， or alternatively to， the first rotation increment， a first translation increment may be determined based at least in part on the camera sensor data. Phase 630 comprises obtaining， based at least in part on the first rotation increment and/or the first translation increment， an updated perspective view. The updated perspective view may be obtained， for example， by retrieving from a memory the updated perspective view， or it may be obtained by requesting the updated perspective view from a server. Finally， phase 640 comprises causing displaying of the updated perspective view on a display

Next are described example mathematical methods usable in implementing at least some example embodiments of the present invention. As is apparent to the skilled reader， these methods are examples to which the present invention is not limited. Rather， it is clear that other mathematical methods can also be used to obtain the benefits of the present invention. First， a translation distance computing algorithm is disclosed. Reference is made to FIGURE 7 for illustration of variables used herein.

According to the size of the object changing， judging the sign of translation amplitude Δdist. If objects disappear， reducing to cannot be detected or enlarging to the outside of the scope， then re-choose an object to do the matching， tracing and reload this computing algorithm. Otherwise， calculating the zoom and Δdist by delta_dist () method may be accomplished as detailed below.

delta_dist () method：

d_prs ： The previous distance between object and lens

d_curr ： The current distance between object and lens

l_prs ： The length of object image in previous frame

l_curr ： The length of object image in previous frame

F： The focal length

L： The length of the object

zoom： The amplification coefficient

Δdist： The translation of distance

There exist the following equations.

Then we will calculate the zoom and Δdist.

In other words， the translation increment Δdist is determined in this example based on d_curr， d_pre， l_curr， and l_pre.

Angle computing algorithm： The steps of angle computing algorithm are as follows： 1) According to the zoom and center position to restore the image， 2) Judging the absolute displacement of the object， 3) If the object reaches or exceeds the boundary of the frame， then call the delta_angle () method and calculate the ψ＝θ+α， abandon the traced object in the target area， and restart the feature matching and tracing in the area again. After that， repeat this angle computing algorithm again. 4) Otherwise， the object is in the scope of this frame， then call delta_angle () method to calculate ψ， and repeat this angle computing algorithm. In other words， a suitable object is used to determine the delta_angle， wherein the object need not be the same in each increment determination as long as the rotation increment is successfully determined.

delta_angle () method： There exist the following equations：

ψ＝α+θ

θ＝90-2α-σ

Then we will calculate the ψ.

The computation method of pitch ω is similar， and is omitted herein.

Street view parameters incremental adjustment： The steps of this phase include： According to the local parameters： heading， or yaw， (ψ) and pitch (ω) form the rotation matrix Cⁿ. The following equation applies：

Based on the rotation matrix Cⁿ， constitute the <X， Y， Z>

A technical effect of the foregoing mathematical methods is， in the concerned embodiments， that an efficient determination of rotational and/or translational increments is enabled in the device， such as for example device 110 of FIGURE 1. Similar considerations would apply where the device is laptop 110’ of FIGURE 1.

FIGURE 7 illustrates angle naming conventions used in the foregoing equations.

It is to be understood that the embodiments of the invention disclosed are not limited to the particular structures， process steps， or materials disclosed herein， but are extended to equivalents thereof as would be recognized by those ordinarily skilled in the relevant arts. It should also be understood that terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature， structure， or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus， appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

As used herein， a plurality of items， structural elements， compositional elements， and/or materials may be presented in a common list for convenience. However， these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus， no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on their presentation in a common group without indications to the contrary. In addition， various embodiments and example of the present invention may be referred to herein along with alternatives for the various components thereof. It is understood that such embodiments， examples， and alternatives are not to be construed as de facto equivalents of one another， but are to be considered as separate and autonomous representations of the present invention.

Furthermore， the described features， structures， or characteristics may be combined in any suitable manner in one or more embodiments. In the following description， numerous specific details are provided， such as examples of lengths， widths， shapes， etc. ， to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize， however， that the invention can be practiced without one or more of the specific details， or with other methods， components， materials， etc. In other instances， well-known structures， materials， or operations are not shown or described in detail to avoid obscuring aspects of the invention.

While the forgoing examples are illustrative of the principles of the present invention in one or more particular applications， it will be apparent to those of ordinary skill in the art that numerous modifications in form， usage and details of implementation can be made without the exercise of inventive faculty， and without departing from the principles and concepts of the invention. Accordingly， it is not intended that the invention be limited， except as by the claims set forth below.

Claims

An apparatus， comprising：

a receiver configured to receive camera sensor data；

at least one processing core configured to determine， based on the camera sensor data， at least one of a first rotation increment and a first translation increment， to obtain， based at least in part on the at least one of the first rotation increment and the translation increment， an updated perspective view， and to cause displaying of the updated perspective view on a display.
The apparatus according to claim 1， wherein the camera sensor data is derived from at least one camera sensor comprised in the apparatus.
The apparatus according to claim 1 or 2， wherein the display is comprised in the apparatus.
The apparatus according to any of claims 2-3， wherein at least one of the at least one camera sensor is comprised in a back-facing camera of the apparatus.
The apparatus according to any of claims 2-4， wherein at least one of the at least one camera sensor is comprised in a front-facing camera of the apparatus.
The apparatus according to any of claims 1-5， wherein the at least one processing core is configured to obtain the updated perspective view from a remote server.
The apparatus according to any of claims 1-6， wherein the at least one processing core is configured to obtain the updated perspective view from a memory comprised in the apparatus.
The apparatus according to any of claims 1-7， wherein the first rotation increment comprises a yaw angle increment of the apparatus.
The apparatus according to any of claims 1-8， wherein the at least one processing core is configured to determine， based on the camera sensor data， a second rotation increment and to obtain， based at least in part on the second rotation increment， the updated perspective view.
The apparatus according to claim 9， wherein the second rotation increment comprises a pitch angle increment of the apparatus.
The apparatus according to any of claims 1-10， wherein the at least one processing core is configured to determine， based on the camera sensor data， a zoom increment and to obtain， based at least in part on the zoom increment， the updated perspective view.
The apparatus according to any of claims 1-11， wherein the at least one processing core is configured to determine the first rotation increment at least in part based on running an edge detection algorithm on the camera sensor data.
The apparatus according to any of claims 1-12， wherein the perspective view comprises a perspective street view.
A method， comprising：

receiving， in an apparatus， camera sensor data；

determining， based on the camera sensor data， at least one of a first rotation increment and a first translation increment；

obtaining， based at least in part on the at least one of the first rotation increment and a first translation increment， an updated perspective view， and

causing displaying of the updated perspective view on a display.
The method according to claim 14， wherein the camera sensor data is derived from at least one camera sensor comprised in the apparatus.
The method according to claim 14 or 15， wherein the display is comprised in the apparatus.
The method according to any of claims 15-16， wherein at least one of the at least one camera sensor is comprised in a back-facing camera of the apparatus.
The method according to any of claims 15-17， wherein at least one of the at least one camera sensor is comprised in a front-facing camera of the apparatus.
The method according to any of claims 14-18， wherein the updated perspective view is obtained from a remote server.
The method according to any of claims 14-18， wherein the updated perspective view is obtained from a memory comprised in the apparatus.
The method according to any of claims 14-20， wherein the first rotation increment comprises a yaw angle increment of the apparatus.
The method according to any of claims 14-21， further comprising determining， based on the camera sensor data， asecond rotation increment and obtaining， based at least in part on the second rotation increment， the updated perspective view.
The method according to claim 22， wherein the second rotation increment comprises a pitch angle increment of the apparatus.
The method according to any of claims 14-23， further comprising determining， based on the camera sensor data， a zoom increment and obtaining， based at least in part on the zoom increment， the updated perspective view.
The method according to any of claims 14-24， wherein the first rotation increment is determined at least in part based on running an edge detection algorithm on the camera sensor data.
The method according to any of claims 14-25， wherein the perspective view comprises a perspective street view.
An apparatus， comprising：

means for receiving， in an apparatus， camera sensor data；

means for determining， based on the camera sensor data， at least one of a first rotation increment and a first translation increment；

means for obtaining， based at least in part on the at least one of the first rotation increment and the first translation increment， an updated perspective view， and

means for causing displaying of the updated perspective view on a display.
A computer program configured to cause a method in accordance with at least one of claims 14-26 to be performed.
A non-transitory computer readable medium having stored thereon a set of computer readable instructions that， when executed by at least one processor， cause an apparatus to at least：

receive， in an apparatus， camera sensor data；

determine， based on the camera sensor data， at least one of a first rotation increment and a first translation increment；

obtain， based at least in part on the at least one of the first rotation increment and the first translation increment， an updated perspective view， and

cause displaying of the updated perspective view on a display.
A system， comprising：

a user device configured to determine， based on camera sensor data， at least one of a first rotation increment and a first translation increment and to transmit a request， based at least in part on the at least one of the first rotation increment and the first translation increment， for perspective view data， and

a server device， configured to receive the request， and to responsively provide the perspective view data to the user device.
The system according to claim 30， wherein the request comprises an indication as to a location concerning which the perspective view data is requested.