US20070147828A1 - Image element identifier - Google Patents

Image element identifier Download PDF

Info

Publication number
US20070147828A1
US20070147828A1 US11/438,088 US43808806A US2007147828A1 US 20070147828 A1 US20070147828 A1 US 20070147828A1 US 43808806 A US43808806 A US 43808806A US 2007147828 A1 US2007147828 A1 US 2007147828A1
Authority
US
United States
Prior art keywords
image information
communication device
portable communication
presented
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US11/438,088
Other versions
US7513700B2 (en
Inventor
Joakim Nelson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Mobile Communications AB
Original Assignee
Sony Ericsson Mobile Communications AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Ericsson Mobile Communications AB filed Critical Sony Ericsson Mobile Communications AB
Priority to US11/438,088 priority Critical patent/US7513700B2/en
Assigned to SONY ERICSSON MOBILE COMMUNICATIONS AB reassignment SONY ERICSSON MOBILE COMMUNICATIONS AB ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NELSON, JOAKIM
Priority to PCT/EP2006/066187 priority patent/WO2007068511A1/en
Priority to EP06793370A priority patent/EP1960070A1/en
Priority to JP2008544901A priority patent/JP2009519066A/en
Priority to KR1020087016968A priority patent/KR20080080631A/en
Publication of US20070147828A1 publication Critical patent/US20070147828A1/en
Application granted granted Critical
Publication of US7513700B2 publication Critical patent/US7513700B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20Input arrangements for video game devices
    • A63F13/21Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/40Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
    • A63F13/42Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
    • A63F13/426Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50Controlling the output signals based on the game progress
    • A63F13/53Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/40Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
    • A63F13/42Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50Controlling the output signals based on the game progress
    • A63F13/53Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
    • A63F13/537Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game using indicators, e.g. showing the condition of a game character on screen
    • A63F13/5372Controlling the output signals based on the game progress involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game using indicators, e.g. showing the condition of a game character on screen for tagging characters, objects or locations in the game scene, e.g. displaying a circle under the character controlled by the player
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1087Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/30Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by output arrangements for receiving control signals generated by the game device
    • A63F2300/303Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by output arrangements for receiving control signals generated by the game device for displaying additional data, e.g. simulating a Head Up Display
    • A63F2300/306Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by output arrangements for receiving control signals generated by the game device for displaying additional data, e.g. simulating a Head Up Display for displaying a marker associated to an object or location in the game field
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/40Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterised by details of platform network
    • A63F2300/406Transmission via wireless network, e.g. pager or GSM
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/60Methods for processing data by generating or executing the game program
    • A63F2300/6045Methods for processing data by generating or executing the game program for mapping control signals received from the input arrangement into game commands

Definitions

  • the invention relates in general to controlling presentation of image information, and in particular to providing positioning information related to the presentation of image information.
  • Portable communication devices such as mobile phones, may be found having memories of 0.5 Gb or more.
  • Gaming using large screens is well known and was performed in the early days of electronic games, such as in television games using TV-sets as screens or in personal computer games using the computer cathode ray tubes as screens.
  • the television games typically provided a photo sensitive gun capable of sensing large bright target symbols on the screen of a television receiver.
  • the targets were distinctly displayed on the screen when imaged using a gun-mounted electro-optical sensor resulting in a series of output pulses indicating that the gun was indeed lined up properly with the target and a hit was scored. None of these attempted to show where the user was aiming when he pulled the trigger preceding a miss or for that matter, how closely to the center of the target on screen he was aiming when he made a hit.
  • a television precision target shooting apparatus and method simulate shooting at displayed images on the screen of a television set using a simulated gun that provides indication on the screen of where the gun was pointed at the time of trigger pull. This is achieved by flashing the screen white subsequent a trigger pull in order for a photo sensor within the gun to respond to short segments of horizontal raster lines such that by counting vertical lines and time intervals with respect to vertical and horizontal synchronization pulses, respectively, identification of where on the screen the gun was pointing is achieved. This permits a symbol to be displayed upon a screen at that location.
  • a gyro is used which is built in the directing device and which reacts to lateral and longitudinal motions of the directing device. Any lateral and longitudinal motions causing lateral and longitudinal displacements are related to a reference position, by which an updated position can be calculated.
  • the application of such a directing device in gaming may allow the position of a cursor or symbol to be determined and displayed on a screen.
  • An alternative to using a gyro to determine an updated position of a cursor or symbol on a screen is to use an accelerometer included in the directing or pointing device. Any motions of the directing or pointing device are detected by the accelerometer, and then converted to a displacement of the location where the directing or pointing device is pointing. A cursor may therefore also be displayed on the screen by using this technique.
  • aspects of the invention are directed towards solving the problem of interactively controlling an image presentation as displayed on a large screen, using a portable communication device.
  • aspects consistent with the invention provide a method for comparing image information being registered with image information being presented using a light registering camera unit of the portable communication device.
  • a method of providing positioning information related to a presentation of image information for a portable communication device comprises providing first image information to be presented by an image information presentation unit, obtaining the first image information by the portable communication device under control of the user of the portable communication device, registering second image information by the camera of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the registered second image information, and determining the position of a predefined area of the registered second image information in relation to the recognized at least one image element, where the determined relative position of the predefined area is used for controlling purposes of the first image information being presented.
  • a second aspect of the invention is directed towards a method including the features of the first aspect, further comprises the step of sending predefined area position information for displaying the predefined area position on the image information presentation unit.
  • a third aspect of the invention is directed towards a method including the features of the first aspect, wherein the step of determining the position of the predefined area comprises determining the position of the center area of second registered image information.
  • a fourth aspect of the invention is directed towards a method including the features of the first aspect, further comprising obtaining updated directing of the camera of the portable communication device under the control of the user of the portable communication device.
  • a fifth aspect of the invention is directed towards a method including the features of the first aspect, further comprising the step of receiving user input by the portable communication device, which user input is used as input data for controlling the first image information being presented.
  • a sixth aspect of the invention is directed towards a method including the features of the fourth aspect, further comprising the step of controlling the first image information being presented based on at least one of the obtained updated directing or received user input.
  • a portable communication device for providing positioning information related to a presentation of a first image information, comprises an image information providing unit, arranged to provide a first image information to an image information presentation unit, a light registering camera unit, arranged to register second image information when being directed towards the image presentation unit, a mapping unit, connected to the light registering camera unit and arranged to map the first image information and the second image information, and a control unit arranged to control providing of the first image information to an image information unit, to control registering second image information by the light registering camera unit, and to control mapping of the first image information and the second image information, and further arranged to recognize at least one image element of the presented first image information as being included in the second image information, to determine the position of a predefined area of the second registered image information in relation to the recognized at least one image element and use the determined relative position of the predefined area to control the first image information being presented.
  • An eighth aspect of the invention is directed toward a portable communication device including the features of the seventh aspect and wherein the control unit is further configured to use the determined relative position of the predefined area to control the first image information being presented.
  • a ninth aspect of the invention is directed towards a portable communication device including the features of the seventh aspect, wherein the portable communication device comprises a mobile phone.
  • Still other aspects of the invention provide a computer program product comprising a computer readable medium, having thereon computer program code means, to make a portable communication device or a computer execute, comparing image information being registered with image information being presented, using a light registering camera unit of the portable communication device.
  • a computer program product comprising a computer readable medium, having thereon computer program code for execution by a portable communication device or a computer, when said computer program code is loaded in the portable communication device or the computer, to cause the portable communication device or computer perform a method comprising: providing of first image information from the portable communication device or the computer, to be presented by an image information presentation unit, registering second image information by the camera of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the second registered image information, and determining the position of a predefined area of the second registered image information in relation to the recognized at least one image element.
  • An eleventh aspect of the invention is directed to a computer program product including the features of the tenth aspect, where the method performed by the portable communication device or computer further comprises using the determined relative position of the predefined area for controlling the first image information being presented.
  • aspects of the invention provide, among other advantages, positioning of an image center relative to recognized surrounding image elements, which enables more advanced gaming using mobile phones in connection with external image presentation units.
  • the invention may also provide a flexible positioning tool, which also can be used for controlling objects based on relative positioning of cursor movements.
  • FIG. 1 schematically illustrates an exemplary portable communication device consistent with the invention
  • FIG. 2 is a flow chart illustrating processing according to one embodiment of the invention
  • FIG. 3 schematically presents one simple example of first and second image information according to the invention
  • FIG. 4 illustrates a portable communication device in the form of a cellular phone, according to one embodiment of the invention.
  • FIG. 5 illustrates a computer program product having stored thereon computer program code to make a portable communication device or a computer execute steps of a method according to one embodiment of the invention.
  • aspects consistent with the invention are directed towards solving the problem of interactively controlling an image presentation as displayed on a large screen, using a portable communication device.
  • FIG. 1 schematically showing a portable communication device 100 , according to one embodiment of the present invention, the invention is now explained in more detail.
  • the portable communication device 100 comprises an image information providing unit 102 , a light registering camera unit 104 , a mapping unit 106 , a control unit 108 , a memory unit 110 , and a user input unit 112 .
  • the control unit 108 is connected to all other units. That is, control unit 108 is connected to the image information providing unit 102 , the light registering unit 104 , the mapping unit 106 , the memory unit 110 and the user input unit 112 .
  • the light registering camera unit 104 and the memory unit 110 are connected to the mapping unit 106 .
  • the image information providing unit 102 is further connected to the memory unit 110 .
  • aspects consistent with the invention use image information that is registered by a light registering camera unit, compare the registered image information with the image information that is being presented, and based on recognized image elements or image objects, to position a predetermined part of the camera image in the image information being presented.
  • the user of the portable mobile phone being one example of the communication device 100 , can utilize the position of the predetermined part of the camera image relative to the displayed image elements in the image information being presented, to control the image information that is being presented.
  • FIG. 2 presenting a flow chart of a method according to one embodiment of the present invention, the method will be described in more detail.
  • This exemplary method starts with sending video or image information for presentation (step 202 ). This step is performed by the image information providing unit 102 under control of the control unit 108 .
  • Video information is herein understood as comprising image information that may be registered by a camera unit.
  • the portable communication device 100 Prior to sending the video information, the portable communication device 100 is connected to an image information presentation unit, such a TV-screen, a TV-set, PC-screen, a video projector or the like, to be used for the presentation of the first image information.
  • an image information presentation unit such as a TV-screen, a TV-set, PC-screen, a video projector or the like, to be used for the presentation of the first image information.
  • the connection between the mobile phone, being one example of the portable communication device 100 , and the image information presentation unit is performed using, for example, a universal serial bus (USB) connector, according to one implementation of the present invention.
  • USB universal serial bus
  • connection may be established using Bluetooth technique using a first Bluetooth module either built-in in the mobile phone or associated with it, and a second Bluetooth module connected to or associated with the image information presentation unit.
  • the mobile phone may send video information, as one example of first image information, to the image information presentation unit.
  • the method of providing positioning information may also include directing the light registering camera unit 104 of the mobile phone 100 towards the image information presentation unit, under control of the user of the mobile phone 100 , and obtaining video information (step 204 ). That is, the user of mobile phone 100 may use the camera function of mobile phone 100 (i.e., light registering camera unit 104 ) as a pointer and point the mobile phone 100 in the direction of the image information presentation unit. This step is included since the light registering camera unit 104 , within the method of the present invention, is arranged to register light from the image information presentation unit.
  • the light registering camera unit 104 may detect a change of the direction of the camera included in mobile phone 100 with respect to the image information presentation unit. This change of direction may be caused by the user of the mobile phone 100 , such as would occur by the user moving mobile phone 100 , or by other influences.
  • the camera image may then be registered (step 206 ). That is, the light registering camera unit 104 may register image information, such as image information associated with the video information being presented on the image information presentation unit, as an example of image information that is being presented. The light that is registered by the light registering camera unit 104 may then form the second image information, that will be used to compare to the first image information, as described in more detail below.
  • the step of registering is performed by the light registering camera unit 104 under the control of the control unit 108 .
  • the subsequent step is mapping presented first image information and registered second image information (step 208 ).
  • mapping unit 106 may be performed by mapping unit 106 , being provided with the second image information from the light registering camera unit 104 and first image information from the memory unit 110 , under control of the control unit 108 .
  • mapping of image or picture information is well known to the skilled person, this mapping step will not be discussed in full detail to not unduly obscure the thrust of the invention. However, it is to be understood that a mapping may include relative size adjustments, rotations and translations of the two kinds of image information.
  • projection of the first image information at an angle may be used in the step of mapping when mapping the first image information with the registered second image information, since the second image information may have been registered at an angle relative to the presentation of the first image information using the image information presentation unit.
  • the step of mapping is performed taking into account shaking of the mobile phone 100 , including the light registering camera unit 104 , caused by unsteady user operation when directing the light registering camera unit 104 towards the image information presentation unit.
  • a next step of the method of providing positioning information is recognizing at least one image element of the first image information as being included in the registered second image information (step 210 ). For example, suppose that an image of a person is included in the first image information. In this case, this step may include recognizing the image of a person in the second image information. This step of recognizing may be performed by the control unit 108 being provided with mapping information by the mapping unit 106 .
  • a complete image object may be recognized by the control unit 108 in step 210 , as being present in both first and second image information.
  • the relative position of the camera image center with respect to the recognized person may then be determined (act 212 ), as one example of determining the relative position of a predefined area of the registered second image information in the first image information, with respect to a recognized image element. Determining the relative position of the camera image center is a step that may be performed by the control unit 108 , according to one embodiment of the present invention.
  • mapping first image information such as the presented video information and the second image information, such as the registered second image information, as well as the step of recognizing image elements as present in both these image information, may be performed in various ways. Mapping may be made at various time intervals following different criteria. Such variations would be known to one ordinary skill in the art and will therefore not be discussed hereinafter.
  • image information providing unit 102 may provide the camera image center position to an image presentation unit on which the center position is displayed together with the video information, under the control of the control unit 108 .
  • displaying the camera image position with the video information is followed by determining whether the obtained camera image (i.e., the second information obtained at step 204 ) is adequate (step 216 ).
  • control unit 108 with assistance of the mapping unit 106 may determine whether any substantial direction change of the mobile phone 100 occurred based on, for example, the degree of recognition of image elements and objects in step 210 .
  • displaying the camera image position with the video information may be followed by determining whether a change of direction is detected by the camera under the control of the control unit 108 .
  • processing may return to step 204 . That is, new video information (e.g., a camera image) from mobile phone 100 may be obtained, under the influence of the user of the mobile phone 100 .
  • new video information e.g., a camera image
  • the subsequent step detects whether user interaction has occurred (step 218 ).
  • User interaction may be exemplified by the removal of an element or the graphical disintegration of an image object, or the displacement of an object, to mention a few examples only. Such interactions may be performed by use of an application program, such as a video game (e.g., a shooting game), being executed by mobile phone 100 .
  • a video game e.g., a shooting game
  • processing may return to step 216 , according to this embodiment of the present invention.
  • the method may end (step 220 ).
  • the relative position of a predetermined area of the second image information on the first image information may be used for obtaining direction orders in aviation games, wherein the position of the camera center (e.g., a cursor) relative to the position of the object may be used to alter the altitude of the object in the aviation game, make the object perform turns, prepare for touch down and landing, etc., to mention a few possibilities only.
  • the position of the camera center e.g., a cursor
  • the camera center position may be associated with one or more recognized objects in order to change the speed, direction or conditions for the objects.
  • One example would be to associate the camera center with a car or becoming your own car, such as in a racing game, after which any position of the camera center relative to the associated car may be used for steering and speed regulating purposes, etc.
  • Many more variations along similar lines may also be provided.
  • An external image information presentation unit 302 in the form of a screen is being used for presenting the first image information, here in the form of an adult 308 to the right in the presentation and a child 306 , to the adult's right positioned close to the center of the presentation.
  • Portable communication device 304 also referred to herein as mobile phone 304 , may include the same or similar elements as portable communication device 100 described above with respect to FIG. 1 .
  • the camera registers the second image information, illustrated as image 310 , containing the adult 314 to the right and the child 312 to the adult's right positioned close to the center of the presentation.
  • mapping unit 106 the step of mapping of the first and second image information is performed, as described above by mapping unit 106 , in FIG. 1 .
  • the camera center position is determined and information is provided to the image presentation unit to display the camera center position.
  • the first image presentation as displayed on screen 316 now comprises the camera center, denoted by the star 322 .
  • the position of the star 322 thus corresponds to the camera image center.
  • This image center 322 on the screen 316 is positioned to the left of the child 318 and even further away from the adult 320 .
  • a new image associated with the movement of the mobile phone 100 may be obtained (step 204 ).
  • the second image information of the screen as viewed by the camera is denoted by 332 , which shows the adult 336 close to the center of the camera picture and the child 334 to the left in the camera picture, being one example of the second image information.
  • the view as denoted by 338 is provided as the first image information.
  • the adult 342 and the child 340 have not changed their position in the first image information, but a cursor indicating the camera center is now displayed as the star 344 at the position of the adult 342 . This is due to the directing of the camera towards the position of the adult 330 in the first image information of the screen 324 .
  • Screen 346 denotes the screen after the adult image has been wiped out or removed after the receipt of user input via mobile phone 304 .
  • the adult may be removed via an input associated with a user playing a video game executed by mobile phone 304 .
  • the child 348 is still present and so is the star 350 denoting the camera center.
  • the portable communication device 304 may be used as a pointing device and a symbol, such as a star in this example, may be used to indicate the center position at which the camera is pointed.
  • the portable communication device 304 acts as a pointing device associated with images on an external image presentation unit, such as the displays illustrated in screens 316 , 338 and 346 .
  • the symbol e.g., the star
  • the user of portable communication device 304 may perform an action, such as remove the image, during execution of a video game being played by the user of portable communication device 304 .
  • Described above is one example of how positioning information may be used in a simple way for controlling the video information being presented by an external image information presentation unit.
  • a mobile phone may be used as a pointer of the game.
  • This pointer may be displayed as a gun or shooting device, allowing the user to control the virtual gun to shoot in the direction at which the camera of the mobile phone is pointed.
  • the pointer may also be used to move images within the display.
  • buttons of the mobile phone are associated with special functions. Buttons may be used to activate certain actions, for instance to set a password simply by entering a certain digit while directing the camera to a specific part of the monitor/TV set.
  • the combination of directing the mobile phone to a particular part of a monitor or display and entering a specific key may thus form a unique pattern that can be used as a secret key or for general security purposes.
  • FIG. 4 illustrates a portable communication device 40 in the form of a mobile phone, according to one embodiment of the present invention.
  • Portable communication device 40 may include the same or similar elements as portable communication device 100 described above with respect to FIG. 1 .
  • FIG. 5 schematically depicts a computer program product 50 according to one embodiment of the invention, having thereon computer program code means.
  • the computer program code means comprised on the computer program product, is loaded in a computer or portable communication device
  • said computer or portable communication device executes providing of first image information from the portable communication device of the computer, to be presented by an image information presentation unit, registering second image information by the camera of the portable communication device under control of the user of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the second registered image information, and determining the position of a predefined area of the second registered image information in relation to the recognized at least one image element, in order to enable using the determined relative position of the predefined area for controlling purposes of the first image information being presented.
  • the computer program product may be provided as a CD-ROM disc according to one embodiment of the invention.
  • the computer program product 50 can alternatively be provided as another type of disc such as a digital video disc (DVD), a hard disc, or be provided as a memory or other storage capacity, such as a flash-based memory, for example a memory stick or a USB (Universal Serial Bus) memory or even a memory of the type being volatile.
  • DVD digital video disc
  • hard disc or be provided as a memory or other storage capacity, such as a flash-based memory, for example a memory stick or a USB (Universal Serial Bus) memory or even a memory of the type being volatile.
  • USB Universal Serial Bus
  • the steps of the method as described above may be executed in a different order, without deviating from the scope of the invention.
  • some steps may be formed concurrently.
  • some steps of the method may be omitted, following yet a different embodiment of the invention.
  • the number of steps may moreover be changed, by for instance incorporating a few steps in others or dividing the function of certain steps in other steps, such that novel steps are created but without deviating from the overall function of the steps.
  • the units as comprised in the portable communication device, as shown in FIG. 1 may at least partly be comprised in other performing units, without deviating from the invention.
  • Other units may co-exist according to other embodiments of the invention.
  • aspects of the invention provide, among other advantages, positioning of an image center relative to recognized surrounding image elements, which enables, for example, more advanced gaming using mobile phones in connection with external image presentation units.
  • the invention may also provide a flexible positioning tool, which also can be used for controlling objects based on relative positioning of cursor movements.

Abstract

A method for providing positioning information related to image information presentation for a portable communication device, includes providing first image information to be presented (202), obtaining the first image information by the portable communication device under control of the user of the portable communication device, registering second image information by the camera of the portable communication device (206) and mapping presented first image information and registered second image information (208). The method may also include recognizing at least one image element of the presented first image information as being included in the registered second image information (210) and determining the position of a predefined area of the registered second image information in relation to the recognized at least one image element 212). The determined relative position of the predefined area may be used for controlling purposes of the first image information being presented.

Description

    RELATED APPLICATIONS
  • This application claims priority under 35 U.S.C. § 119 based on U.S. Provisional Application Ser. No. 60/749,621, filed Dec. 13, 2005, the disclosure of which is incorporated herein by reference.
  • TECHNICAL FIELD OF THE INVENTION
  • The invention relates in general to controlling presentation of image information, and in particular to providing positioning information related to the presentation of image information.
  • DESCRIPTION OF RELATED ART
  • Electronic games within portable electronic devices have recently developed into advanced and complex games requiring large memory space. Portable communication devices, such as mobile phones, may be found having memories of 0.5 Gb or more.
  • Whereas the games have increased in complexity, the size of the screen of the mobile phones, being one example of portable communication devices, has not had the same development. Screens are still relatively small, which sets limitations to the user experience when gaming.
  • Gaming using large screens is well known and was performed in the early days of electronic games, such as in television games using TV-sets as screens or in personal computer games using the computer cathode ray tubes as screens.
  • Interactions between the user of the game and the displayed gaming information were allowed in television target aiming and shooting games. The television games typically provided a photo sensitive gun capable of sensing large bright target symbols on the screen of a television receiver. The targets were distinctly displayed on the screen when imaged using a gun-mounted electro-optical sensor resulting in a series of output pulses indicating that the gun was indeed lined up properly with the target and a hit was scored. None of these attempted to show where the user was aiming when he pulled the trigger preceding a miss or for that matter, how closely to the center of the target on screen he was aiming when he made a hit.
  • In U.S. Pat. No. 4,395,045, a television precision target shooting apparatus and method simulate shooting at displayed images on the screen of a television set using a simulated gun that provides indication on the screen of where the gun was pointed at the time of trigger pull. This is achieved by flashing the screen white subsequent a trigger pull in order for a photo sensor within the gun to respond to short segments of horizontal raster lines such that by counting vertical lines and time intervals with respect to vertical and horizontal synchronization pulses, respectively, identification of where on the screen the gun was pointing is achieved. This permits a symbol to be displayed upon a screen at that location.
  • These techniques to identify a flashing item by using a photo optical sensor may, however, not be used for a plurality of identical targets, since each individual position is associated with unique position descriptors, such as vertical lines and time intervals.
  • There are also other techniques with which a user could directly interact with an image presentation. By using a directing or pointing device, the direction in which such a device is directed or pointed can be determined.
  • In one technique, a gyro is used which is built in the directing device and which reacts to lateral and longitudinal motions of the directing device. Any lateral and longitudinal motions causing lateral and longitudinal displacements are related to a reference position, by which an updated position can be calculated. The application of such a directing device in gaming may allow the position of a cursor or symbol to be determined and displayed on a screen.
  • An alternative to using a gyro to determine an updated position of a cursor or symbol on a screen is to use an accelerometer included in the directing or pointing device. Any motions of the directing or pointing device are detected by the accelerometer, and then converted to a displacement of the location where the directing or pointing device is pointing. A cursor may therefore also be displayed on the screen by using this technique.
  • These two directing techniques are based on motions relative to a reference direction, for which reason these techniques become inapplicable for determining a plurality of positions of targets on a screen.
  • There is hence a need to overcome the limitations of prior art techniques and to provide a technique that is more flexible and applicable to the positioning of a plurality of targets.
  • SUMMARY OF THE INVENTION
  • Aspects of the invention are directed towards solving the problem of interactively controlling an image presentation as displayed on a large screen, using a portable communication device.
  • Aspects consistent with the invention provide a method for comparing image information being registered with image information being presented using a light registering camera unit of the portable communication device.
  • According to one aspect of the invention, a method of providing positioning information related to a presentation of image information for a portable communication device, comprises providing first image information to be presented by an image information presentation unit, obtaining the first image information by the portable communication device under control of the user of the portable communication device, registering second image information by the camera of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the registered second image information, and determining the position of a predefined area of the registered second image information in relation to the recognized at least one image element, where the determined relative position of the predefined area is used for controlling purposes of the first image information being presented.
  • A second aspect of the invention is directed towards a method including the features of the first aspect, further comprises the step of sending predefined area position information for displaying the predefined area position on the image information presentation unit.
  • A third aspect of the invention is directed towards a method including the features of the first aspect, wherein the step of determining the position of the predefined area comprises determining the position of the center area of second registered image information.
  • A fourth aspect of the invention is directed towards a method including the features of the first aspect, further comprising obtaining updated directing of the camera of the portable communication device under the control of the user of the portable communication device.
  • A fifth aspect of the invention is directed towards a method including the features of the first aspect, further comprising the step of receiving user input by the portable communication device, which user input is used as input data for controlling the first image information being presented.
  • A sixth aspect of the invention is directed towards a method including the features of the fourth aspect, further comprising the step of controlling the first image information being presented based on at least one of the obtained updated directing or received user input.
  • Other aspects consistent with the invention provide a device for comparing image information being registered with image information being presented, using a light registering camera unit of the portable communication device.
  • According to a seventh aspect of the invention, a portable communication device for providing positioning information related to a presentation of a first image information, comprises an image information providing unit, arranged to provide a first image information to an image information presentation unit, a light registering camera unit, arranged to register second image information when being directed towards the image presentation unit, a mapping unit, connected to the light registering camera unit and arranged to map the first image information and the second image information, and a control unit arranged to control providing of the first image information to an image information unit, to control registering second image information by the light registering camera unit, and to control mapping of the first image information and the second image information, and further arranged to recognize at least one image element of the presented first image information as being included in the second image information, to determine the position of a predefined area of the second registered image information in relation to the recognized at least one image element and use the determined relative position of the predefined area to control the first image information being presented.
  • An eighth aspect of the invention is directed toward a portable communication device including the features of the seventh aspect and wherein the control unit is further configured to use the determined relative position of the predefined area to control the first image information being presented.
  • A ninth aspect of the invention is directed towards a portable communication device including the features of the seventh aspect, wherein the portable communication device comprises a mobile phone.
  • Still other aspects of the invention provide a computer program product comprising a computer readable medium, having thereon computer program code means, to make a portable communication device or a computer execute, comparing image information being registered with image information being presented, using a light registering camera unit of the portable communication device.
  • According to a tenth aspect of the invention, a computer program product comprising a computer readable medium, having thereon computer program code for execution by a portable communication device or a computer, when said computer program code is loaded in the portable communication device or the computer, to cause the portable communication device or computer perform a method comprising: providing of first image information from the portable communication device or the computer, to be presented by an image information presentation unit, registering second image information by the camera of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the second registered image information, and determining the position of a predefined area of the second registered image information in relation to the recognized at least one image element.
  • An eleventh aspect of the invention is directed to a computer program product including the features of the tenth aspect, where the method performed by the portable communication device or computer further comprises using the determined relative position of the predefined area for controlling the first image information being presented.
  • Aspects of the invention provide, among other advantages, positioning of an image center relative to recognized surrounding image elements, which enables more advanced gaming using mobile phones in connection with external image presentation units.
  • The invention may also provide a flexible positioning tool, which also can be used for controlling objects based on relative positioning of cursor movements.
  • It should be emphasized that the term “comprises/comprising” when used in this specification is taken to specify the presence of stated features, integers, steps or components, but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention will now be described in more detail in relation to the enclosed drawings, in which:
  • FIG. 1 schematically illustrates an exemplary portable communication device consistent with the invention;
  • FIG. 2 is a flow chart illustrating processing according to one embodiment of the invention;
  • FIG. 3 schematically presents one simple example of first and second image information according to the invention;
  • FIG. 4 illustrates a portable communication device in the form of a cellular phone, according to one embodiment of the invention; and
  • FIG. 5 illustrates a computer program product having stored thereon computer program code to make a portable communication device or a computer execute steps of a method according to one embodiment of the invention.
  • DETAILED DESCRIPTION OF THE EMBODIMENTS
  • Aspects consistent with the invention are directed towards solving the problem of interactively controlling an image presentation as displayed on a large screen, using a portable communication device.
  • With reference to FIG. 1, schematically showing a portable communication device 100, according to one embodiment of the present invention, the invention is now explained in more detail.
  • The portable communication device 100 comprises an image information providing unit 102, a light registering camera unit 104, a mapping unit 106, a control unit 108, a memory unit 110, and a user input unit 112. As is shown in FIG. 1, the control unit 108 is connected to all other units. That is, control unit 108 is connected to the image information providing unit 102, the light registering unit 104, the mapping unit 106, the memory unit 110 and the user input unit 112. Moreover, the light registering camera unit 104 and the memory unit 110 are connected to the mapping unit 106. The image information providing unit 102 is further connected to the memory unit 110.
  • Aspects consistent with the invention use image information that is registered by a light registering camera unit, compare the registered image information with the image information that is being presented, and based on recognized image elements or image objects, to position a predetermined part of the camera image in the image information being presented. By displaying this position, the user of the portable mobile phone, being one example of the communication device 100, can utilize the position of the predetermined part of the camera image relative to the displayed image elements in the image information being presented, to control the image information that is being presented.
  • By referring to FIG. 2 presenting a flow chart of a method according to one embodiment of the present invention, the method will be described in more detail.
  • This exemplary method starts with sending video or image information for presentation (step 202). This step is performed by the image information providing unit 102 under control of the control unit 108. Video information is herein understood as comprising image information that may be registered by a camera unit.
  • Prior to sending the video information, the portable communication device 100 is connected to an image information presentation unit, such a TV-screen, a TV-set, PC-screen, a video projector or the like, to be used for the presentation of the first image information.
  • The connection between the mobile phone, being one example of the portable communication device 100, and the image information presentation unit is performed using, for example, a universal serial bus (USB) connector, according to one implementation of the present invention.
  • According to one alternative of the present invention, the connection may be established using Bluetooth technique using a first Bluetooth module either built-in in the mobile phone or associated with it, and a second Bluetooth module connected to or associated with the image information presentation unit.
  • The mobile phone may send video information, as one example of first image information, to the image information presentation unit.
  • The method of providing positioning information may also include directing the light registering camera unit 104 of the mobile phone 100 towards the image information presentation unit, under control of the user of the mobile phone 100, and obtaining video information (step 204). That is, the user of mobile phone 100 may use the camera function of mobile phone 100 (i.e., light registering camera unit 104) as a pointer and point the mobile phone 100 in the direction of the image information presentation unit. This step is included since the light registering camera unit 104, within the method of the present invention, is arranged to register light from the image information presentation unit.
  • According to an alternative embodiment of the invention, the light registering camera unit 104 may detect a change of the direction of the camera included in mobile phone 100 with respect to the image information presentation unit. This change of direction may be caused by the user of the mobile phone 100, such as would occur by the user moving mobile phone 100, or by other influences.
  • The camera image may then be registered (step 206). That is, the light registering camera unit 104 may register image information, such as image information associated with the video information being presented on the image information presentation unit, as an example of image information that is being presented. The light that is registered by the light registering camera unit 104 may then form the second image information, that will be used to compare to the first image information, as described in more detail below. The step of registering is performed by the light registering camera unit 104 under the control of the control unit 108.
  • Having registered the camera image as the second image information by the light registering camera unit 104 and having access to the video information (as one example of the first image information), the subsequent step is mapping presented first image information and registered second image information (step 208).
  • This mapping may be performed by mapping unit 106, being provided with the second image information from the light registering camera unit 104 and first image information from the memory unit 110, under control of the control unit 108.
  • Since mapping of image or picture information is well known to the skilled person, this mapping step will not be discussed in full detail to not unduly obscure the thrust of the invention. However, it is to be understood that a mapping may include relative size adjustments, rotations and translations of the two kinds of image information.
  • In an alternative embodiment of the present invention, projection of the first image information at an angle may be used in the step of mapping when mapping the first image information with the registered second image information, since the second image information may have been registered at an angle relative to the presentation of the first image information using the image information presentation unit.
  • According to another embodiment the step of mapping is performed taking into account shaking of the mobile phone 100, including the light registering camera unit 104, caused by unsteady user operation when directing the light registering camera unit 104 towards the image information presentation unit.
  • Having mapped the first image information and the second image information, a next step of the method of providing positioning information is recognizing at least one image element of the first image information as being included in the registered second image information (step 210). For example, suppose that an image of a person is included in the first image information. In this case, this step may include recognizing the image of a person in the second image information. This step of recognizing may be performed by the control unit 108 being provided with mapping information by the mapping unit 106.
  • According to another embodiment of the present invention, a complete image object may be recognized by the control unit 108 in step 210, as being present in both first and second image information.
  • The relative position of the camera image center with respect to the recognized person may then be determined (act 212), as one example of determining the relative position of a predefined area of the registered second image information in the first image information, with respect to a recognized image element. Determining the relative position of the camera image center is a step that may be performed by the control unit 108, according to one embodiment of the present invention.
  • It is noted that image information and especially motion pictures, that is video information, may be formatted in various different ways. For this reason the steps of mapping first image information, such as the presented video information and the second image information, such as the registered second image information, as well as the step of recognizing image elements as present in both these image information, may be performed in various ways. Mapping may be made at various time intervals following different criteria. Such variations would be known to one ordinary skill in the art and will therefore not be discussed hereinafter.
  • Next, a symbol, such as a cursor may be displayed on the presented video information at the camera image center position (step 214). In one implementation, image information providing unit 102 may provide the camera image center position to an image presentation unit on which the center position is displayed together with the video information, under the control of the control unit 108.
  • In one embodiment of the present invention, displaying the camera image position with the video information is followed by determining whether the obtained camera image (i.e., the second information obtained at step 204) is adequate (step 216). For example, in one implementation, control unit 108 with assistance of the mapping unit 106 may determine whether any substantial direction change of the mobile phone 100 occurred based on, for example, the degree of recognition of image elements and objects in step 210.
  • Various other criteria for determining whether the obtained camera image is adequate (e.g., whether a substantial change in direction of the mobile phone 100 has occurred) may be used in alternative embodiments of the invention.
  • For example, according to one embodiment of the present invention, displaying the camera image position with the video information (i.e., step 214) may be followed by determining whether a change of direction is detected by the camera under the control of the control unit 108.
  • If it is determined in step 216 that the obtained camera image is not adequate, or according to the alternative embodiment that a change of direction has been detected, processing may return to step 204. That is, new video information (e.g., a camera image) from mobile phone 100 may be obtained, under the influence of the user of the mobile phone 100.
  • If, however, it is determined in step 216 that the image is adequate or according to an alternative embodiment that a change of direction has not been detected, the subsequent step detects whether user interaction has occurred (step 218). User interaction may be exemplified by the removal of an element or the graphical disintegration of an image object, or the displacement of an object, to mention a few examples only. Such interactions may be performed by use of an application program, such as a video game (e.g., a shooting game), being executed by mobile phone 100.
  • If it is determined in step 218 that user interaction is received, processing may return to step 216, according to this embodiment of the present invention.
  • If, on the other hand, it is determined that no user interaction is received in step 218, the method may end (step 220).
  • According to another embodiment of the invention, the relative position of a predetermined area of the second image information on the first image information, such as, for example, the camera image center of the video presentation may be used for obtaining direction orders in aviation games, wherein the position of the camera center (e.g., a cursor) relative to the position of the object may be used to alter the altitude of the object in the aviation game, make the object perform turns, prepare for touch down and landing, etc., to mention a few possibilities only.
  • According to yet another embodiment of the invention, the camera center position may be associated with one or more recognized objects in order to change the speed, direction or conditions for the objects. One example would be to associate the camera center with a car or becoming your own car, such as in a racing game, after which any position of the camera center relative to the associated car may be used for steering and speed regulating purposes, etc. Many more variations along similar lines may also be provided.
  • With reference to FIG. 3, an example of using the invention for controlling purposes of an electronic game is schematically illustrated. An external image information presentation unit 302 in the form of a screen is being used for presenting the first image information, here in the form of an adult 308 to the right in the presentation and a child 306, to the adult's right positioned close to the center of the presentation.
  • A portable communication device 304 containing a light registering camera unit, illustrated as box 305 in FIG. 3, is directed towards the screen of image information presentation unit 302. Portable communication device 304, also referred to herein as mobile phone 304, may include the same or similar elements as portable communication device 100 described above with respect to FIG. 1. By registering incident light, the camera registers the second image information, illustrated as image 310, containing the adult 314 to the right and the child 312 to the adult's right positioned close to the center of the presentation.
  • Having obtained first and second image information, the step of mapping of the first and second image information is performed, as described above by mapping unit 106, in FIG. 1. After having recognized both the adult and the child as being included in both images (i.e., images displayed by image information presentation unit 302 and image 310), the camera center position is determined and information is provided to the image presentation unit to display the camera center position.
  • As a result, the first image presentation as displayed on screen 316 now comprises the camera center, denoted by the star 322. The position of the star 322 thus corresponds to the camera image center. This image center 322 on the screen 316 is positioned to the left of the child 318 and even further away from the adult 320.
  • After having determined that the camera image information was not adequate or that the camera was moved in step 216, a new image associated with the movement of the mobile phone 100 may be obtained (step 204).
  • This is schematically depicted by the mobile phone 304 being directed in a new direction towards screen 324, as compared to the previous exemplary picture. That is, mobile phone 304 has been pointed in a new direction. As result, the adult 330 appears to be located close to the center of the camera view and the child 328 is located to the left in the camera view.
  • Consequently, the second image information of the screen as viewed by the camera is denoted by 332, which shows the adult 336 close to the center of the camera picture and the child 334 to the left in the camera picture, being one example of the second image information.
  • By performing the steps of mapping, recognizing and displaying the camera center position on the first image presentation, the view as denoted by 338 is provided as the first image information. As expected, the adult 342 and the child 340 have not changed their position in the first image information, but a cursor indicating the camera center is now displayed as the star 344 at the position of the adult 342. This is due to the directing of the camera towards the position of the adult 330 in the first image information of the screen 324.
  • Screen 346 denotes the screen after the adult image has been wiped out or removed after the receipt of user input via mobile phone 304. For example, the adult may be removed via an input associated with a user playing a video game executed by mobile phone 304. The child 348 is still present and so is the star 350 denoting the camera center.
  • Therefore, in this example, the portable communication device 304 may be used as a pointing device and a symbol, such as a star in this example, may be used to indicate the center position at which the camera is pointed. In this manner, the portable communication device 304 acts as a pointing device associated with images on an external image presentation unit, such as the displays illustrated in screens 316, 338 and 346. When the symbol (e.g., the star) is located over the desired image, the user of portable communication device 304 may perform an action, such as remove the image, during execution of a video game being played by the user of portable communication device 304.
  • Described above is one example of how positioning information may be used in a simple way for controlling the video information being presented by an external image information presentation unit.
  • In another example, other interactions between the user of a mobile phone and a game during gaming applications are enabled according to the invention. For example, during gaming sessions using large displays, such as TV sets, monitors, etc., a mobile phone may be used as a pointer of the game. This pointer may be displayed as a gun or shooting device, allowing the user to control the virtual gun to shoot in the direction at which the camera of the mobile phone is pointed. The pointer may also be used to move images within the display.
  • In yet another example of the application of the invention, a combination of the camera as a pointing and registering tool, and other user interfaces is provided. For example, when pointing the camera at specific positions on the screen, certain buttons of the mobile phone are associated with special functions. Buttons may be used to activate certain actions, for instance to set a password simply by entering a certain digit while directing the camera to a specific part of the monitor/TV set. The combination of directing the mobile phone to a particular part of a monitor or display and entering a specific key may thus form a unique pattern that can be used as a secret key or for general security purposes.
  • FIG. 4 illustrates a portable communication device 40 in the form of a mobile phone, according to one embodiment of the present invention. Portable communication device 40 may include the same or similar elements as portable communication device 100 described above with respect to FIG. 1.
  • In addition, FIG. 5 schematically depicts a computer program product 50 according to one embodiment of the invention, having thereon computer program code means. When the computer program code means, comprised on the computer program product, is loaded in a computer or portable communication device, said computer or portable communication device, executes providing of first image information from the portable communication device of the computer, to be presented by an image information presentation unit, registering second image information by the camera of the portable communication device under control of the user of the portable communication device, mapping presented first image information and registered second image information, recognizing at least one image element of the presented first image information as being included in the second registered image information, and determining the position of a predefined area of the second registered image information in relation to the recognized at least one image element, in order to enable using the determined relative position of the predefined area for controlling purposes of the first image information being presented.
  • The computer program product may be provided as a CD-ROM disc according to one embodiment of the invention. However, the computer program product 50 can alternatively be provided as another type of disc such as a digital video disc (DVD), a hard disc, or be provided as a memory or other storage capacity, such as a flash-based memory, for example a memory stick or a USB (Universal Serial Bus) memory or even a memory of the type being volatile.
  • It is emphasized that this invention can be varied in many ways, of which the embodiments discussed above only are examples of a few. These different embodiments are hence non-limiting examples. The scope of the invention, however, is only limited by the subsequently following claims.
  • According to one alternative embodiment of the invention, the steps of the method as described above may be executed in a different order, without deviating from the scope of the invention. In addition, some steps may be formed concurrently. Still further, some steps of the method may be omitted, following yet a different embodiment of the invention.
  • The number of steps may moreover be changed, by for instance incorporating a few steps in others or dividing the function of certain steps in other steps, such that novel steps are created but without deviating from the overall function of the steps.
  • The units as comprised in the portable communication device, as shown in FIG. 1, may at least partly be comprised in other performing units, without deviating from the invention. Other units may co-exist according to other embodiments of the invention.
  • Aspects of the invention, as described above, provide, among other advantages, positioning of an image center relative to recognized surrounding image elements, which enables, for example, more advanced gaming using mobile phones in connection with external image presentation units.
  • The invention may also provide a flexible positioning tool, which also can be used for controlling objects based on relative positioning of cursor movements.
  • The scope of the invention is defined by the claims and their equivalents.

Claims (12)

1. A method of providing positioning information related to a presentation of image information for a portable communication device, comprising:
providing first image information to be presented by an image information presentation unit;
obtaining the first image information by the portable communication device under control of the user of the portable communication device;
registering second image information by the camera of the portable communication device;
mapping presented first image information and registered second image information;
recognizing at least one image element of the presented first image information as being included in the registered second image information; and
determining the position of a predefined area of the registered second image information in relation to the recognized at least one image element,
wherein the determined relative position of the predefined area is used for controlling purposes of the first image information being presented.
2. The method of claim 1, further comprising:
sending predefined area position information for displaying the predefined area position on the image information presentation unit.
3. The method of claim 1, wherein the step of determining the position of the predefined area comprises determining the position of the center area of the registered second image information.
4. The method of claim 1, further comprising:
obtaining updated directing of the camera of the portable communication device under the control of the user of the portable communication device.
5. The method of claim 1, further comprising:
receiving user input by the portable communication device, which user input is used as input data for controlling the first image information being presented.
6. The method of claim 4, further comprising:
controlling the first image information being presented based on at least one of the obtained updated directing or received user input.
7. The method of claim 1, further comprising:
using the determined relative position of the predefined area for controlling the first image information being presented.
8. A portable communication device for providing positioning information related to a presentation of a first image information, comprising:
an image information providing unit, arranged to provide a first image information to an image information presentation unit;
a light registering camera unit, arranged to register second image information when being directed towards the image information presentation unit;
a mapping unit, connected to the light registering camera unit and arranged to map the first image information and the second image information; and
a control unit arranged to control providing of the first image information to an image information unit, to control registering second image information by the light registering camera unit, and to control mapping of the first image information and the second image information, and further arranged to recognize at least one image element of the first image information as being included in the second image information, to determine the position of a predefined area of the second image information in relation to the recognized at least one image element, and use the determined relative position of the predefined area to control the first image information being presented.
9. The portable communication device of claim 8, wherein the control unit is further configured to use the determined relative position of the predefined area to control the first image information being presented.
10. The portable communication device of claim 8, wherein the portable communication device comprises a mobile phone.
11. A computer program product comprising a computer readable medium, having stored thereon computer program code for execution by a portable communication device or a computer, when said computer program code is loaded in the portable communication device or the computer, to cause the portable communication device or computer to perform a method comprising:
providing of first image information from the portable communication device or the computer, to be presented by an image information presentation unit;
registering second image information by the camera of the portable communication device;
mapping presented first image information and registered second image information;
recognizing at least one image element of the presented first image information as being included in second registered image information; and
determining the position of a predefined area of the second registered image information in relation to the recognized at least one image element.
12. The computer program product of claim 11, wherein the method further comprises:
using the determined relative position of the predefined area for controlling the first image information being presented.
US11/438,088 2005-12-13 2006-05-22 Image element identifier Expired - Fee Related US7513700B2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US11/438,088 US7513700B2 (en) 2005-12-13 2006-05-22 Image element identifier
PCT/EP2006/066187 WO2007068511A1 (en) 2005-12-13 2006-09-08 Image element identifier
EP06793370A EP1960070A1 (en) 2005-12-13 2006-09-08 Image element identifier
JP2008544901A JP2009519066A (en) 2005-12-13 2006-09-08 Image element identifier
KR1020087016968A KR20080080631A (en) 2005-12-13 2006-09-08 Image element identifier

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US74962105P 2005-12-13 2005-12-13
US11/438,088 US7513700B2 (en) 2005-12-13 2006-05-22 Image element identifier

Publications (2)

Publication Number Publication Date
US20070147828A1 true US20070147828A1 (en) 2007-06-28
US7513700B2 US7513700B2 (en) 2009-04-07

Family

ID=37116087

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/438,088 Expired - Fee Related US7513700B2 (en) 2005-12-13 2006-05-22 Image element identifier

Country Status (5)

Country Link
US (1) US7513700B2 (en)
EP (1) EP1960070A1 (en)
JP (1) JP2009519066A (en)
KR (1) KR20080080631A (en)
WO (1) WO2007068511A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130169820A1 (en) * 2011-03-15 2013-07-04 David Alexander Stewart Camera device to capture and generate target lead and shooting technique data and images
US20140078319A1 (en) * 2007-02-22 2014-03-20 Panasonic Corporation Imaging apparatus
US9267761B2 (en) * 2011-03-15 2016-02-23 David A. Stewart Video camera gun barrel mounting and programming system

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8217997B2 (en) * 2010-03-16 2012-07-10 Interphase Corporation Interactive display system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4395045A (en) * 1980-06-16 1983-07-26 Sanders Associates, Inc. Television precision target shooting apparatus and method
US5138304A (en) * 1990-08-02 1992-08-11 Hewlett-Packard Company Projected image light pen
US20050107160A1 (en) * 2003-11-14 2005-05-19 Zeroplus Technology Co., Ltd. [photographic pointer positioning system and its operation process]

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1175345C (en) 1999-03-31 2004-11-10 皇家菲利浦电子有限公司 Remote control for display apparatus
US6287200B1 (en) 1999-12-15 2001-09-11 Nokia Corporation Relative positioning and virtual objects for mobile devices
DK1573498T3 (en) 2002-11-20 2012-03-19 Koninkl Philips Electronics Nv User interface system based on a pointing device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4395045A (en) * 1980-06-16 1983-07-26 Sanders Associates, Inc. Television precision target shooting apparatus and method
US5138304A (en) * 1990-08-02 1992-08-11 Hewlett-Packard Company Projected image light pen
US20050107160A1 (en) * 2003-11-14 2005-05-19 Zeroplus Technology Co., Ltd. [photographic pointer positioning system and its operation process]

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140078319A1 (en) * 2007-02-22 2014-03-20 Panasonic Corporation Imaging apparatus
US20130169820A1 (en) * 2011-03-15 2013-07-04 David Alexander Stewart Camera device to capture and generate target lead and shooting technique data and images
US8908045B2 (en) * 2011-03-15 2014-12-09 David Alexander Stewart Camera device to capture and generate target lead and shooting technique data and images
US9267761B2 (en) * 2011-03-15 2016-02-23 David A. Stewart Video camera gun barrel mounting and programming system
US9546846B2 (en) 2011-03-15 2017-01-17 David A. Stewart Video camera gun barrel mounting system

Also Published As

Publication number Publication date
US7513700B2 (en) 2009-04-07
WO2007068511A1 (en) 2007-06-21
KR20080080631A (en) 2008-09-04
EP1960070A1 (en) 2008-08-27
JP2009519066A (en) 2009-05-14

Similar Documents

Publication Publication Date Title
US11195307B2 (en) Image processing apparatus, image processing method, and program
US10864433B2 (en) Using a portable device to interact with a virtual space
US8477099B2 (en) Portable data processing appartatus
US9084938B2 (en) Handheld device for spectator viewing of an interactive application
US9504920B2 (en) Method and system to create three-dimensional mapping in a two-dimensional game
CN101189049B (en) Method of executing an application in a mobile device
US9262015B2 (en) System for portable tangible interaction
JP5448073B2 (en) Information processing apparatus, information processing program, information processing system, and selection target selection method
CN111566596B (en) Real world portal for virtual reality displays
JP2015187884A (en) Pointing device with camera and mark output
CN104870941A (en) Display of separate computer vision based pose and inertial sensor based pose
US10537814B2 (en) Screen coding methods and camera based game controller for video shoot game
US10979700B2 (en) Display control apparatus and control method
JP2015118556A (en) Augmented reality overlay for control devices
US7513700B2 (en) Image element identifier
CN114706489A (en) Virtual method, device, equipment and storage medium of input equipment
US20170354866A1 (en) Non-transitory storage medium having game program stored therein, information processing apparatus, information processing system, game processing method
KR101914660B1 (en) Method and apparatus for controlling displaying of augmented reality contents based on gyro sensor
CN104580967A (en) Map projection method based on portable projector, and device for projection
US20100053080A1 (en) Method For Setting Up Location Information On Display Screens And A Recognition Structure Thereof
EP4311584A2 (en) Data processing apparatus and method
JP6935921B2 (en) Terminal devices and programs
CN101330948A (en) Image element identifier
CN115463409A (en) Frame supplementing method for picture information of handheld device and handheld device

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY ERICSSON MOBILE COMMUNICATIONS AB, SWEDEN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NELSON, JOAKIM;REEL/FRAME:017916/0565

Effective date: 20060522

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20170407