US9160931B2 - Modifying captured image based on user viewpoint - Google Patents

Modifying captured image based on user viewpoint Download PDF

Info

Publication number
US9160931B2
US9160931B2 US14/110,707 US201114110707A US9160931B2 US 9160931 B2 US9160931 B2 US 9160931B2 US 201114110707 A US201114110707 A US 201114110707A US 9160931 B2 US9160931 B2 US 9160931B2
Authority
US
United States
Prior art keywords
user
image
viewpoint
view
image capture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
US14/110,707
Other languages
English (en)
Other versions
US20140049667A1 (en
Inventor
Ian N. Robinson
April Slayden Mitchell
Daniel G. Gelb
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Development Co LP
Original Assignee
Hewlett Packard Development Co LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Development Co LP filed Critical Hewlett Packard Development Co LP
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GELB, DANIEL G., MITCHELL, APRIL SLAYDEN, ROBINSON, IAN N.
Publication of US20140049667A1 publication Critical patent/US20140049667A1/en
Application granted granted Critical
Publication of US9160931B2 publication Critical patent/US9160931B2/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H04N5/23296—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
    • H04N5/2251—
    • H04N5/23293—
    • H04N5/232—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/30—Transforming light or analogous information into electric information
    • H04N5/33—Transforming infrared radiation

Definitions

  • One step in taking a photograph or capturing video is framing of the desired image.
  • Many cameras include an LCD display or viewfinder facing the user which displays the framed image before and after image capture. Users can, for example, adjust the image displayed in the LCD display using a combination of camera movement (the user physically moving the camera location) and zoom control to frame the desired image.
  • camera movement the user physically moving the camera location
  • zoom control to frame the desired image.
  • the image displayed on the viewfinder does not relate to the user's view of the scene past the camera, but instead depends only on the orientation of the camera and the position of its zoom control. This mismatch between the user's and the camera's view directions can be especially problematic when the view of the viewfinder is impaired, washed out by strong sunlight, for example.
  • FIG. 1 illustrates a block diagram of an image capture system for framing an image according to an embodiment of the invention
  • FIG. 2A illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a first position relative to the image capture system of FIG. 1 according to an embodiment of the invention
  • FIG. 2B illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a second position relative to the image capture system of FIG. 1 according to an embodiment of the invention
  • FIG. 2C illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a third position relative to the image capture system of FIG. 1 according to an embodiment of the invention
  • FIG. 3 illustrates a side cross-sectional view illustrating the field of view of the framing system described in FIG. 1 according to an embodiment of the invention
  • FIG. 4A illustrates a flow diagram of a method of framing an image for display according to an embodiment of the invention
  • FIG. 4B illustrates a flow diagram of a step of modifying the first captured image for display according to an embodiment of the invention
  • FIG. 5 shows a computer system for implementing the method shown in FIGS. 4A and 4B described in accordance with embodiments of the present invention.
  • FIG. 1 shows a block diagram of an image capture system according to an embodiment of the invention.
  • the image capture system 100 includes at least a first image capture device and at least a first viewpoint assessment sensor.
  • the first image capture device captures a scene or object of interest while the viewpoint assessment sensor captures information that can be used in determining the user viewpoint.
  • the image capture system 100 is comprised of: a first image capture device 110 for capturing a first image, the first image capture device having a first field of view 112 ; a viewpoint assessment sensor 116 for capturing user viewpoint information, the viewpoint assessment sensor 116 having a second field of view 118 ; a viewpoint assessment component 122 for determining the location of the user viewpoint in the second field of view; and an image modification component 124 , for modifying the first image responsive to the user viewpoint in the second field of view.
  • the image capture system 100 includes at least a first image capture device 110 and a viewpoint assessment sensor 116 .
  • the image capture system 100 of the present invention includes a viewpoint assessment sensor 116 for capturing the viewpoint of the user taking the photograph or video images.
  • the viewpoint assessment sensor 116 is positioned to capture information about a user and also his position in front of the viewfinder display area 128 . This additional information can be used to determine the user viewpoint.
  • the user viewpoint information is used to modify the image of the scene captured by the first image capture device, so that the modified image appears to be from the user's viewpoint.
  • the view framed by the viewfinder display 128 being centered on the direction of the camera lens of the first image capture device 110 , it is instead reflective of the viewpoint of the user taking the photograph.
  • a mismatch between what is displayed and what is seen by the user can occur if there is a difference between the view direction of the camera lens and the view of the scene past the camera seen by the user's eyes.
  • a goal of the present invention is to change the viewfinder display content so that it substantially eliminates any human perceivable mismatch that occurs between what the user is viewing past the camera and what is being framed by the viewfinder display 128 . Having the captured image display what the user is actually seeing provides an intuitive and easy to use interface for the user.
  • a simple opening through the body of the camera serves as the viewfinder display 128 , instead of the commonly available LCD or OLED display.
  • the captured first image would be modified to correspond to the user's view framed by this opening in the camera.
  • an embodiment that uses a conventional display screen for the viewfinder 128 has increased functionality, enabling, for example, review of captured images.
  • the display would appear to simulate a rectangular opening through the camera.
  • the image capture device 110 or the viewpoint assessment sensor 116 may be a camera or video camera device capable of capturing an image in analog or digital form.
  • the first image capture device 110 and/or the viewpoint assessment sensors may be depth cameras. Using a depth camera provides increased flexibility with respect to the methods available for determining user viewpoint and potentially increases the accuracy of the position dependent correction methods used to provide improved modification of the first image.
  • the image capture system housing 132 shown has both a front surface 136 and a back surface 138 , where the front and back surfaces are substantially planar and parallel to each other.
  • the first image capture device 110 is rear facing and the lens of the camera is embedded in or extends from the first rear surface 138 .
  • the first image capture device 110 has a first field of view 112 , the boundaries of which are represented by dotted lines 112 a , 112 b .
  • the user positions the camera so that the field of view 112 of the lens of the first camera 110 encompasses the objects that the user wants to take a photograph or video of.
  • the viewpoint assessment sensor 116 is front facing and is embedded in or extends from the second front surface 136 .
  • the viewpoint assessment sensor 116 has a second field of view 118 the boundaries of which are represented by dotted lines 118 a , 118 b .
  • the goal of the viewpoint assessment sensor 116 is to determine the user's viewpoint in space with respect to the camera, more specifically with respect to the plane of the viewfinder frame 128 . As long as the user is in the range and field of view of the viewpoint assessment sensor, information can be extracted that can be used to determine the user viewpoint.
  • the viewpoint assessment sensor 116 is a second image capture device positioned to face towards the user to capture the user's head location, position and/or facial details.
  • the dotted lines 118 a , 118 b show the capture range or field of view of the lens of the image capture device 116 .
  • the field of view of the image capture device is dependent upon the upon the type of image capture device used and the position of the camera.
  • the data 144 from the second image capture device 116 is used by the viewpoint assessment component 120 to determine the user's viewpoint.
  • the display controller component 148 includes: a viewpoint assessment component 122 and an image modification component 124 .
  • the captured image data 144 from the second image capture device is input into the viewpoint assessment component 122 .
  • the second image includes image data regarding a user within the field of view of the second image capture device. Information regarding the viewpoint of the user can be extracted from the second image to determine the user viewpoint.
  • Image information 148 from the first image capture device 110 is input into the image modification component 124 .
  • the image modification component 124 determines how to modify the first captured image to more closely match the user viewpoint.
  • the modified image 154 is output to the LCD display screen 128 for viewing by the user.
  • the user when viewing the LCD display will see the image reflective of the camera location with respect to his own viewpoint—and not the view direction of the camera lens on the rear facing side of the camera.
  • the timing of the image capture of the first and second image capture is coordinated so that the image that is captured by the first image capture device corresponds to the user viewpoint (captured by the second image capture device) at substantially the same time t 1 or within a predefined delay. This way as the user viewpoint changes, the corresponding image/scene that is seen by the user at the same time is modified.
  • the present invention is attempting to simulate the image capture of the image in front of the user as if the image capture system was transparent and the user was placing a frame in the air to capture the scene behind the frame.
  • the view captured is the view from the camera lens
  • the view does not change unless you change the camera position.
  • the view being captured is viewpoint dependent, if the user moves his head position, the displayed view changes.
  • FIGS. 2A-2C illustrate how changes in the user viewpoint with respect to the image capture system changes the image captured by the system 100 .
  • FIG. 2A illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a first position relative to the image capture system of FIG. 1 .
  • the lines 210 a , 210 b extending from the user's eyes to the side boundaries of the display screen 128 of the camera are illustrative of the boundaries of the view frustum defined by the user viewpoint and the boundaries of the plane of the viewfinder display.
  • FIG. 2A illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a first position relative to the image capture system of FIG. 1 .
  • the lines 210 a , 210 b extending from the user's eyes to the side boundaries of the display screen 128 of the camera are illustrative of the boundaries of the view frustum defined by the user viewpoint and the boundaries of the
  • a user view frustum shown by the lines 210 a , 210 b , looking through the LCD display screen 128 would encompass the cylindrical object 220 .
  • the cube 230 which is not in the user's view frustum, would not appear on the LCD display screen 128 .
  • FIG. 2B illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a second position relative to the image capture system of FIG. 1 .
  • the user eye position shown in FIG. 2B is shifted upwards (see arrow 226 ).
  • a user view frustum shown by the lines 210 a , 210 b looking through the LCD display screen 128 , would encompass the cube 230 .
  • the object in the view frustum—the cube 230 — would be captured by the image capture system 100 and displayed on the display 128 .
  • the cylinder 220 which is not in the user's view frustum, would not appear on the LCD display screen 128 .
  • FIG. 2C illustrates a side cross-sectional view of a user eye position, where the user's eyes are in a third position.
  • the user's eye position (see arrow 232 ) is moved closer to the display.
  • the user view frustum shown by the lines 210 a , 210 b looking through the LCD display screen 128 , would now encompass both the cylinder 220 and the cube 230 .
  • Changing the distance between the user's eyes and the display of the image capture system 100 changes the apparent zoom of the camera. Thus, for instance, moving the device closer to the user's face would have the effect of zooming the view out.
  • FIG. 3 illustrates a side cross-sectional view illustrating the field of view of the image capture system described in FIG. 1 .
  • the image that is displayed on the viewfinder display screen 128 is at least in part dependent upon a user viewpoint.
  • the display system 100 determines how to modify the first captured image.
  • the lens of the first image capture device is a wide angle lens and the first captured image is modified by only choosing the portion of the first captured image that corresponds to the user's view “through” the viewfinder display.
  • the displayed image presents a different view of the captured image than would be presented by the first image capture device, such that the view of the image displayed on the display matches the user's view direction with respect to the camera. Further, as the user changes his position (angle from the display screen, distance from the display screen, etc.) then his viewpoint (V) and thus the image on the display 128 changes.
  • FIG. 3 shows a view of the points used in one implementation to generate a viewpoint from the perspective of a user.
  • user viewpoint can be used, with respect to FIG. 1 the location of the user viewpoint is described (for purposes of discussion only) as the position (V) located midway between the user's two eyes.
  • the dotted line 310 is representative of the line between the camera lens of the second image capture device and the user viewpoint.
  • the term “user” is used throughout, when used with respect to the viewpoint assessment sensors, the term user refers to the features of the user (eyes, head and shoulders, face, etc.) necessary for the viewpoint assessment component, to determine the user viewpoint. Eye position or location can be found using various techniques including through the user of face detection software or by approximation from the position of a tracking device being worn by the user.
  • determining the user's eye position is found using IR (infrared) illumination.
  • an IR illumination source floods IR light in the vicinity of the user.
  • the viewpoint assessment sensor 1 116 is an IR sensitive device, typically an IR sensitive camera. IR light is reflected off the user's retina and the IR sensitive camera (the viewpoint assessment sensor) captures the user's eye location relative to the display screen and the object positioned behind the display screen. Based on the location of the user's eyes, the viewpoint assessment component determines the user's viewpoint with respect to the screen and the object.
  • information about the user's eye position can be achieved using face detection software 180 .
  • the viewpoint assessment sensor 116 can be simple cameras. Based on the feature detection software 180 in the viewpoint assessment component 122 , the eye position of the user is located. Based on the eye position of the user, the user viewpoint can be determined.
  • some force of active tracking of the user's position is used.
  • the user could wear a device that is capable of being tracked in 3D space.
  • retro-reflective markers could be applied to an object worn by the user (i.e. a headset, a pair of glasses, etc.). Based on the position of the tracked object, the viewpoint assessment component software 116 provides an accurate approximation of the user's eye location.
  • the user could wear a device that emits a sound that would allow the user to be tracked in 3D space.
  • the eye location of the user can be approximated using depth cameras as the viewpoint assessment sensor 116 .
  • Depth cameras can include sensors that compute depth from stereo, depth from IR time-of-flight, or depth from observing projected IR structured light.
  • a depth camera is used to locate the user (user's head and shoulder at a depth or depth range nearer to the depth camera) and the background (at a depth or depth range further from the camera). The viewpoint location can then be approximated as being at the center of the head shaped region in the depth image. Any of the above techniques listed in the preceding paragraphs can be combined to give a more accurate determination of the viewpoint location.
  • FIG. 4A illustrates a flow diagram of a method of framing an image for display according to an embodiment of the invention
  • FIG. 4A illustrates a flow diagram of a method of framing an image for display according to an embodiment of the invention.
  • the method for framing an image for display includes the steps of: capturing a first image with a first image device, the first image capture device having a first field of view ( 410 ); capturing viewpoint assessment information with a viewpoint assessment sensor, the viewpoint assessment sensor having a second field of view (step 420 ); using viewpoint assessment information to determine the location of the user viewpoint ( 430 ); and responsive to the user viewpoint, modifying the first image ( 440 ).
  • the modified image is then displayed ( 446 ) on the display screen 128 of the image capture system 100 .
  • step 410 shows the step of capturing a first image with a first image capture device 110 having a first field of view 112 .
  • the boundaries of the first field of view are represented by lines 112 a and 112 b.
  • step 420 is the step of capturing viewpoint assessment information with a viewpoint assessment sensor having a second field of view.
  • This step covers multiple types of sensors that can be used to capture or gather information that can be used for determining user viewpoint.
  • the viewpoint assessment sensor 116 is an ultrasound sensor and active tracking of the user is used to determine the user viewpoint.
  • the field of view is the capture zone of the sensor.
  • the captured viewpoint assessment information is a second captured image.
  • the boundaries of the second field of view of the second image capture device 116 are represented by lines 118 a and 118 b.
  • the viewpoint assessment information is used to determine the user viewpoint.
  • the viewpoint assessment sensor is an image capture device—the front-facing camera (the second image capture device) is used to ascertain the user's viewpoint.
  • the viewpoint assessment sensor is an image capture device—the front-facing camera (the second image capture device) is used to ascertain the user's viewpoint.
  • the determination of the user viewpoint is achieved by using a face detector software 180 on the output of the front-facing camera to determine user eye location and from user eye location—the user viewpoint.
  • other techniques are used to determine the user viewpoint including techniques such as, but not limited to the techniques discussed with reference to FIG.
  • 3 including the following: using a pair of front-facing cameras could supply depth from stereo when the assumption is that the nearest object is the user's head; a camera could be augmented using a depth sensor (ultrasound, structured light, or time-of-flight, for instance), information captured by the depth sensor could then be used to match a 3D face to determine user viewpoint in 3D; a wearable headset could be augmented to assist in locating the user's head (e.g. act as an IR or ultrasound beacon). From the output image 144 , the position of the user's eyes and thus the location of the user's viewpoint can be established. The location and size of the face in the image is used to calculate the viewpoint in three dimensions relative to the device.
  • a depth sensor ultrasound, structured light, or time-of-flight, for instance
  • step 440 of FIG. 4A is expanded to provide further detail regarding step ( 440 ) of modifying the first captured image for display, for the case where the viewpoint assessment sensor is an image capture device.
  • the first image is modified by: calculating the view frustum ( 320 in FIG. 3 ) based on the user viewpoint and the boundaries of the display (step 450 ); based on the calculated view frustum, determining a user view portion of the first image that corresponds to the calculated view frustum ( 330 in FIG. 3 ) ( 460 ); and modifying the first image to correspond to the user view portion ( 470 ).
  • a view frustum can be calculated ( 450 ) using the known user viewpoint and the boundaries of the device display 128 .
  • the apex of the view frustum is the viewpoint (V).
  • the view frustum is defined by the apex (V) and the boundaries of the display screen 128 .
  • the boundaries of the user's view frustum are represented by dotted lines 320 a and 320 b.
  • a user view portion of the first image is determined that corresponds to the calculated view frustum.
  • the user view portion is a portion of the first image that is within the field of view of the first image capture device that corresponds to the view frustum boundaries 320 a , 320 b .
  • the goal is to define the user view portion of the first image that corresponds to or approximately matches the boundaries of the user view frustum 320 a , 320 b .
  • the portion of the first image that corresponds to the boundaries 330 a and 330 b is chosen as being a close match.
  • the user view portion (the portion of the image falling within the boundaries 330 a , 330 b ) chosen for the embodiment shown in FIG. 3 , the user view portion is selected from the image information in the first field of view ( 112 a , 112 b ) that approximates the user's field of view 320 a , 320 b in the first image.
  • this match is an approximation since the portion of the first image selected within the boundaries 330 a and 330 a does not include all of the image information in the user view frustum.
  • Different techniques which select more or less of the first image may be selected, however, some techniques may be more computational intensive and simpler techniques may be chosen.
  • the angle between 330 a / 330 b will be very close to the angle 320 a / 320 b .
  • Information on the distance of objects in a scene normally obtained by a typical camera system to enable automatic focusing can be used to determine how far out the lines 320 and 330 should intersect.
  • the boundaries of the user view frustum 320 a , 320 b may not fall completely within the boundaries of the first field of view, the user view portion may not include all of the image information that falls within the user view frustum boundaries since this information may not be captured by the first image capture device.
  • the goal is to include as much of the image information encompassed by the user view frustum as possible.
  • the first image capture device 110 may have an optical zoom capability.
  • the image modification component 124 could control the amount of optical zoom (the angle between 112 a and 112 b in FIG. 3 ) to maximize the image sensor area corresponding to the user view frustum.
  • the field of view of the lens could be reduced (zooming in) to more closely match the view frustum 330 .
  • the modified image is created, it is displayed on the display screen 128 .
  • the displayed image is reflective of the user's viewpoint.
  • the image capture system includes an interface for modifying the display so that the displayed image presents a wider or narrower field of view than the user's view frustum.
  • the image capture system switches between display modes dependent upon whether the user's eyes/face or detected. For example, in the case where the image capture system might be positioned high above the user's face to shoot over a crowd, for example—the image capture system would not detect the user's face. In this case, the image capture system would switch between a mode which displays a modified image reflective of the user viewpoint according to the present invention, to a conventional display output mode which simply displays the first image.
  • FIG. 5 shows a computer system for implementing the methods shown in FIGS. 4A-4B and described in accordance with embodiments of the present invention.
  • the method 400 represents generalized illustrations and that other steps may be added or existing steps may be removed, modified or rearranged without departing from the scopes of the method 400 .
  • the descriptions of the method 400 are made with reference to the system 100 illustrated in FIG. 1 and the system 500 illustrated in FIG. 5 and thus refers to the elements cited therein. It should, however, be understood that the method 400 is not limited to the elements set forth in the system 500 . Instead, it should be understood that the method 400 may be practiced by a system having a different configuration than that set forth in the system 500 .
  • the operations set forth in the method 400 may be contained as utilities, programs or subprograms, in any desired computer accessible medium.
  • the method 400 may be embodied by computer programs, which may exist in a variety of forms both active and inactive. For example, they may exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats. Any of the above may be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form.
  • FIG. 5 illustrates a block diagram of a computing apparatus 500 configured to implement or execute the methods 400 depicted in FIGS. 4A and 4B and described in accordance with embodiments of the present invention.
  • the computing apparatus 500 may be used as a platform for executing one or more of the functions described hereinabove with respect to the display controller component 130 .
  • the computing apparatus 500 includes one or more processor(s) 502 that may implement or execute some or all of the steps described in the methods 400 . Commands and data from the processor 502 are communicated over a communication bus 504 .
  • the computing apparatus 500 also includes a main memory 506 , such as a random access memory (RAM), where the program code for the processor 502 , may be executed during runtime, and a secondary memory 508 .
  • the secondary memory 508 includes, for example, one or more hard drives 510 and/or a removable storage drive 512 , representing a removable flash memory card, etc., where a copy of the program code for the method 400 may be stored.
  • the removable storage drive 512 reads from and/or writes to a removable storage unit 514 in a well-known manner.
  • Non-transitory computer readable storage devices that may be used to implement the present invention include but are not limited to conventional computer system RAM, ROM, EPROM, EEPROM and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any electronic device and/or system capable of executing the functions of the above-described embodiments are encompassed by the present invention.
  • any of the memory components described 506 , 508 , 514 may also store an operating system 530 , such as Mac OS, MS Windows, Unix, or Linux; network applications 532 ; and a display controller component 148 .
  • the operating system 530 may be multi-participant, multiprocessing, multitasking, multithreading, real-time and the like.
  • the operating system 530 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to the display 520 ; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one or more buses 504 .
  • the network applications 532 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet, USB, and FireWire.
  • the computing apparatus 500 may also include an input devices 516 , such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors, etc., and a display(s) 520 , such as the LCD screen display 128 shown for Example in FIG. 1 .
  • a display adaptor 522 may interface with the communication bus 504 and the display 520 and may receive display data from the processor 502 and convert the display data into display commands for the display 520 .
  • the processor(s) 502 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or more network interfaces 524 such as a Local Area Network LAN, a wireless 802.11x LAN, a 3G mobile WAN or a WiMax WAN.
  • a network interface 524 such as a Local Area Network LAN, a wireless 802.11x LAN, a 3G mobile WAN or a WiMax WAN.
  • an interface 526 may be used to receive an image or sequence of images from imaging components 528 , such as the image capture device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
US14/110,707 2011-04-08 2011-04-08 Modifying captured image based on user viewpoint Expired - Fee Related US9160931B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2011/031833 WO2012138355A1 (fr) 2011-04-08 2011-04-08 Système et procédé de modification d'une image

Publications (2)

Publication Number Publication Date
US20140049667A1 US20140049667A1 (en) 2014-02-20
US9160931B2 true US9160931B2 (en) 2015-10-13

Family

ID=46969486

Family Applications (1)

Application Number Title Priority Date Filing Date
US14/110,707 Expired - Fee Related US9160931B2 (en) 2011-04-08 2011-04-08 Modifying captured image based on user viewpoint

Country Status (2)

Country Link
US (1) US9160931B2 (fr)
WO (1) WO2012138355A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9691152B1 (en) * 2015-08-14 2017-06-27 A9.Com, Inc. Minimizing variations in camera height to estimate distance to objects
US10956791B2 (en) 2019-07-19 2021-03-23 LayerJot, Inc. Interactive generation and publication of an augmented-reality application

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101897549B1 (ko) * 2012-01-09 2018-09-12 삼성전자주식회사 휴대 단말기에서 카메라 뷰 영역을 표시하는 장치 및 방법
KR101983288B1 (ko) * 2012-11-22 2019-05-29 삼성전자주식회사 카메라 촬영 제어장치 및 방법
US20140184854A1 (en) * 2012-12-28 2014-07-03 Motorola Mobility Llc Front camera face detection for rear camera zoom function
JP2014160982A (ja) 2013-02-20 2014-09-04 Sony Corp 画像処理装置および撮影制御方法、並びにプログラム
JP6499583B2 (ja) * 2013-09-24 2019-04-10 シャープ株式会社 画像処理装置及び画像表示装置
US9525817B2 (en) * 2013-11-22 2016-12-20 Samsung Electro-Mechanics Co., Ltd. System and method of controlling imaging direction and angle of view of camera
US10799109B2 (en) * 2016-01-15 2020-10-13 Jand, Inc. Systems and methods for determining distance from an object
CN106341607A (zh) * 2016-10-24 2017-01-18 深圳市金立通信设备有限公司 一种提醒用户调整拍照角度的方法及终端
TWI633336B (zh) * 2017-02-24 2018-08-21 宏碁股份有限公司 頭戴式顯示器、其視野校正方法以及混合實境顯示系統
KR102036204B1 (ko) * 2018-06-18 2019-10-24 양경호 휴대형 이동통신 단말기의 자기 영상 촬영장치 및 그 제어 방법

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6522360B1 (en) * 1993-10-04 2003-02-18 Canon Kabushiki Kaisha Image pickup apparatus performing autofocus processing and image enlargement in a common selected image plane region
US20040212712A1 (en) * 2003-04-28 2004-10-28 Stavely Donald J. Digital imaging method and apparatus using eye-tracking control
US20060044399A1 (en) 2004-09-01 2006-03-02 Eastman Kodak Company Control system for an image capture device
US7023483B2 (en) 2000-02-21 2006-04-04 Hewlett-Packard Development Company, L.P. Framing aid for a document capture device
US7139445B2 (en) 2001-12-06 2006-11-21 Hewlett-Packard Development Company L.P. Image capture device and method of selecting and capturing a desired portion of text
US20090110235A1 (en) 2007-10-26 2009-04-30 Samsung Electronics Co., Ltd. System and method for selection of an object of interest during physical browsing by finger framing
JP2009288613A (ja) 2008-05-30 2009-12-10 Univ Of Tokyo 情報提示装置
US20100026780A1 (en) 2008-07-31 2010-02-04 Nokia Corporation Electronic device directional audio capture
US20100079449A1 (en) 2008-09-30 2010-04-01 Apple Inc. System and method for rendering dynamic three-dimensional appearing imagery on a two-dimensional user interface
US20100171691A1 (en) 2007-01-26 2010-07-08 Ralph Cook Viewing images with tilt control on a hand-held device
US20100253825A1 (en) 2003-10-15 2010-10-07 Kenichi Horie Camera
US7860382B2 (en) * 2006-10-02 2010-12-28 Sony Ericsson Mobile Communications Ab Selecting autofocus area in an image
US20110216060A1 (en) * 2010-03-05 2011-09-08 Sony Computer Entertainment America Llc Maintaining Multiple Views on a Shared Stable Virtual Space
US20140204003A1 (en) * 2008-01-23 2014-07-24 Michael F. Deering Systems Using Eye Mounted Displays

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6522360B1 (en) * 1993-10-04 2003-02-18 Canon Kabushiki Kaisha Image pickup apparatus performing autofocus processing and image enlargement in a common selected image plane region
US7023483B2 (en) 2000-02-21 2006-04-04 Hewlett-Packard Development Company, L.P. Framing aid for a document capture device
US7139445B2 (en) 2001-12-06 2006-11-21 Hewlett-Packard Development Company L.P. Image capture device and method of selecting and capturing a desired portion of text
US20040212712A1 (en) * 2003-04-28 2004-10-28 Stavely Donald J. Digital imaging method and apparatus using eye-tracking control
US20100253825A1 (en) 2003-10-15 2010-10-07 Kenichi Horie Camera
US20060044399A1 (en) 2004-09-01 2006-03-02 Eastman Kodak Company Control system for an image capture device
US7860382B2 (en) * 2006-10-02 2010-12-28 Sony Ericsson Mobile Communications Ab Selecting autofocus area in an image
US20100171691A1 (en) 2007-01-26 2010-07-08 Ralph Cook Viewing images with tilt control on a hand-held device
US20090110235A1 (en) 2007-10-26 2009-04-30 Samsung Electronics Co., Ltd. System and method for selection of an object of interest during physical browsing by finger framing
US20140204003A1 (en) * 2008-01-23 2014-07-24 Michael F. Deering Systems Using Eye Mounted Displays
JP2009288613A (ja) 2008-05-30 2009-12-10 Univ Of Tokyo 情報提示装置
US20100026780A1 (en) 2008-07-31 2010-02-04 Nokia Corporation Electronic device directional audio capture
US20100079449A1 (en) 2008-09-30 2010-04-01 Apple Inc. System and method for rendering dynamic three-dimensional appearing imagery on a two-dimensional user interface
US20110216060A1 (en) * 2010-03-05 2011-09-08 Sony Computer Entertainment America Llc Maintaining Multiple Views on a Shared Stable Virtual Space

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
François, Alexandre RJ, and E-Ye Kang. "A handheld mirror simulation." Multimedia and Expo, 2003. ICME'03. Proceedings. 2003 International Conference on. vol. 2. IEEE, 2003.
Written opinion and international search report for counterpart PCT patent application, PCT/US2011/031833, dated Dec. 26, 2011.

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9691152B1 (en) * 2015-08-14 2017-06-27 A9.Com, Inc. Minimizing variations in camera height to estimate distance to objects
US10037614B2 (en) 2015-08-14 2018-07-31 A9.Com, Inc. Minimizing variations in camera height to estimate distance to objects
US10956791B2 (en) 2019-07-19 2021-03-23 LayerJot, Inc. Interactive generation and publication of an augmented-reality application

Also Published As

Publication number Publication date
WO2012138355A1 (fr) 2012-10-11
US20140049667A1 (en) 2014-02-20

Similar Documents

Publication Publication Date Title
US9160931B2 (en) Modifying captured image based on user viewpoint
EP3467585B1 (fr) Système de suivi de visiocasque
KR101741335B1 (ko) 안구추적을 기반으로 한 홀로그램 디스플레이 방법 및 홀로그램 디스플레이 장치
US10116922B2 (en) Method and system for automatic 3-D image creation
US10666856B1 (en) Gaze-directed photography via augmented reality feedback
JP6860488B2 (ja) 複合現実システム
US10242454B2 (en) System for depth data filtering based on amplitude energy values
KR101730737B1 (ko) 안구추적을 기반으로 한 상이한 거리 자동적응 홀로그램 디스플레이 방법 및 장치
US10165176B2 (en) Methods, systems, and computer readable media for leveraging user gaze in user monitoring subregion selection systems
US8139935B2 (en) 3D camera with foreground object distance sensing
EP2410733A2 (fr) Système de caméra et méthode d'affichage de photos
KR20150093831A (ko) 혼합 현실 환경에 대한 직접 상호작용 시스템
KR20160094190A (ko) 시선 추적 장치 및 방법
US20200267309A1 (en) Focusing method and device, and readable storage medium
KR101690646B1 (ko) 시 스루 디스플레이를 위한 카메라 구동 장치 및 그 방법
CN112655202B (zh) 用于头戴式显示器的鱼眼镜头的减小带宽立体失真校正
JP2022535520A (ja) 電子機器の制御方法及び電子機器
CN115209057B (zh) 一种拍摄对焦方法及相关电子设备
CN110895433B (zh) 用于增强现实中用户交互的方法和装置
WO2019128400A1 (fr) Procédé de commande de déplacement d'angle de vue photographique de lunettes intelligentes
WO2016197494A1 (fr) Procédé et dispositif de réglage de zone de mise au point
US9426446B2 (en) System and method for providing 3-dimensional images
CN113870213A (zh) 图像显示方法、装置、存储介质以及电子设备
CN107852457B (zh) 取景器装置和操作该装置的方法
CN112073632A (zh) 图像处理方法、设备及存储介质

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ROBINSON, IAN N.;MITCHELL, APRIL SLAYDEN;GELB, DANIEL G.;REEL/FRAME:031369/0726

Effective date: 20110504

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction
MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20231013