US20130187854A1 - Pointing Device Using Camera and Outputting Mark - Google Patents
Pointing Device Using Camera and Outputting Mark Download PDFInfo
- Publication number
- US20130187854A1 US20130187854A1 US13/795,749 US201313795749A US2013187854A1 US 20130187854 A1 US20130187854 A1 US 20130187854A1 US 201313795749 A US201313795749 A US 201313795749A US 2013187854 A1 US2013187854 A1 US 2013187854A1
- Authority
- US
- United States
- Prior art keywords
- mark
- image
- pointing
- captured image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims 2
- 230000000295 complement effect Effects 0.000 claims 1
- 238000012015 optical character recognition Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 1
- 239000004579 marble Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000010409 thin film Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
- G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
- G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
- G06F3/0321—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
- G06F3/03545—Pens or stylus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F3/0386—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry for light pen
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/142—Image acquisition using hand-held instruments; Constructional details of the instruments
- G06V30/1423—Image acquisition using hand-held instruments; Constructional details of the instruments the instrument generating sequences of position coordinates corresponding to handwriting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/51—Housings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/19—Image acquisition by sensing codes defining pattern positions
Definitions
- the present invention relates to the pointing device like mouse or joystick with camera for capturing the image of display like PC monitor and image processing means for recognizing and tracking the icon of pointing cursor or mark.
- the pointing device of present invention can be used in the form of TV remote controller or digital stylus pen.
- There is the similar invention Korean patent 10-0532525-0000, 3 dimensional pointing device using camera.
- the said similar invention has the problem that it requires the optical mark (light source like infrared LED) attached on the display to be captured by camera and the pointing device of electronic blackboard has the problem that it requires the ultra sonic sensor or infrared sensor.
- the pointing device of PDA or tablet PC has the problem that they requires the pressure sensor or touch sensor. It is difficult for the portable flexible thin film display like OLED to adopt such conventional heavy and volumetric sensor systems on it.
- a pointing device which does not require any sensor system (like infrared LED, ultra sonic sensor, infrared sensor and pressure sensor) attached on the display.
- any sensor system like infrared LED, ultra sonic sensor, infrared sensor and pressure sensor
- the present invention provides the pointing device which uses the cursor icon or pattern displayed on screen as a mark instead of physical mark like infrared light source or ultrasonic source.
- the pointing device of present invention it is possible to move the pointing cursor like the mouse or joystick cursor without attaching physical sensor system or tracking mark on display including flexible display like OLED.
- FIG. 1 is the embodiment of present invention as tablet PC and pen camera.
- FIG. 2 is the arrow mark moving in left direction.
- FIG. 3 is the arrow mark moving in left and bottom direction.
- FIG. 4 is the 2 dimensional array of cell of display.
- FIG. 5 is an example of mark image.
- FIG. 6 is the negative image of FIG. 5 .
- FIG. 7 is the display which is outputting mark image of 2 dimensional array of cell of pattern.
- the pointing device of present invention includes mark outputting portion like conventional display (computer monitor, TV monitor, beam-projected screen), camera portion for capturing the said mark outputting portion and image processing portion which recognizes the mark from the captured image and produces the pointing signal.
- the appearance of camera portion can be a remote controller for digital TV, stylus pen for tablet PC or gun controller for shooting game.
- the image processing portion can be image processing program in DSP (digital signal processor), microcontroller or computer.
- the mark can be the conventional mouse cursor of arrow shape, or any type of pattern like +, hand, or some user defined icon for game. There is no limit on size, shape and color of the mark if the mark is recognizable by the image processing portion.
- FIG. 1 shows the pointing device of present invention which is pen type camera (ca) on the display (mo) of tablet PC.
- the camera captures the mark (mr) which is the arrow icon (mk) on the display like the conventional mouse cursor icon of Microsoft Windows.
- the captured images motion video
- the user must move the pen camera onto the cursor icon of display so that the cursor icon can be captured by the pen camera.
- the position of the mark in the captured image moves from the center of the image to the boundary of the image and the movement (in other words, motion vector) of the mark in the captured image can be recognized by the image processing portion by comparing the previous frame image and current frame image.
- the image processing portion transfers the detected motion vector to the mark outputting portion and the the mark outputting portion produces the control signal to move back the mark (cursor icon) to the center of the captured image so that the mark follows the movement of pen camera. For example, If the pen camera in FIG. 1 is moved in x direction (dx) then the mark in the captured image moves in ⁇ x direction ( ⁇ dx) as shown in FIG.
- the image processing portion produces signal so that the mark outputting portion can increase the x coordinate of the mark where the amount of the increment is proportional to the distance between the center of the captured image and the position of the mark in the captured image.
- the image processing portion finds the motion vector of the mark in the captured image and the mark outputting portion changes the coordinate of the mark in the negative direction of the found motion vector.
- such a moveing of cursor can be controlled by using the Windows API (application program interface) which can read and change the coordinate of mouse cursor. If the mark in the captured image is located in the center of the captured image then the motion vector is zero vector and there is no change of position of the mark.
- 3 shows the motion vector of mark from the dotted arrow in the previous frame to the solid arrow in the current frame.
- the smaller mark means the larger distance between pen camera and the display and the larger mark means the smaller distance between pen camera and the display.
- Such a size information of mark can be used as the another coordinate (z) of mouse cursor (x,y).
- the direction of the mark in the captured image also can be used as another coordinate (rotation angle r in FIG. 1 ).
- the viewing direction of pen camera can be detected and used as pointing signal by recognizing the distortion of the mark which contains feature points like vertex of rectangle and triangle.
- the image processing portion can not detect the mark from the captured image and the movement of the mark is stopped.
- user In order to continue the pointing procedure, user must carry the pen camera to the mark and change the viewing direction of the pen camera so that the mark can be captured by the pen camera.
- the reset button By adding the reset button to the pen camera, such a carrying action can be removed. If the user presses the reset button then the mark changes its position. More specifically, the mark outputting portion sequentially changes the position of mark as shown in FIG. 4 by the trigger signal of the reset button. The mark moves horizontally
- the mark scans all the cells sequentially. If the mark image is captured and recognized by the image processing portion during the scanning, the scanning is stopped at that time and the pointing procedure is started.
- the 6 ⁇ 6 cells of the display in FIG. 4 is an example and the real number of cells must be adjusted for a given display and camera. It is recommended to move the mark fast and use the fast camera so that the human eye can not recognize the scanning.
- the above embodiment 1 is the pen camera which is used by touching the display. If the camera is far from the display then the captured mark is too small to be recognized. In such a case, it is recommended to use the auto focusing system of camera and telescope lens or zoom lens with camera. By using such a optical apparatus, it is possible to use the pointing device of present invention as the electronic pen for tablet PC and remote controller for digital TV.
- the mark in the above embodiment 1 is fixed pattern but in this embodiment the mark is the whole image of display and the distance between the camera and the display must be adjusted so that the whole image of display can be captured.
- the mark outputting portion includes the image transferring portion which transfers the image of display to the image processing portion.
- the image processing portion finds the display region from the captured image by comparing the sub regions of the captured image with the transferred image of display (It is known as the model based vision). In Microsoft Windows XP, pressing the Print Screen Sys Rq key of computer keyboard captures the image of display and stores the image into the clipboard.
- Such an image transferring can be done by software by emulating the pressing the key or by using device driver.
- the image transferring portion can also be implemented by hardware.
- the image processing portion finds feature points from the found display and the relative distance and the direction between camera and the display can be obtained by using the formula of the perspective n point problem and such a distance and the direction information can be use to produce the pointing signal.
- Korean patent 10-0532525-0000 is the 3 dimensional pointing device by analyzing the feature points of rectangle.
- the pointing device of present invention selects the feature points from the image of display in real time and the feature points is not fixed for each frame.
- the model based vision is the technology to find the correspondence between the known model (transferred image of display) and given image (captured image by camera) and is published in chapter 18 of Computer vision a modern approach by David A. Forsyth and Jean Ponce (ISBN:0-13-085198-1).
- the flicker generating portion can be added to the mark outputting portion of embodiment 3 and the difference image calculating portion can be added to the image processing portion of embodiment 3. More specifically, the mark outputting portion outputs the blank image for every even frames (0, 2, 4, . . . ) and outputs normal image for every odd frames (1, 3, 5, . . . ). (such odd and even frame is an example and in real implementation it is possible to use 0, 4, 8, . . .
- the blank image means the image whose all the pixels have the same brightness and color. It is recommended to keep the frame rate (number of frame per second) of display large so that the human eye can not recognize the flicker and to keep the frame rate of camera also large so that the camera can capture the even and odd frame of display.
- the image processing portion obtains the difference image between the captured image of previous frame and the captured image of current frame.
- the difference image is well known concept in image processing technology whose pixel value is defined as the difference of two corresponding pixels of two images.
- the two corresponding pixels of two images means that the (x,y) positions of two pixels are the same.
- the non zero pixels of the difference image calculated by the image processing portion corresponds to the flickering display region and the zero pixels of the difference image corresponds to the background of display (non flickering region).
- the flickering display can be detected by calculating the difference image and selecting non zero pixels from the difference image.
- edge lines of background of display may corresponds to the non zero pixels if the camera is not fixed but such non zero pixels can be minimized by using high speed flickering frequency and high speed camera.
- the regions of non zero pixels of difference image are the candidates for the flickering display region in captured image and the more exact region of display can be determined by the model based vision than embodiment 3.
- the found region of display can be compared with the transferred image of display and pointing signal can be generated like the embodiment 3.
- the blank image for every even frames (0, 2, 4, . . . ) of embodiment 4 can be replaced by recognizable pattern (mark) and the image processing portion can recognize the pattern by analyzing the captured image of only even frame.
- FIG. 5 shows the example of the pattern (mark) which contains the opened rectangle and + at the center of the rectangle.
- the + mark represents the center of mark and the rectangle can be use for 3 dimensional pointing.
- the recognizable pattern of the embodiment 5 can be splitted into image of pattern and negative image of the said image of pattern. If the mark outputting portion outputs the pattern image (for 0, 3, 6, . . . frames), the negative pattern image (for 1, 4, 7, . . . frames) and the normal image (for 2, 5, 8, . . . frames) sequentially and repeatedly at enough high frequency, then human eye can not recognize the pattern image but can recognize only the normal image because the pattern and its negative pattern are time-averaged out. But high speed camera can capture the pattern image and can be recognized by image processing portion.
- FIG. 5 and FIG. 6 are the example of the pattern image and its negative image.
- the mark image of the embodiment 4 ⁇ 6 can be 2 dimensional array of patterns where the each pattern represents the 2 dimensional position (x,y) of display.
- the pattern can be 2 dimensional bar code or number.
- FIG. 7 shows the 2 dimensional array of cells where each cell contains pattern.
- the reset button of embodiment 1 can be removed by adopting such cells of pattern as the mark image with pen type camera.
- the captured image of pattern in cell can be recognized by image processing portion and can be translated into 2 dimensional position (x,y) which is corresponding to the pointing signal.
- PCT/US1999/030507 which presents the mouse for outputting absolute coordinate with special pad where the pad contains patterns and can be recognized by the camera in mouse.
- the pattern can be alphabet, number, 2 dimensional bar code. By including rectangle into pattern and recognizing it, it is possible to generate 3 dimensional pointing signal by the formula of perspective n point problem.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Position Input By Displaying (AREA)
Abstract
Pointing device like mouse or joystick comprises camera for capturing the display screen and image processing means for recognizing and tracking the pointing cursor icon or mark from the captured image and producing the pointing signal. The pointing device of present invention can be used with any type of display without and additional tracking means like ultra sonic sensor, infrared sensor or touch sensor. The pointing device of present invention includes mark outputting portion, camera portion for capturing the said mark outputting portion and image processing portion for recognizing the said mark outputting portion from the captured image and producing the pointing signal.
Description
- The present invention relates to the pointing device like mouse or joystick with camera for capturing the image of display like PC monitor and image processing means for recognizing and tracking the icon of pointing cursor or mark. The pointing device of present invention can be used in the form of TV remote controller or digital stylus pen. There is the similar invention (Korean patent 10-0532525-0000, 3 dimensional pointing device using camera). The said similar invention has the problem that it requires the optical mark (light source like infrared LED) attached on the display to be captured by camera and the pointing device of electronic blackboard has the problem that it requires the ultra sonic sensor or infrared sensor. The pointing device of PDA or tablet PC has the problem that they requires the pressure sensor or touch sensor. It is difficult for the portable flexible thin film display like OLED to adopt such conventional heavy and volumetric sensor systems on it.
- To solve the problem, it is an object of present invention to provide a pointing device which does not require any sensor system (like infrared LED, ultra sonic sensor, infrared sensor and pressure sensor) attached on the display.
- The present invention provides the pointing device which uses the cursor icon or pattern displayed on screen as a mark instead of physical mark like infrared light source or ultrasonic source.
- By using the pointing device of present invention, it is possible to move the pointing cursor like the mouse or joystick cursor without attaching physical sensor system or tracking mark on display including flexible display like OLED.
-
FIG. 1 is the embodiment of present invention as tablet PC and pen camera. -
FIG. 2 is the arrow mark moving in left direction. -
FIG. 3 is the arrow mark moving in left and bottom direction. -
FIG. 4 is the 2 dimensional array of cell of display. -
FIG. 5 is an example of mark image. -
FIG. 6 is the negative image ofFIG. 5 . -
FIG. 7 is the display which is outputting mark image of 2 dimensional array of cell of pattern. - mo: monitor, mk: mark
- ca,: camera st: stylus pen
- r: rotation of stylus pen
- mkb: mark image of 2 dimensional array of cell of pattern
- The pointing device of present invention includes mark outputting portion like conventional display (computer monitor, TV monitor, beam-projected screen), camera portion for capturing the said mark outputting portion and image processing portion which recognizes the mark from the captured image and produces the pointing signal. The appearance of camera portion can be a remote controller for digital TV, stylus pen for tablet PC or gun controller for shooting game. The image processing portion can be image processing program in DSP (digital signal processor), microcontroller or computer. The mark can be the conventional mouse cursor of arrow shape, or any type of pattern like +, hand, or some user defined icon for game. There is no limit on size, shape and color of the mark if the mark is recognizable by the image processing portion.
FIG. 1 shows the pointing device of present invention which is pen type camera (ca) on the display (mo) of tablet PC. The camera captures the mark (mr) which is the arrow icon (mk) on the display like the conventional mouse cursor icon of Microsoft Windows. The captured images (motion video) are transferred to the image processing portion which recognizes the mark and produces the pointing signal. In order to do the pointing job, Firstly, the user must move the pen camera onto the cursor icon of display so that the cursor icon can be captured by the pen camera. And if the user moves the pen camera by writing character or drawing polygon on the display then the position of the mark in the captured image moves from the center of the image to the boundary of the image and the movement (in other words, motion vector) of the mark in the captured image can be recognized by the image processing portion by comparing the previous frame image and current frame image. The image processing portion transfers the detected motion vector to the mark outputting portion and the the mark outputting portion produces the control signal to move back the mark (cursor icon) to the center of the captured image so that the mark follows the movement of pen camera. For example, If the pen camera inFIG. 1 is moved in x direction (dx) then the mark in the captured image moves in −x direction (−dx) as shown inFIG. 2 where the x direction is horizontal and the y direction is vertical as shown inFIG. 1 . Then the image processing portion produces signal so that the mark outputting portion can increase the x coordinate of the mark where the amount of the increment is proportional to the distance between the center of the captured image and the position of the mark in the captured image. In other words, the image processing portion finds the motion vector of the mark in the captured image and the mark outputting portion changes the coordinate of the mark in the negative direction of the found motion vector. In Microsoft Windows, such a moveing of cursor can be controlled by using the Windows API (application program interface) which can read and change the coordinate of mouse cursor. If the mark in the captured image is located in the center of the captured image then the motion vector is zero vector and there is no change of position of the mark.FIG. 3 shows the motion vector of mark from the dotted arrow in the previous frame to the solid arrow in the current frame. By recognizing the size and distortion of shape of mark, 3 dimensional pointing is also possible. For example, the smaller mark means the larger distance between pen camera and the display and the larger mark means the smaller distance between pen camera and the display. Such a size information of mark can be used as the another coordinate (z) of mouse cursor (x,y). The direction of the mark in the captured image also can be used as another coordinate (rotation angle r inFIG. 1 ). The viewing direction of pen camera can be detected and used as pointing signal by recognizing the distortion of the mark which contains feature points like vertex of rectangle and triangle. Such a distortion analysis and calculating the relative direction between camera and feature points is well known technology as the perspective n point problem in image processing technology and the detail description can be found in http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL COPIES/MARBLE/high/pia/solving.htm - If the mark is out of viewing direction of the pen camera then the image processing portion can not detect the mark from the captured image and the movement of the mark is stopped. In order to continue the pointing procedure, user must carry the pen camera to the mark and change the viewing direction of the pen camera so that the mark can be captured by the pen camera. By adding the reset button to the pen camera, such a carrying action can be removed. If the user presses the reset button then the mark changes its position. More specifically, the mark outputting portion sequentially changes the position of mark as shown in
FIG. 4 by the trigger signal of the reset button. The mark moves horizontally - from (0.0) to (5,0) and
- from (0,1) to (5,1) and
- from (0,2) to (5,2) and
- from (0,3) to (5,3) and
- from (0,4) to (5,4) and
- and finally from (0.5) to (5,5). In other words, the mark scans all the cells sequentially. If the mark image is captured and recognized by the image processing portion during the scanning, the scanning is stopped at that time and the pointing procedure is started. The 6×6 cells of the display in
FIG. 4 is an example and the real number of cells must be adjusted for a given display and camera. It is recommended to move the mark fast and use the fast camera so that the human eye can not recognize the scanning. - The above embodiment 1 is the pen camera which is used by touching the display. If the camera is far from the display then the captured mark is too small to be recognized. In such a case, it is recommended to use the auto focusing system of camera and telescope lens or zoom lens with camera. By using such a optical apparatus, it is possible to use the pointing device of present invention as the electronic pen for tablet PC and remote controller for digital TV.
- The mark in the above embodiment 1 is fixed pattern but in this embodiment the mark is the whole image of display and the distance between the camera and the display must be adjusted so that the whole image of display can be captured. The mark outputting portion includes the image transferring portion which transfers the image of display to the image processing portion. The image processing portion finds the display region from the captured image by comparing the sub regions of the captured image with the transferred image of display (It is known as the model based vision). In Microsoft Windows XP, pressing the Print Screen Sys Rq key of computer keyboard captures the image of display and stores the image into the clipboard. Such an image transferring can be done by software by emulating the pressing the key or by using device driver. The image transferring portion can also be implemented by hardware. The image processing portion finds feature points from the found display and the relative distance and the direction between camera and the display can be obtained by using the formula of the perspective n point problem and such a distance and the direction information can be use to produce the pointing signal. Korean patent 10-0532525-0000 is the 3 dimensional pointing device by analyzing the feature points of rectangle. The pointing device of present invention selects the feature points from the image of display in real time and the feature points is not fixed for each frame. The model based vision is the technology to find the correspondence between the known model (transferred image of display) and given image (captured image by camera) and is published in chapter 18 of Computer vision a modern approach by David A. Forsyth and Jean Ponce (ISBN:0-13-085198-1).
- If the background of the display is simple (for example, the beam projected onto the white wall), then the detecting the display region from the captured image is simple procedure but if the background of display is not simple then the detecting the region of display from the captured image is not so simple. In order to easily detect the display region from the captured image, the flicker generating portion can be added to the mark outputting portion of embodiment 3 and the difference image calculating portion can be added to the image processing portion of embodiment 3. More specifically, the mark outputting portion outputs the blank image for every even frames (0, 2, 4, . . . ) and outputs normal image for every odd frames (1, 3, 5, . . . ). (such odd and even frame is an example and in real implementation it is possible to use 0, 4, 8, . . . as even frames and 1, 2, 3, 5, 6, 7, . . . as odd frames, in other word the frame frequency can be adjusted in real implementation.) The blank image means the image whose all the pixels have the same brightness and color. It is recommended to keep the frame rate (number of frame per second) of display large so that the human eye can not recognize the flicker and to keep the frame rate of camera also large so that the camera can capture the even and odd frame of display. The image processing portion obtains the difference image between the captured image of previous frame and the captured image of current frame. The difference image is well known concept in image processing technology whose pixel value is defined as the difference of two corresponding pixels of two images. (The two corresponding pixels of two images means that the (x,y) positions of two pixels are the same.) The non zero pixels of the difference image calculated by the image processing portion corresponds to the flickering display region and the zero pixels of the difference image corresponds to the background of display (non flickering region). In other words, the flickering display can be detected by calculating the difference image and selecting non zero pixels from the difference image. In real world, edge lines of background of display may corresponds to the non zero pixels if the camera is not fixed but such non zero pixels can be minimized by using high speed flickering frequency and high speed camera. The regions of non zero pixels of difference image are the candidates for the flickering display region in captured image and the more exact region of display can be determined by the model based vision than embodiment 3. The found region of display can be compared with the transferred image of display and pointing signal can be generated like the embodiment 3.
- The blank image for every even frames (0, 2, 4, . . . ) of embodiment 4 can be replaced by recognizable pattern (mark) and the image processing portion can recognize the pattern by analyzing the captured image of only even frame.
FIG. 5 shows the example of the pattern (mark) which contains the opened rectangle and + at the center of the rectangle. The + mark represents the center of mark and the rectangle can be use for 3 dimensional pointing. There is no limitation to the size, shape and color of pattern. For example, polygon, line, bar code, alphabet and number can be such a pattern. Recognizing character is well known technology as OCR (optical character recognition). - The recognizable pattern of the embodiment 5 can be splitted into image of pattern and negative image of the said image of pattern. If the mark outputting portion outputs the pattern image (for 0, 3, 6, . . . frames), the negative pattern image (for 1, 4, 7, . . . frames) and the normal image (for 2, 5, 8, . . . frames) sequentially and repeatedly at enough high frequency, then human eye can not recognize the pattern image but can recognize only the normal image because the pattern and its negative pattern are time-averaged out. But high speed camera can capture the pattern image and can be recognized by image processing portion.
FIG. 5 andFIG. 6 are the example of the pattern image and its negative image. - The mark image of the embodiment 4˜6 can be 2 dimensional array of patterns where the each pattern represents the 2 dimensional position (x,y) of display. The pattern can be 2 dimensional bar code or number.
FIG. 7 shows the 2 dimensional array of cells where each cell contains pattern. The reset button of embodiment 1 can be removed by adopting such cells of pattern as the mark image with pen type camera. The captured image of pattern in cell can be recognized by image processing portion and can be translated into 2 dimensional position (x,y) which is corresponding to the pointing signal. There is similar invention PCT/US1999/030507 which presents the mouse for outputting absolute coordinate with special pad where the pad contains patterns and can be recognized by the camera in mouse. There is no difference between current embodiment and the embodiment 5˜6 except the mark. There is no limit on pattern in cell. The pattern can be alphabet, number, 2 dimensional bar code. By including rectangle into pattern and recognizing it, it is possible to generate 3 dimensional pointing signal by the formula of perspective n point problem.
Claims (8)
1-10. (canceled)
11. A pointing device comprising: a mark outputting portion for displaying a mark on a monitor; a camera portion for capturing an image of the mark; and an image processing portion for recognizing the mark from the captured image and generating a pointing signal based on the recognized mark,
wherein the mark is a whole image which is displayed in a monitor, the mark outputting portion includes an image transferring portion which transfers the whole image to the image processing portion, and
the image processing portion detects a region corresponding to the whole image from the captured image, and generates the pointing signal based on a position of the detected region.
12. The pointing device of claim 11 , wherein the image processing portion detects a rotation angle of the camera portion by detecting a rotation of the mark included in one captured image, detects a direction of the camera portion towards the monitor by detecting a distortion of the mark and outputs the pointing signal based on the result of detecting the rotation angle and the direction of the camera portion.
13. A pointing device comprising: a mark outputting portion for displaying a mark on a monitor; a camera portion for capturing an image of the mark; and an image processing portion for recognizing the mark from the captured image and generating a pointing signal based on the recognized mark,
wherein the mark outputting portion outputs the mark, negative image or complementary color image of the mark and normal image sequentially and repeatedly, and the image processing portion detects the mark from the captured image and generates the pointing signal determined by a position of the mark in the captured image.
14. A pointing device comprising: a mark outputting portion for displaying a mark on a monitor; a camera portion for capturing an image of the mark; and an image processing portion for recognizing the mark from the captured image and generating a pointing signal based on the recognized mark,
wherein the mark outputting portion outputs blank image and normal image sequentially and repeatedly, and
the image processing portion detects the whole image from the captured image by detecting non zero pixels from the difference image whose pixel value is a difference between the captured image of a current frame and the captured image of a previous frame.
15. A pointing device comprising: a mark outputting portion for displaying a mark on a monitor; a camera portion for capturing an image of the mark; and an image processing portion for recognizing the mark from the captured image and generating a pointing signal based on the recognized mark,
wherein the mark outputting output the mark and normal image sequentially and repeatedly, and
the image processing portion detects the mark from the captured image and generates the pointing signal determined by a position of the mark in the captured image.
16. A computer-readable recording medium having stored thereon a computer program for executing a pointing method, wherein the pointing method comprising:
outputting a mark and normal image sequentially and repeatedly;
capturing an image of the mark;
recognizing the mark from the captured image and generating a pointing signal based on the recognized mark.
17. A computer-readable recording medium having stored thereon a computer program for executing a pointing method, wherein the pointing method comprising:
receiving a captured image of a mark displayed in a monitor;
recognizing the mark from the captured image; and
generating a pointing signal based on the recognized mark;
wherein the mark is a whole image which is displayed in a monitor, and
the generating of the pointing signal based on the recognized mark comprises:
receiving the whole image;
detecting a region corresponding to the whole image from the captured image; and
generating the pointing signal based on a position of the detected region.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/795,749 US20130187854A1 (en) | 2007-05-26 | 2013-03-12 | Pointing Device Using Camera and Outputting Mark |
US14/574,879 US9785253B2 (en) | 2007-05-26 | 2014-12-18 | Pointing device using camera and outputting mark |
Applications Claiming Priority (12)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR20070051168 | 2007-05-26 | ||
KR10-2007-0051168 | 2007-05-26 | ||
KR20070080925 | 2007-08-10 | ||
KR10-2007-0080925 | 2007-08-10 | ||
KR20070095580 | 2007-09-19 | ||
KR10-2007-0095580 | 2007-09-19 | ||
KR20070098528 | 2007-09-30 | ||
KR10-2007-0098528 | 2007-09-30 | ||
KR10-2008-0041623 | 2008-05-05 | ||
KR20080041623A KR100936816B1 (en) | 2007-05-26 | 2008-05-05 | Pointing device using camera and outputting mark |
US12/526,527 US20100103099A1 (en) | 2007-05-26 | 2008-05-25 | Pointing device using camera and outputting mark |
US13/795,749 US20130187854A1 (en) | 2007-05-26 | 2013-03-12 | Pointing Device Using Camera and Outputting Mark |
Related Parent Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/526,527 Division US20100103099A1 (en) | 2007-05-26 | 2008-05-25 | Pointing device using camera and outputting mark |
PCT/KR2008/002913 Division WO2008147083A2 (en) | 2007-05-26 | 2008-05-25 | Pointing device using camera and outputting mark |
US12526527 Division | 2009-08-10 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/574,879 Division US9785253B2 (en) | 2007-05-26 | 2014-12-18 | Pointing device using camera and outputting mark |
Publications (1)
Publication Number | Publication Date |
---|---|
US20130187854A1 true US20130187854A1 (en) | 2013-07-25 |
Family
ID=56291304
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/795,749 Abandoned US20130187854A1 (en) | 2007-05-26 | 2013-03-12 | Pointing Device Using Camera and Outputting Mark |
US14/574,879 Expired - Fee Related US9785253B2 (en) | 2007-05-26 | 2014-12-18 | Pointing device using camera and outputting mark |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/574,879 Expired - Fee Related US9785253B2 (en) | 2007-05-26 | 2014-12-18 | Pointing device using camera and outputting mark |
Country Status (1)
Country | Link |
---|---|
US (2) | US20130187854A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130314489A1 (en) * | 2010-10-04 | 2013-11-28 | Sony Corporation | Information processing apparatus, information processing system and information processing method |
US9648301B2 (en) | 2011-09-30 | 2017-05-09 | Moon Key Lee | Image processing system based on stereo image |
CN110168540A (en) * | 2017-01-09 | 2019-08-23 | 微软技术许可有限责任公司 | Capture annotation on an electronic display |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7161596B2 (en) * | 2001-12-21 | 2007-01-09 | British Telecommunications Public Limited Company | Display location calculation means |
US20070236451A1 (en) * | 2006-04-07 | 2007-10-11 | Microsoft Corporation | Camera and Acceleration Based Interface for Presentations |
US20080062124A1 (en) * | 2006-09-13 | 2008-03-13 | Electronics And Telecommunications Research Institute | Mouse interface apparatus using camera, system and method using the same, and computer recordable medium for implementing the same |
Family Cites Families (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07121293A (en) | 1993-10-26 | 1995-05-12 | Nippon Telegr & Teleph Corp <Ntt> | Remote controller accessing display screen |
JP3277052B2 (en) | 1993-11-19 | 2002-04-22 | シャープ株式会社 | Coordinate input device and coordinate input method |
JPH07234755A (en) | 1994-02-25 | 1995-09-05 | Hitachi Ltd | Coordinate input means and information processor |
JP2897954B2 (en) | 1997-06-11 | 1999-05-31 | 義人 安久津 | Controller device for towing machine mounted on tractor |
JPH1185395A (en) | 1997-09-08 | 1999-03-30 | Sharp Corp | Liquid crystal projector device with pointing function |
US20010010514A1 (en) | 1999-09-07 | 2001-08-02 | Yukinobu Ishino | Position detector and attitude detector |
JP3554517B2 (en) | 1999-12-06 | 2004-08-18 | 株式会社ナムコ | Game device, position detection device, and information storage medium |
JP2001325069A (en) | 2000-03-07 | 2001-11-22 | Nikon Gijutsu Kobo:Kk | Device and method for detecting position |
FR2812955A1 (en) | 2000-08-11 | 2002-02-15 | Yves Jean Paul Guy Reza | Equipment for pointing and guiding a screen cursor from a distance, comprises personal computer, marking transmitters located on or near screen and video camera supplying pointing data to computer |
JP2002222043A (en) | 2001-01-29 | 2002-08-09 | Nissan Motor Co Ltd | Cursor controller |
US6731330B2 (en) | 2001-01-30 | 2004-05-04 | Hewlett-Packard Development Company, L.P. | Method for robust determination of visible points of a controllable display within a camera view |
JP4055388B2 (en) | 2001-10-12 | 2008-03-05 | ソニー株式会社 | Information processing apparatus, information processing system, and program |
JP2003280813A (en) | 2002-03-25 | 2003-10-02 | Ejikun Giken:Kk | Pointing device, pointer controller, pointer control method and recording medium with the method recorded thereon |
KR100532525B1 (en) | 2002-05-07 | 2005-11-30 | 이문기 | 3 dimensional pointing apparatus using camera |
JP2004171414A (en) | 2002-11-21 | 2004-06-17 | Nippon Telegr & Teleph Corp <Ntt> | Device, method, and program for inputting three-dimensional position and attitude, and medium recording the program |
US7256772B2 (en) | 2003-04-08 | 2007-08-14 | Smart Technologies, Inc. | Auto-aligning touch system and method |
JP2005052306A (en) | 2003-08-01 | 2005-03-03 | Sony Corp | Position detection system |
JP2005258694A (en) | 2004-03-10 | 2005-09-22 | Asahi Kasei Microsystems Kk | Pointing device |
US20060197742A1 (en) | 2005-03-04 | 2006-09-07 | Gray Robert H Iii | Computer pointing input device |
JP2007114820A (en) | 2005-10-18 | 2007-05-10 | Sharp Corp | Portable pointer device and display system |
KR100708875B1 (en) | 2006-02-10 | 2007-04-17 | (주)소프트가족 | Apparatus and method for calculating position on a display pointed by a pointer |
US7553229B2 (en) * | 2006-03-21 | 2009-06-30 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Active referencing method with interleaved crosshair navigation frames |
JP4725383B2 (en) | 2006-03-24 | 2011-07-13 | カシオ計算機株式会社 | Pointing device, external information processing device, pointing position specifying device, and pointing position specifying method |
KR101040700B1 (en) | 2006-11-16 | 2011-06-10 | 주식회사 엘지화학 | Purification method of terephthal aldehyde |
KR100936816B1 (en) | 2007-05-26 | 2010-01-14 | 이문기 | Pointing device using camera and outputting mark |
-
2013
- 2013-03-12 US US13/795,749 patent/US20130187854A1/en not_active Abandoned
-
2014
- 2014-12-18 US US14/574,879 patent/US9785253B2/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7161596B2 (en) * | 2001-12-21 | 2007-01-09 | British Telecommunications Public Limited Company | Display location calculation means |
US20070236451A1 (en) * | 2006-04-07 | 2007-10-11 | Microsoft Corporation | Camera and Acceleration Based Interface for Presentations |
US20080062124A1 (en) * | 2006-09-13 | 2008-03-13 | Electronics And Telecommunications Research Institute | Mouse interface apparatus using camera, system and method using the same, and computer recordable medium for implementing the same |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130314489A1 (en) * | 2010-10-04 | 2013-11-28 | Sony Corporation | Information processing apparatus, information processing system and information processing method |
US9013535B2 (en) * | 2010-10-04 | 2015-04-21 | Sony Corporation | Information processing apparatus, information processing system and information processing method |
US9860484B2 (en) | 2010-10-04 | 2018-01-02 | Saturn Licensing Llc | Information processing apparatus, information processing system and information processing method |
US9648301B2 (en) | 2011-09-30 | 2017-05-09 | Moon Key Lee | Image processing system based on stereo image |
CN110168540A (en) * | 2017-01-09 | 2019-08-23 | 微软技术许可有限责任公司 | Capture annotation on an electronic display |
Also Published As
Publication number | Publication date |
---|---|
US20150103005A1 (en) | 2015-04-16 |
US9785253B2 (en) | 2017-10-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20100103099A1 (en) | Pointing device using camera and outputting mark | |
JP6539816B2 (en) | Multi-modal gesture based interactive system and method using one single sensing system | |
JP4915367B2 (en) | Display imaging apparatus and object detection method | |
TW201324235A (en) | Gesture input method and system | |
TWI581127B (en) | Input device and electrical device | |
US9785253B2 (en) | Pointing device using camera and outputting mark | |
JP2014029656A (en) | Image processor and image processing method | |
JP2016103137A (en) | User interface system, image processor and control program | |
TWI499938B (en) | Touch control system | |
TWI506479B (en) | Optical touch-control system | |
Yoo et al. | Symmetrisense: Enabling near-surface interactivity on glossy surfaces using a single commodity smartphone | |
JP2010272078A (en) | System, and control unit of electronic information board, and cursor control method | |
US20180059811A1 (en) | Display control device, display control method, and recording medium | |
KR20160055407A (en) | Holography touch method and Projector touch method | |
JP2005346453A (en) | Image display device | |
TWI444875B (en) | Multi-touch input apparatus and its interface method using data fusion of a single touch sensor pad and imaging sensor | |
JP6514889B2 (en) | INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING PROGRAM, INFORMATION PROCESSING SYSTEM, AND INFORMATION PROCESSING METHOD | |
Ramsundar | Interactive touch board using IR camera | |
JP2019060963A (en) | Display processing apparatus, display processing method, and program | |
Pullan et al. | High Resolution Touch Screen Module | |
KR20140071170A (en) | Projection system for supporting user interface based on hand movements and interface method thereof | |
Mustafa et al. | Multi Finger Gesture Recognition and Classification in Dynamic Environment under Varying Illumination upon Arbitrary Background | |
KR20160002620U (en) | Holography touch method and Projector touch method | |
KR20160080107A (en) | Holography touch method and Projector touch method | |
KR20160017020A (en) | Holography touch method and Projector touch method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |