WO2012137463A1 - Image processing apparatus, display control method and program - Google Patents
Image processing apparatus, display control method and program Download PDFInfo
- Publication number
- WO2012137463A1 WO2012137463A1 PCT/JP2012/002269 JP2012002269W WO2012137463A1 WO 2012137463 A1 WO2012137463 A1 WO 2012137463A1 JP 2012002269 W JP2012002269 W JP 2012002269W WO 2012137463 A1 WO2012137463 A1 WO 2012137463A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- display
- virtual object
- stability
- real object
- image
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
- H04N23/634—Warning indications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
Definitions
- the present disclosure relates to an image processing apparatus, a display control method, and a program.
- AR augmented reality
- presentations of information to the user are sometimes called “annotations,” and can be visualized using virtual objects of various forms such as text, icons, or animations.
- a virtual object is arranged to be aligned to a position of a real object (e.g., an object present in the real world which may be, for example, recognized in an image of the real world).
- a real object e.g., an object present in the real world which may be, for example, recognized in an image of the real world.
- Patent Literature 1 (“PTL 1”) has proposed a technique capable of suppressing degradation in accuracy of position alignment of a virtual object.
- a user can be prevented from being confused due to the unsteady display of a virtual object.
- Disclosed embodiments include an apparatus comprising a recognition unit configured to recognize real object in an image.
- the apparatus may further comprise a determining unit configured to determine a stability indicator indicating a stability of the recognition, and a display control unit configured to modify a display of a virtual object according to the stability indicator.
- Disclosed embodiments further include a method comprising recognizing real object in an image.
- the method may further comprise determining a stability indicator indicating a stability of the recognition, and modifying a display of a virtual object, according to the stability indicator.
- Disclosed embodiments further include a tangibly embodied non-transitory computer-readable medium storing instructions which, when executed by a processor, perform a method comprising recognizing real object in an image.
- the method may further comprise determining a stability indicator indicating a stability of the recognition, and modifying a display of a virtual object, according to the stability indicator.
- Fig. 1 is a diagram for explaining an overview of an image processing apparatus according to an embodiment
- Fig. 2 is a diagram for explaining an arrangement of a virtual object.
- Fig. 3 is a diagram for explaining a problem related to an arrangement of a virtual object;
- Fig. 4 is a block diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment;
- Fig. 5 is a block diagram illustrating an example of a configuration of logical functions of an image processing apparatus according to an embodiment;
- Fig. 6 is a diagram for explaining a state quantity of a recognized real object in an embodiment;
- Fig. 7 is a diagram for explaining a first example of a determination technique for determining stability of recognition;
- Fig. 1 is a diagram for explaining an overview of an image processing apparatus according to an embodiment
- Fig. 2 is a diagram for explaining an arrangement of a virtual object.
- Fig. 3 is a diagram for explaining a problem related to an arrangement of a virtual object;
- Fig. 8 is a diagram for explaining a second example of a determination technique for determining stability of recognition
- Fig. 9 is a diagram for explaining a third example of a determination technique for determining stability of recognition
- Fig. 10 is a diagram for explaining a fourth example of a determination technique for determining stability of recognition.
- Fig. 11 is a diagram for explaining a fifth example of a determination technique for determining stability of recognition;
- Fig. 12 is a diagram for explaining an example of control of a tracking pace of a virtual object for tracking a real object;
- Fig. 13 is a diagram for explaining control of sharpness of a virtual object;
- Fig. 14 is a diagram for explaining control of transparency of a virtual object;
- Fig. 15 is a diagram for explaining an example of a display prompting a user to take an action for increasing stability; and
- Fig. 16 is a flowchart illustrating an example of the flow of a display control process according to an embodiment.
- FIG. 1 is a diagram for explaining an overview of an embodiment of a technique disclosed in this disclosure. Referring to Fig. 1, an image processing apparatus 100 which a user carries in a real space 1 is shown.
- the image processing apparatus 100 is an apparatus that provides the user with an AR application.
- the image processing apparatus 100 may be, for example, a terminal apparatus such as a smartphone, a personal digital assistant (PDA), a game machine, or a portable music player or may be an information processing apparatus such as a personal computer (PC) or a workstation.
- a terminal apparatus such as a smartphone, a personal digital assistant (PDA), a game machine, or a portable music player
- PDA personal digital assistant
- PC personal computer
- the real space 1 is an example of a space where an AR application can be used.
- Real objects 12a, 12b, 12c, 12d, and 12e are present in the real space 1. It is to be understood that real space 1 and real objects 12a, 12b, 12c, 12d, and 12e are merely exemplary. Although each is endowed with forms and features for the purposes of illustration, such features are arbitrary and not meant to be limiting.
- the real object 12a is a vehicle.
- the real object 12b is a guide plate.
- the real object 12c is an advertising signboard.
- the real objects 12d and 12e are buildings.
- the image processing apparatus 100 acquires an image obtained by imaging a real space where various real objects can be present as an input image.
- the image processing apparatus 100 superimposes virtual objects for AR on the input image based on a recognition result of the real objects projected in the input image.
- Information presented to the user through the virtual objects may be arbitrary information such as navigation information, advertising information, store information, news, or weather forecast information.
- the AR application provided by the image processing apparatus 100 arranges a virtual object related to a real object in a real space in an image according to the position or the posture of the real object.
- Fig. 2 is a diagram for explaining an arrangement of a virtual object.
- An image Im01 shown on the left of Fig. 2 is an example of the input image acquired by the image processing apparatus 100.
- the real objects 12a and 12b are projected in the input image Im01.
- An output image Im02 generated such that virtual objects are superimposed on the input image Im01 is shown on the right of Fig. 2.
- a virtual object 13a is superimposed on the real object 12a
- virtual objects 13b and 14b are superimposed on the real object 12b.
- the virtual objects 13a and 13b function as indications (indicators) representing a result of recognizing a real object through the image processing apparatus 100.
- the virtual object 14b functions as information content representing information (advertising information of a product in the example of Fig. 2) related to the real object 12b.
- the virtual object may include both the indication and the information content.
- the virtual object may include one of the indication or the information content.
- the virtual object 13a is arranged according to a status of the real object 12a.
- the virtual objects 13b and 14b are arranged according to a status of the real object 12b. How accurately the virtual object is arranged depends on the accuracy of recognizing the status of the real object projected in the image.
- Fig. 3 is a diagram for explaining a problem related to an arrangement of a virtual object.
- an arrangement of the virtual object 13a matches a status of the real object 12a.
- arrangements of the virtual objects 13b and 14b may match a status of the real object 12b.
- an arrangement of the virtual object 13a has deviated from a status of the real object 12a.
- arrangements of virtual objects 13b and 14b are deviated from a status of the real object 12b.
- Such deviation may be caused, for example, by the degradation in the accuracy of recognizing the status of the real object.
- the degradation in the recognition accuracy may be suddenly caused by various events such as a partial deficit of a real object in an input image, blurring of an image, the presence of a plurality of objects having a similar external appearance, or a variation in a lighting condition. Unless an environment is particularly prepared, it may be difficult to completely exclude causes causing the degradation in the recognition accuracy.
- a sudden deviation in the arrangement of the virtual object illustrated in Fig. 3 may not only provide the user with an uncomfortable feeling but may also bring an undesired result such as the degradation in visibility of information represented by the virtual object or a bad impression of an advertising target product.
- tracking of a virtual object tracking a recognized real object may be delayed.
- the image processing apparatus 100 determines how stable the recognition accuracy is, and controls a display of a virtual object based on the determination result.
- Hardware Configuration Example of Image Processing Apparatus According to Embodiment 2-1.
- Hardware Configuration Fig. 4 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus 100 according to the present embodiment.
- the image processing apparatus 100 includes an imaging unit 102, a sensor unit 104, an input unit 106, a storage unit 108, a display unit 112, a communication unit 114, a bus 118, and a control unit 120.
- the imaging unit 102 may include a camera module for imaging an image.
- the imaging unit 102 may image a real space using an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and generates an imaged image.
- CMOS complementary metal oxide semiconductor
- the imaged image generated by the imaging unit 102 may configure each frame of a video input.
- the sensor unit 104 may include a group of motion sensors that measure a motion of the imaging unit 102.
- the motion sensor includes an acceleration sensor that measures an acceleration of the imaging unit 102, a gyro sensor that measures a gradient angle, and a geomagnetic sensor that measures a direction toward which the imaging unit 102 is directed.
- the sensor unit 104 may include a global positioning system (GPS) sensor that receives a GPS signal and measures the latitude, the longitude, and the altitude of an apparatus.
- GPS global positioning system
- the input unit 106 is an input device that allows the user to operate the image processing apparatus 100 or to input information to the image processing apparatus 100.
- the input unit 106 typically includes a touch sensor that detects a touch which the user makes on a screen of the display unit 112.
- the input unit 106 may include a pointing device such as a mouse or a touch pad, a gesture recognizing module that recognizes a user's gesture projected in an image, or a line-of-sight detecting module that includes a head mounted display (HMD) and detects a direction of a user's line of sight.
- the input unit 106 may include any other kind of input device such as a keyboard, a keypad, a button, or a switch.
- the storage unit 108 is configured with a storage medium such as a semiconductor memory or a hard disk, and stores a program and data used for performing processing by the image processing apparatus 100. Examples of data stored in the storage unit 108 include image data generated by the imaging unit 102 and sensor data generated by the sensor unit 104. In addition, for example, model data used when the image processing apparatus 100 recognizes a real object and object data for defining a virtual object may be included as data stored in the storage unit 108.
- the display unit 112 may include a display module configured with a liquid crystal display (LCD), an organic light-emitting diode (OLED), a cathode ray tube (CRT), or the like.
- the display unit 112 displays an image imaged by the imaging unit 102 or an image of an AR application realized by the control unit 120 on a screen.
- the display unit 112 may be a screen of the image processing apparatus 100 gripped by the user or an HMD of a through type or a non-through type mounted by the user.
- the communication unit 114 may include a communication interface that mediates communication between the image processing apparatus 100 and another apparatus.
- the communication unit 114 may support an arbitrary wireless communication protocol or an arbitrary wire-line communication protocol, and establishes a communication connection with another apparatus.
- the bus 118 connects the imaging unit 102, the sensor unit 104, the input unit 106, the storage unit 108, the display unit 112, the communication unit 114, and the control unit 120 to one another.
- the control unit 120 may include a processor such as a central processing unit (CPU) or a digital signal processor (DSP).
- the control unit 120 implements various functions of the image processing apparatus 100, which will be described later, by executing a program stored in the storage unit 108 or another storage medium.
- the control unit 120 may include or be connected to a display control unit for, among other things, controlling display unit 112.
- the image processing apparatus 100 includes an image acquiring unit 130, a model database (DB) 138, an image recognizing unit 140, a detecting unit 150, a determining unit 160, a data acquiring unit 170, an object DB 172, and a display control unit 190.
- DB model database
- the image processing apparatus 100 includes an image acquiring unit 130, a model database (DB) 138, an image recognizing unit 140, a detecting unit 150, a determining unit 160, a data acquiring unit 170, an object DB 172, and a display control unit 190.
- DB model database
- the image acquiring unit 130 acquires an image obtained by capturing a real space as an input image. For example, the image acquiring unit 130 may sequentially acquire each frame of a video input from the imaging unit 102 as the input image. The image acquiring unit 130 may acquire each frame of a video input, which has been imaged in the past, stored in the storage unit 108 as the input image. The image acquiring unit 130 outputs the acquired input image to the image recognizing unit 140.
- the image recognizing unit 140 may recognize a real object using data stored in the model DB 138.
- model data related to shapes or external appearances of real objects which the image processing apparatus 100 regards as recognition targets is accumulated in advance.
- the target recognized by the image processing apparatus 100 may be an arbitrary object present in a real space such as the real objects 12a to 12e illustrated in Fig. 1.
- Examples of the model data include data defining the shape of each real object, image data such as a predetermined symbol mark or a text label attached to each real object, and data of a feature quantity set extracted from a known image of each real object.
- the image recognizing unit 140 may recognize a real object projected in the input image input from the image acquiring unit 130 and recognizes a status of the real object projected in the input image.
- a status of the real object recognized by the image recognizing unit 140 may include at least one of the position and the posture of the real object.
- the image recognizing unit 140 recognizes the position, the posture, and the scale of the real object in the image.
- the image recognizing unit 140 may collate a set of feature points extracted from the input image with the shape of the real object which can be defined by the model data.
- the image recognizing unit 140 may collate the image data such as the symbol mark or the text label which can be defined by the model data with the input image.
- the image recognizing unit 140 may collate a known feature quantity of an image of the real object which can be defined by the model data with the feature quantity extracted from the input image. Even in any of the above cases, the image recognizing unit 140 can recognize that a real object whose collation score is higher than a predetermined threshold value is being projected in the input image at the position, the posture, and the scale corresponding to the collation score.
- Fig. 6 is a diagram for explaining a state quantity of a real object recognized by the image recognizing unit 140 in the present embodiment.
- a global coordinate system formed by an x axis and a y axis on a plane vertical to an optical axis direction of an imaging apparatus that has imaged the input image and a z axis parallel to the optical axis direction is shown.
- a position P of a real object 12 recognized by the image recognizing unit 140 is given as position coordinates of a reference point of the real object in the global coordinate system.
- a posture W of the real object 12 is given as a rotational angle of each axis periphery.
- a scale Sc of the real object 12 is given as a magnification for each axis direction.
- the position P, the posture W, and the scale Sc of each real object are described as discrete variables (state quantities).
- the position P, the posture W, and the scale Sc of each real object may be expressed, in an integrated manner, by, for example, one 4 by 4 homogeneous transformation matrix representing transformation between the global coordinate system and a model coordinate system of each real object.
- each state quantity that is, each of the position P, the posture W, and the scale Sc
- the global coordinate system may be a coordinate system representing relative coordinates using the position of the imaging apparatus as an original point.
- the global coordinate system may be a coordinate system representing absolute coordinates fixedly defined in a real space.
- the image recognizing unit 140 outputs an identifier, the position P, the posture W, and the scale Sc of each real object projected in the input image to the determining unit 160, the data acquiring unit 170 and the display control unit 190.
- the detecting unit 150 detects a user input, and outputs the user input to the display control unit 190.
- Examples of the user input detected by the detecting unit 150 include a touch input to a touch sensor of the input unit 106, pressing of a button, a motion of an apparatus measured by a sensor group of the sensor unit 104 (for example, a gradient, shaking, or the like), a user's gesture, and a user's voice.
- the determining unit 160 may determine stability of recognition which the image recognizing unit 140 makes on the status of the real object projected in the input image.
- the stability determined by the determining unit 160 may be represented by a continuous value or a discrete value (for example, a stepwise value having three or more steps or two values of "high” and "low”). Five examples of a method of determining stability by the determining unit 160 will be described below with reference to Figs. 7 and 11.
- the determining unit 160 may determine the stability based on a temporal change in the state quantity of the real object recognized by the image recognizing unit 140.
- the temporal change in the state quantity of the real object can be detected by comparing a frequency characteristic of the state quantity of the real object, a differential (or a second order differential) of the state quantity, or a history of the state quantity within a predetermined time period with a current value of the state quantity.
- Fig. 7 illustrates an example of the temporal change in a state quantity of the real object.
- a horizontal axis denotes a time t
- a vertical axis denotes a state quantity X and a first order differential (dX/dt) thereof.
- the state quantity may be any one of the position P, the posture W, and the scale Sc on a certain real object or a combination of two or more thereof.
- the state quantity X gently changes during a time period between a time T0 and a time T1.
- a frequency component of the state quantity X is extracted by the Fourier transform
- a high frequency component of the state quantity X decreases.
- An absolute value of the first order differential of the state quantity X does not exceed a certain level (a level corresponding to a realistic limit at which the real object moves relative to the image apparatus). This is similarly applied to the second order differential of the state quantity X.
- a current value of the state quantity X does not significantly differ from a history of previous m samples of the state quantity X.
- the history of the state quantity X may be a simple average value of the previous m samples or a weighted average value of the previous m samples (e.g., the more recent the sample is, the larger a set weight).
- the state quantity X drastically changes during a time period between the time T1 and a time T2.
- the frequency component of the state quantity X is extracted, the high frequency component of the state quantity X further increases.
- the first order differential of the state quantity X oscillates around zero.
- the absolute value of the first order differential is higher than the predetermined level.
- a current value of the state quantity X may significantly differ from the history of the previous m samples of the state quantity X.
- the determining unit 160 evaluates a temporal change in the state quantity by comparing one of the magnitude of the high frequency component of the state quantity of the real object, an absolute value (an integrated value) of the first order differential (or the second order differential) of the state quantity, and the history of the state quantity with the current value of the state quantity. Then, the determining unit 160 determines that the more drastic the evaluated temporal change is, the lower the stability of recognition is.
- a temporal change in the state quantity by comparing one of the magnitude of the high frequency component of the state quantity of the real object, an absolute value (an integrated value) of the first order differential (or the second order differential) of the state quantity, and the history of the state quantity with the current value of the state quantity.
- a recognition result obtained by the image recognizing unit 140 during a time period between a time T1 and a time T2, represents that a real object 12 is moving and the posture of the real object 12 is rotating. Movement of the real object 12 during the time period between the time T1 and the time T2 may be represented by a difference dP in the position P, and rotation may be represented by a difference dW in the posture W.
- the determining unit 160 can determine that the stability of recognition made by the image recognizing unit 140 is decreasing.
- the determining unit 160 determines the stability based on consistency between the change in the state quantity of the real object and a motion of the imaging unit 102 measured by a motion sensor.
- the user is moving the image processing apparatus 100 so that an angle of view of the imaging unit 102 can change during a time period between a time T1 and a time T2.
- the real object 12b projected on the lower right of an input image Im11-1 is being moved to the upper left of an input image Im11-2.
- a change in an angle of view of the imaging unit 102 can be recognized based on sensor data from a motion sensor (for example, an acceleration sensor, a gyro sensor, or a geomagnetic sensor) of the sensor unit 104.
- a motion sensor for example, an acceleration sensor, a gyro sensor, or a geomagnetic sensor
- the determining unit 160 can determine that the stability of recognition made by the image recognizing unit 140 is decreasing.
- the determining unit 160 determines the stability based on consistency between a change in a state quantity of a real object recognized by the image recognizing unit 140 and a direction of a motion of a real space shown in a plurality of input images. For example, a direction of a motion of a real space shown in a plurality of input images may be recognized based on an optical flow obtained from a plurality of input images.
- an optical flow F from an input image Im12-1 imaged at a time T1 to an input image Im12-2 imaged at a time T2 is shown.
- the optical flow F represents a motion of a real object which is moved toward the upper left of the image.
- a recognition result obtained by the image recognizing unit 140 represents that the real object 12b is rotating while moving upward.
- the determining unit 160 can determine that the stability of recognition made by the image recognizing unit 140 is decreasing.
- the stability of the real object may be calculated as the continuous value, based on a correlation (for example, a scalar product) between a motion vector of the real object predicted from the optical flow around the real object 12b and a motion vector of the real object recognized as a result of recognizing the real object (for example, the larger the scalar product is, the more stable the recognition is).
- a correlation for example, a scalar product
- a probability that the accuracy of recognition will decrease is high.
- the determining unit 160 determines that the stability of recognition is lower than when the whole real object is projected in the input image. For example, a part of a recognized real object may not be projected in an input image when a part of the real object is positioned outside an angle of view, or when a part of the real object is hidden by another real object (in case of so-called occlusion).
- the real objects 12a and 12b are projected in an input image Im13.
- a part of the real object 12a is hidden by the real object 12b.
- a part of the real object 12b is out of an angle of view.
- a plurality of triangular symbols illustrated in Fig. 11 represent the positions of feature points which can be used when the image recognizing unit 140 recognizes the real object 12a.
- the determining unit 160 calculates a ratio of the number of feature points of the real object 12a recognized by the image recognizing unit 140 to the total number of feature points thereof. When the calculated ratio is lower than a predetermined threshold value, the determining unit 160 can determine that a part of the real object 12 is not projected in the input image.
- the determining unit 160 can easily determine that there is a high possibility that a part of the real object is not projected in the input image.
- a center Pa of the real object 12a is located inside a stable detection region R
- a center Pb of the real object 12b is located outside the stable detection region R.
- the stable detection region R can be set to the center of the input image so that a boundary thereof can have a predetermined offset from the edge of the input image.
- the determining unit 160 determines the stability of recognition made by the image recognizing unit 140 using the above described techniques, and outputs the stability of each real object to the display control unit 190.
- the image processing apparatus 100 may allow the determining unit 160 to select two or more of the above described techniques, and may adaptively change a technique to be used by the determining unit 160 according to a condition. For example, when stability is determined on a dynamic real object, the first technique may be selected, whereas when stability is determined on a stationary real object, any one of the second to fourth techniques may be selected. Further, the fifth technique may be selected in a scene in which a small processing cost is desirable.
- the data acquiring unit 170 acquires data related to the virtual object to be superimposed on the input image through the display control unit 190.
- the data acquired by the data acquiring unit 170 includes object data defining the virtual object. Examples of the object data include the type of virtual object, an identifier of an associated real object, a relative display position from the real object, and data defining the content of the information content.
- the data acquiring unit 170 may acquire object data which is previously stored in the object DB 172.
- the data acquiring unit 170 may acquire newest object data from a data server installed in the real space via the communication unit 114.
- the object data provided from the data server may be data which differs according to the position of the image processing apparatus 100 measured by the sensor unit 104.
- the data acquiring unit 170 outputs the object data to the display control unit 190.
- the display control unit 190 may control a display of the virtual object by the AR application provided by the image processing apparatus 100. More specifically, for example, the object data of the virtual object associated with the real object recognized as being projected in the input image by the image recognizing unit 140 is input from the data acquiring unit 170 to the display control unit 190. The display control unit 190 generates an object image of the virtual object based on the object data. The object image generated by the display control unit 190 is typically an image obtained by projecting the virtual object virtually arranged in a three-dimensional real space on an imaging surface of the image processing apparatus 100. Then, the display control unit 190 outputs the generated output image to the display unit 112. As a result, an image of the AR application is presented to the user.
- An arrangement of a virtual object is decided according to a status of an associated real object. That is, the display control unit 190 causes the virtual object to be displayed on a display device so that the virtual object can track the position, the posture, and the scale of each real object.
- the display control unit 190 changes a display of the virtual object associated with each recognized real object according to the stability of recognition made by the image recognizing unit 140.
- the display control unit 190 changes a tracking pace of the virtual object for tracking each real object according to the stability.
- the display control unit 190 increases the tracking pace of the virtual object for tracking each real object when the stability is high, but decreases the tracking pace of the virtual object for tracking each real object when the stability is low.
- the tracking pace may be represented by an interpolation coefficient k in Formula (1):
- Fig. 12 is a diagram for explaining an example of control of a tracking pace of a virtual object for tracking a real object.
- the display control unit 190 may change a display attribute of the virtual object according to the stability instead of (or while) changing the tracking pace of the virtual object for tracking each real object according to the stability.
- the display attribute of the virtual object may include a static attribute and a dynamic attribute related to a display mode of the virtual object. Examples of the static attribute related to a display mode of the virtual object include transparency, sharpness, color, scale, shape, the number, and an afterimage pattern of the virtual object. Examples of a dynamic attribute related to a display mode of the virtual object include a blinking pattern of the virtual object, a blinking cycle, and the type of animation.
- sharpness of the virtual object 13b is set to high in an input image Im22-2a whose stability of recognition of the real object is determined to be high.
- the contour of the virtual object 13b of the input image Im22-2a is clearly displayed.
- sharpness of the virtual object 13b is set to low in an input image Im22-2b whose stability of recognition of the real object is determined to be low.
- the contour of the virtual object 13b of the input image Im22-2b is dimly displayed.
- transparency of the virtual object 13b is set to low in an input image Im23-2a whose stability of recognition of the real object is determined to be high.
- the contour of the virtual object 13b of the input image Im23-2a is not transparent.
- transparency of the virtual object 13b is set to high in an input image Im23-2b whose stability of recognition of the real object is determined to be low.
- the contour of the virtual object 13b of the input image Im23-2b is transparent.
- the transparency of the virtual object 13b which functions as the indication for representing a recognition result of the real object, changes according to the stability of recognition, whereas the transparency of the virtual object 14b, which functions as the information content, is maintained constant.
- the display control unit 190 may change a display of only one of the indication and the information content. By changing a display of only the indication, when the stability of recognition of the real object has decreased, it is possible to cause the user to understand the degradation in the stability without decreasing visibility of the information content. On the other hand, by changing a display of only the information content, when the stability of recognition of the real object has decreased, it may be possible to prevent an impression of the information content from getting worse while informing the user of how the real object has been recognized.
- the display control unit 190 may further cause information prompting the user to take an action for increasing the stability to be displayed on the display device.
- the information to display may be a message (see a message 15 of an image Im30 of Fig. 15) prompting movement of an apparatus so that the real object shown outside an angle of view can be located within an angle of view. The user who has referred to the information takes an action for increasing the stability, and thus the unsteady display of the virtual object can be avoided.
- the display control unit 190 may cause the display device to perform a display of the virtual object under the assumption that the stability is high, regardless of the stability of recognition determined by the determining unit 160.
- the predetermined user input include an input to a touch sensor, an input sensed by a motion sensor, pressing down of a button provided on the image processing apparatus 100, and an arbitrary user input such as the user's gesture or voice.
- Flow of Display Control Process is a flowchart illustrating an example of the flow of a display control process performed by the image processing apparatus 100 according to the present embodiment.
- the image acquiring unit 130 acquires an image projected in a real space as an input image (step S102).
- the image recognizing unit 140 may recognize a real object projected in each input image input from the image acquiring unit 130 and recognizes a status of each real object projected in the input image (step S104).
- the status of the recognized real object may include the position, the posture, and the scale of the real object.
- the determining unit 160 determines whether or not there is a real object recognized as being projected in the input image by the image recognizing unit 140 (step S106).
- the process returns to step S102.
- the process proceeds to step S108.
- the determining unit 160 may determine a stability of recognition which the image recognizing unit 140 has made on the status of the real object projected in the input image.
- the data acquiring unit 170 acquires object data of a virtual object related to each real object recognized by the image recognizing unit 140 (step S110).
- the display control unit 190 decides a display attribute of the virtual object related to each recognized real object according to the stability of recognition determined by the determining unit 160 (step S112).
- the display attribute may be the interpolation coefficient k, or transparency, sharpness, a blinking pattern, a blinking cycle, or the scale of an afterimage of the virtual object.
- the display control unit 190 generates an image of the virtual object related to each real object according to the decided display attribute (step S114). Then, the display control unit 190 superimposes the generated image of the virtual object on the input image (step S116).
- a display of the virtual object related to the real object dynamically changes according to the stability of recognition made on the status of the real object projected in the image. As a result, it is possible to prevent the user from being confused due to the unsteady display of the virtual object caused by the degradation in the accuracy of recognition.
- an application which allows the virtual object to track the position or the posture of the real object, may control the tracking pace of the virtual object for tracking the real object according to the stability, for example.
- a tracking delay can be kept small while the accuracy of recognition is high, and sudden deviation in an arrangement of the virtual object when the accuracy of recognition has been lowered can be effectively reduced.
- a display attribute of the virtual object such as transparency, sharpness, a blinking pattern, a blinking cycle, or the scale of an afterimage of the virtual object, can be controlled according to the stability.
- the stability can be determined based on the temporal change in the state quantity of the real object. In this case, the stability can be accurately determined, regardless of whether a target real object is a dynamic real object or a stationary real object, without needing an auxiliary input by a motion sensor or the like. Further, the stability may be determined based on consistency between the recognized state quantities, consistency between the state quantity and a motion of an apparatus, or consistency between the state quantity and a direction of a motion of a real space shown in the input image. In this case, the stability can be determined without needing a calculation such as analysis or a differential of a frequency of the state quantity of the real object. Further, the stability can be easily determined according to whether or not the real object has a part which is not projected in the input image.
- the processing by the image processing apparatus 100 described in the present disclosure may be implemented using software, hardware, or a combination of software and hardware.
- a program configuring software is previously stored in a storage medium provided inside or outside each apparatus.
- each program is read to a random access memory (RAM) at the time of execution and executed by a processor such as a CPU.
- RAM random access memory
- an image processing apparatus that receives an input image from a terminal apparatus carried by a user may recognize the position or the posture of a real object projected in the input image and control a display of a virtual object related to the real object according to a stability of recognition.
- an image for displaying the virtual object may be transmitted from the image processing apparatus to the terminal apparatus and then displayed on a screen of the terminal apparatus.
- An image processing apparatus comprising: a recognizing unit configured to recognize a position or a posture of a real object projected in an image; and a display control unit configured to change a display of a virtual object related to the real object according to stability of recognition made by the recognizing unit.
- the display control unit displays the virtual object on a display device for the virtual object to track a position or a posture of the real object.
- the display control unit changes a tracking pace of the virtual object for the real object according to the stability.
- the image processing apparatus determines the stability based on a temporal change in a state quantity of the real object recognized by the recognizing unit, and the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
- the determining unit determines the stability based on consistency between a first state quantity and a second state quantity of the real object recognized by the recognizing unit, and each of the first state quantity and the second state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
- the image processing apparatus wherein, when the real object recognized by the recognizing unit includes a part which is not projected in the image, the determining unit determines that the stability is lower than when the whole real object is projected in the image.
- the image processing apparatus further comprising: an imaging unit configured to capture the image; and a motion sensor configured to measure a motion of the imaging unit, wherein the determining unit determines the stability based on consistency between a change in a state quantity of the real object recognized by the recognizing unit and a motion of the imaging unit measured by the motion sensor, and the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
- the image processing apparatus determines the stability based on consistency between a change in a state quantity of the real object recognized by the recognizing unit and a direction of a motion of a real space shown in a plurality of input images, and the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
- the display control unit displays information prompting a user to take an action for increasing the stability.
- the image processing apparatus according to any one of (1) to (12), wherein, when a predetermined user input is detected, the display control unit causes a display device to perform a display of the virtual object under the assumption that the stability is high regardless of the stability of recognition made by the recognizing unit.
- the virtual object is an indication representing that the real object has been recognized.
- the virtual object is information content related to the recognized real object.
- a display control unit causes an indication representing that the real object has been recognized and information content related to the recognized real object to be displayed on a display device as the virtual objects, and changes a display of either of the indication and the information content according to the stability.
- a display control method comprising: recognizing a position or a posture of a real object projected in an image; and changing a display of a virtual object related to the real object according to stability of the recognition.
- Image processing apparatus 102 Imaging unit 104 Sensor unit 140 Image recognizing unit 160 Determining unit 190 Display control unit
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- User Interface Of Digital Computer (AREA)
- Processing Or Creating Images (AREA)
- Studio Devices (AREA)
Abstract
Description
1. Overview
2. Hardware Configuration Example of Image Processing Apparatus According to Embodiment
3. Functional Configuration Example of Image Processing Apparatus According to Embodiment
3-1. Image Acquiring Unit
3-2. Image Recognizing Unit
3-3. Detecting Unit
3-4. Determining Unit
3-5. Data Acquiring Unit
3-6. Display Control Unit
4. Flow of Display Control Process According to Embodiment
5. Summary
Fig. 1 is a diagram for explaining an overview of an embodiment of a technique disclosed in this disclosure. Referring to Fig. 1, an
2-1. Hardware Configuration
Fig. 4 is a block diagram illustrating an example of a hardware configuration of the
The
The
The
The
The
The
The
The
Fig. 5 is a block diagram illustrating an example of a configuration of logical functions implemented by the
The
The image recognizing unit 140 (or "recognizing unit") may recognize a real object using data stored in the
The detecting
The determining
For example, when recognition made by the
When recognition made by the
In a third determination technique, the determining
In a fourth determination technique, the determining
For example, when a part of a real object is not projected in an input image, a probability that the accuracy of recognition will decrease is high. In this regard, in a fifth determination technique, when a part of a recognized real object is not projected in an input image, the determining
The
The
Fig. 16 is a flowchart illustrating an example of the flow of a display control process performed by the
The
(1)
An image processing apparatus, comprising:
a recognizing unit configured to recognize a position or a posture of a real object projected in an image; and
a display control unit configured to change a display of a virtual object related to the real object according to stability of recognition made by the recognizing unit.
(2)
The image processing apparatus according to (1), wherein the display control unit displays the virtual object on a display device for the virtual object to track a position or a posture of the real object.
(3)
The image processing apparatus according to (2), wherein the display control unit changes a tracking pace of the virtual object for the real object according to the stability.
(4)
The image processing apparatus according to (3), wherein the display control unit set the tracking pace of the virtual object for the real object faster when the stability is higher.
(5)
The image processing apparatus according to any one of (1) to (4), wherein the display control unit changes either or both of a static attribute and a dynamic attribute related to a display mode of the virtual object according to the stability.
(6)
The image processing apparatus according to any one of (1) to (5), further comprising a determining unit configured to determine the stability.
(7)
The image processing apparatus according to (6), wherein the determining unit determines the stability based on a temporal change in a state quantity of the real object recognized by the recognizing unit, and
the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
(8)
The image processing apparatus according to (6), wherein
the determining unit determines the stability based on consistency between a first state quantity and a second state quantity of the real object recognized by the recognizing unit, and
each of the first state quantity and the second state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
(9)
The image processing apparatus according to (6), wherein, when the real object recognized by the recognizing unit includes a part which is not projected in the image, the determining unit determines that the stability is lower than when the whole real object is projected in the image.
(10)
The image processing apparatus according to (6), further comprising:
an imaging unit configured to capture the image; and
a motion sensor configured to measure a motion of the imaging unit,
wherein the determining unit determines the stability based on consistency between a change in a state quantity of the real object recognized by the recognizing unit and a motion of the imaging unit measured by the motion sensor, and
the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
(11)
The image processing apparatus according to (6), wherein the determining unit determines the stability based on consistency between a change in a state quantity of the real object recognized by the recognizing unit and a direction of a motion of a real space shown in a plurality of input images, and
the state quantity of the real object relates to at least one of a position, a posture, and a scale of the real object.
(12)
The image processing apparatus according to any one of (1) to (11), wherein, when the stability is low, the display control unit displays information prompting a user to take an action for increasing the stability.
(13)
The image processing apparatus according to any one of (1) to (12), wherein, when a predetermined user input is detected, the display control unit causes a display device to perform a display of the virtual object under the assumption that the stability is high regardless of the stability of recognition made by the recognizing unit.
(14)
The image processing apparatus according to any one of (1) to (13), wherein the virtual object is an indication representing that the real object has been recognized.
(15)
The image processing apparatus according to any one of (1) to (13), wherein the virtual object is information content related to the recognized real object.
(16)
The image processing apparatus according to any one of (1) to (13), wherein the display control unit causes an indication representing that the real object has been recognized and information content related to the recognized real object to be displayed on a display device as the virtual objects, and changes a display of either of the indication and the information content according to the stability.
(17)
A display control method, comprising:
recognizing a position or a posture of a real object projected in an image; and
changing a display of a virtual object related to the real object according to stability of the recognition.
(18)
A program causing a computer that controls an image processing apparatus to function as:
a recognizing unit configured to recognize a position or a posture of a real object projected in an image; and
a display control unit configured to change a display of a virtual object related to the real object according to stability of recognition made by the recognizing unit.
102 Imaging unit
104 Sensor unit
140 Image recognizing unit
160 Determining unit
190 Display control unit
Claims (27)
- An apparatus comprising:
a recognition unit configured to recognize a real object in an image;
a determining unit configured to determine a stability indicator indicating a stability of the recognition; and
a display control unit configured to modify a display of a virtual object according to the stability indicator. - The apparatus of claim 1, wherein the display control unit is further configured to modify the display of the virtual object according to the stability indicator by modifying at least one attribute of the virtual object.
- The apparatus of claim 1, wherein the display control unit is further configured to modify the display the virtual object according to a tracked position or orientation of the real object.
- The apparatus of claim 3, wherein the tracked position is tracked at a tracking rate and the display control unit is further configured to choose the tracking rate according to the stability indicator.
- The apparatus of claim 4, further comprising choosing the tracking rate before displaying the virtual object.
- The apparatus of claim 4, further comprising choosing a higher tracking rate when the stability indicator increases and choosing a lower tracking rate when the stability indicator decreases.
- The apparatus of claim 6, wherein choosing a tracking rate comprises choosing a tracking rate such that the modified display of the virtual object moves in tandem with the image.
- The apparatus of claim 2, wherein the modified image attribute is at least one of a static attribute or a dynamic attribute.
- The apparatus of claim 2, wherein the modified attribute comprises an indication to the user of a change in the stability indicator.
- The apparatus of claim 8, wherein the static attribute comprises at least one of a transparency of the display of the virtual object, a sharpness of the display of the virtual object, a color of the display of the virtual object, a scale of the display of the virtual object, a shape of the display of the virtual object, or a pattern superimposed on the display of the virtual object.
- The apparatus of claim 8, wherein the dynamic attribute comprises at least one of an animation related to the display of the virtual object, a pattern displayed on the display of the virtual object, or a timing of display of a pattern displayed on the display of the virtual object.
- The apparatus of claim 1, wherein determining a stability indicator comprises comparing a current value of an aspect of a state quantity of the real object with at least one prior value of the aspect of the state quantity of the real object.
- The apparatus of claim 12, comprising a sensor; and wherein the determining unit is configured to determine a stability indicator by comparing a change in a sensor output with a change in the aspect of a state quantity of the real object.
- The apparatus of claim 13, wherein the sensor comprises a sensor having an output which indicates a movement of the apparatus.
- The apparatus of claim 12, wherein the state quantity is at least one of a position, an orientation, or a scale of the real object.
- The apparatus of claim 12, wherein the aspect of the state quantity is at least one of a high frequency component of the state quantity, an absolute value of the state quantity, or a differential of the state quantity.
- The apparatus of claim 12, wherein the determining unit is configured to determine a stability indicator by comparing the aspect of the state quantity with a direction of motion of the real object in the image.
- The apparatus of claim 1, wherein the determining unit is configured to determine a stability indicator by:
comparing a current value of a first aspect of a state quantity of the real object to a prior value of the first aspect of the state quantity; and
comparing a current value of a second aspect of the state quantity with a prior value of the second aspect of the state quantity. - The apparatus of claim 18, wherein the first and second aspects of the state quantity include a position of the real object and an orientation of the real object.
- The apparatus of claim 1, wherein a value of the stability indicator is an inverse function of a fraction of the real object shown in the image.
- The apparatus of claim 1, further comprising a control unit configured to alert a user when the stability indicator falls below a threshold.
- The apparatus of claim 1, wherein the display control unit comprises an input unit for receiving a user input to inhibit modification of the display of the virtual object attribute.
- The apparatus of claim 2, wherein the display control unit is configured to:
modify the display of the virtual object to include an indicator of the recognition of the real object; and
to superimpose an informational display of the virtual object onto at least one of the image or the modified display of the virtual object. - The apparatus of claim 1, wherein:
the least one attribute is a sharpness of the display of the virtual object; and
the display control unit is configured to increase the sharpness when the stability indicator indicates increased stability and to decrease the sharpness when the stability indicator indicates decreased stability. - The apparatus of claim 1, wherein:
the least one attribute is a transparency of the display of the virtual object; and
the display control unit is configured to decrease the transparency when the stability indicator indicates increased stability and increase the transparency when the stability indicator indicates decreased stability. - A method comprising:
recognizing a real object in an image;
determining a stability indicator indicating a stability of the recognition; and
modifying a display of a virtual object, according to the stability indicator. - A tangibly embodied non-transitory computer-readable medium storing instructions which, when executed by a processor, perform a method comprising:
recognizing a real object in an image;
determining a stability indicator indicating a stability of the recognition; and
modifying a display of a virtual object, according to the stability indicator.
Priority Applications (9)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP12767610.4A EP2695143B1 (en) | 2011-04-08 | 2012-04-02 | Image processing apparatus, display control method and program |
BR112013025416A BR112013025416A2 (en) | 2011-04-08 | 2012-04-02 | apparatus, method, and computer readable medium |
US14/008,789 US9269002B2 (en) | 2011-04-08 | 2012-04-02 | Image processing apparatus, display control method and program |
KR1020137019062A KR20140017527A (en) | 2011-04-08 | 2012-04-02 | Image processing apparatus, display control method and program |
RU2013143978/08A RU2013143978A (en) | 2011-04-08 | 2012-04-02 | IMAGE PROCESSING DEVICE, DISPLAY MANAGEMENT METHOD AND PROGRAM |
US15/015,926 US9582938B2 (en) | 2011-04-08 | 2016-02-04 | Image processing apparatus, display control method and program |
US15/378,444 US9836887B2 (en) | 2011-04-08 | 2016-12-14 | Image processing apparatus and method to control display of a virtual object |
US15/807,680 US10395429B2 (en) | 2011-04-08 | 2017-11-09 | Image processing apparatus and display control method |
US16/511,216 US10930083B2 (en) | 2011-04-08 | 2019-07-15 | Image processing apparatus and display control method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011-086684 | 2011-04-08 | ||
JP2011086684A JP5732988B2 (en) | 2011-04-08 | 2011-04-08 | Image processing apparatus, display control method, and program |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/008,789 A-371-Of-International US9269002B2 (en) | 2011-04-08 | 2012-04-02 | Image processing apparatus, display control method and program |
US15/015,926 Continuation US9582938B2 (en) | 2011-04-08 | 2016-02-04 | Image processing apparatus, display control method and program |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012137463A1 true WO2012137463A1 (en) | 2012-10-11 |
Family
ID=46968874
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/002269 WO2012137463A1 (en) | 2011-04-08 | 2012-04-02 | Image processing apparatus, display control method and program |
Country Status (9)
Country | Link |
---|---|
US (5) | US9269002B2 (en) |
EP (1) | EP2695143B1 (en) |
JP (1) | JP5732988B2 (en) |
KR (1) | KR20140017527A (en) |
CN (2) | CN202976122U (en) |
BR (1) | BR112013025416A2 (en) |
RU (1) | RU2013143978A (en) |
TW (1) | TW201243767A (en) |
WO (1) | WO2012137463A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104090973A (en) * | 2014-07-18 | 2014-10-08 | 百度在线网络技术(北京)有限公司 | Information presentation method and device |
US20150346816A1 (en) * | 2014-05-30 | 2015-12-03 | Moriahtown Co., Ltd. | Display device using wearable eyeglasses and method of operating the same |
US9436874B2 (en) | 2013-01-28 | 2016-09-06 | Huawei Device Co., Ltd. | Method for discovering augmented reality object, and terminal |
EP3070681A4 (en) * | 2013-11-13 | 2017-07-12 | Sony Corporation | Display control device, display control method and program |
Families Citing this family (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9953618B2 (en) | 2012-11-02 | 2018-04-24 | Qualcomm Incorporated | Using a plurality of sensors for mapping and localization |
US9753534B2 (en) * | 2012-11-09 | 2017-09-05 | Sony Corporation | Information processing apparatus, information processing method, and computer-readable recording medium |
CN104937641A (en) | 2013-02-01 | 2015-09-23 | 索尼公司 | Information processing device, terminal device, information processing method, and programme |
US9927949B2 (en) | 2013-05-09 | 2018-03-27 | Amazon Technologies, Inc. | Recognition interfaces for computing devices |
JP6369461B2 (en) * | 2013-05-27 | 2018-08-08 | ソニー株式会社 | Image processing apparatus, image processing method, and program |
CN103455969B (en) | 2013-08-28 | 2019-06-04 | 腾讯科技(成都)有限公司 | The method and device of image procossing |
US9569892B2 (en) * | 2013-09-26 | 2017-02-14 | Qualcomm Incorporated | Image capture input and projection output |
US10671766B2 (en) * | 2013-10-21 | 2020-06-02 | Autodesk, Inc. | Evaluating the stability of three-dimensional models |
TWI529663B (en) * | 2013-12-10 | 2016-04-11 | 財團法人金屬工業研究發展中心 | Virtual image orientation method and apparatus thereof |
US9652893B2 (en) * | 2014-04-29 | 2017-05-16 | Microsoft Technology Licensing, Llc | Stabilization plane determination based on gaze location |
JP2016018367A (en) * | 2014-07-08 | 2016-02-01 | 沖電気工業株式会社 | Information processing apparatus, information processing method, and program |
US10943395B1 (en) * | 2014-10-03 | 2021-03-09 | Virtex Apps, Llc | Dynamic integration of a virtual environment with a physical environment |
US20180176483A1 (en) * | 2014-12-29 | 2018-06-21 | Metaio Gmbh | Method and sytem for generating at least one image of a real environment |
EP3113159A1 (en) * | 2015-06-30 | 2017-01-04 | Thomson Licensing | Method and device for processing a part of an immersive video content according to the position of reference parts |
JP6576177B2 (en) * | 2015-09-09 | 2019-09-18 | キヤノン株式会社 | Information processing apparatus, information processing apparatus control method, and program |
CN108027652B (en) | 2015-09-16 | 2021-06-22 | 索尼公司 | Information processing apparatus, information processing method, and recording medium |
TWI574223B (en) * | 2015-10-26 | 2017-03-11 | 行政院原子能委員會核能研究所 | Navigation system using augmented reality technology |
JP6723743B2 (en) * | 2015-12-28 | 2020-07-15 | キヤノン株式会社 | Information processing apparatus, information processing method, and program |
WO2017117446A1 (en) * | 2015-12-30 | 2017-07-06 | Daqri, Llc | 3d video reconstruction system |
JP2017123050A (en) * | 2016-01-07 | 2017-07-13 | ソニー株式会社 | Information processor, information processing method, program, and server |
KR102649197B1 (en) * | 2016-07-29 | 2024-03-20 | 삼성전자주식회사 | Electronic apparatus for displaying graphic object and computer readable recording medium |
US11563895B2 (en) | 2016-12-21 | 2023-01-24 | Motorola Solutions, Inc. | System and method for displaying objects of interest at an incident scene |
CN110199324B (en) * | 2017-01-31 | 2023-12-29 | 株式会社和冠 | Display device and control method thereof |
JP6406393B2 (en) * | 2017-05-24 | 2018-10-17 | カシオ計算機株式会社 | Display control apparatus, display control method, and program |
WO2019087513A1 (en) | 2017-10-31 | 2019-05-09 | ソニー株式会社 | Information processing device, information processing method, and program |
WO2019230169A1 (en) | 2018-05-30 | 2019-12-05 | ソニー株式会社 | Display control device, program, and display control method |
GB2574882B (en) | 2018-06-22 | 2020-08-12 | Sony Interactive Entertainment Inc | Method and system for displaying a virtual object |
CN109189210A (en) * | 2018-08-06 | 2019-01-11 | 百度在线网络技术(北京)有限公司 | Mixed reality exchange method, device and storage medium |
CN109241385A (en) * | 2018-08-08 | 2019-01-18 | 百度在线网络技术(北京)有限公司 | Searching method, equipment and storage medium |
KR102091217B1 (en) * | 2018-12-12 | 2020-03-19 | 주식회사 하이쓰리디 | Augmented reality video editing system for a mobile device |
JP7207210B2 (en) * | 2019-07-12 | 2023-01-18 | 日本電信電話株式会社 | Action recognition device, action recognition method, and action recognition program |
US11908090B2 (en) | 2019-07-29 | 2024-02-20 | Sony Group Corporation | Information processing device, information processing method, and program |
KR20210030147A (en) * | 2019-09-09 | 2021-03-17 | 삼성전자주식회사 | 3d rendering method and 3d rendering apparatus |
TWI759764B (en) * | 2020-06-12 | 2022-04-01 | 光時代科技有限公司 | Superimpose virtual object method based on optical communitation device, electric apparatus, and computer readable storage medium |
EP4290468A1 (en) | 2021-02-05 | 2023-12-13 | Sony Semiconductor Solutions Corporation | Information processing device, information processing method, and program |
WO2023021757A1 (en) * | 2021-08-20 | 2023-02-23 | ソニーグループ株式会社 | Information processing device, information processing method, and program |
WO2023026546A1 (en) | 2021-08-25 | 2023-03-02 | ソニーセミコンダクタソリューションズ株式会社 | Information processing device |
JP7506135B2 (en) | 2022-11-15 | 2024-06-25 | ソフトバンク株式会社 | Information processing device, program, and display method |
JP2024073077A (en) | 2022-11-17 | 2024-05-29 | ソニーセミコンダクタソリューションズ株式会社 | Information processing apparatus, and information processing method |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009289035A (en) * | 2008-05-29 | 2009-12-10 | Jiro Makino | Image display system, portable display, server computer, and archaeological sightseeing system |
JP2010231459A (en) * | 2009-03-26 | 2010-10-14 | Kyocera Corp | Image synthesizing apparatus |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6469660B1 (en) * | 2000-04-13 | 2002-10-22 | United Parcel Service Inc | Method and system for displaying target icons correlated to target data integrity |
US6765569B2 (en) * | 2001-03-07 | 2004-07-20 | University Of Southern California | Augmented-reality tool employing scene-feature autocalibration during camera motion |
JP4231216B2 (en) * | 2001-07-31 | 2009-02-25 | 日本放送協会 | Virtual scene shooting method and apparatus |
US6867753B2 (en) * | 2002-10-28 | 2005-03-15 | University Of Washington | Virtual image registration in augmented display field |
JP4378118B2 (en) * | 2003-06-27 | 2009-12-02 | 学校法人早稲田大学 | 3D image presentation device |
US7466650B2 (en) | 2004-09-08 | 2008-12-16 | Visualgate Systems, Inc. | System, method and apparatus for adaptive video surveillance over power lines |
JP4726194B2 (en) * | 2005-04-01 | 2011-07-20 | キヤノン株式会社 | Calibration method and apparatus |
JP2007156561A (en) * | 2005-11-30 | 2007-06-21 | Canon Inc | Augmented reality presenting method and system |
JP4976756B2 (en) * | 2006-06-23 | 2012-07-18 | キヤノン株式会社 | Information processing method and apparatus |
JP4804256B2 (en) * | 2006-07-27 | 2011-11-02 | キヤノン株式会社 | Information processing method |
TW200843989A (en) * | 2007-05-02 | 2008-11-16 | Automotive Res & Testing Ct | Automobile lateral collision warning system |
JP4894741B2 (en) * | 2007-12-03 | 2012-03-14 | ソニー株式会社 | Information processing apparatus, information processing method, program, and recording medium |
KR101188588B1 (en) * | 2008-03-27 | 2012-10-08 | 주식회사 만도 | Monocular Motion Stereo-Based Free Parking Space Detection Apparatus and Method |
JP5111210B2 (en) * | 2008-04-09 | 2013-01-09 | キヤノン株式会社 | Image processing apparatus and image processing method |
JP2010134649A (en) * | 2008-12-03 | 2010-06-17 | Canon Inc | Information processing apparatus, its processing method, and program |
CN101520904B (en) * | 2009-03-24 | 2011-12-28 | 上海水晶石信息技术有限公司 | Reality augmenting method with real environment estimation and reality augmenting system |
CN101551732A (en) * | 2009-03-24 | 2009-10-07 | 上海水晶石信息技术有限公司 | Method for strengthening reality having interactive function and a system thereof |
US8384770B2 (en) * | 2010-06-02 | 2013-02-26 | Nintendo Co., Ltd. | Image display system, image display apparatus, and image display method |
JP5662787B2 (en) * | 2010-12-14 | 2015-02-04 | 株式会社Nttドコモ | Mobile terminal and image processing method |
-
2011
- 2011-04-08 JP JP2011086684A patent/JP5732988B2/en active Active
-
2012
- 2012-03-28 TW TW101110902A patent/TW201243767A/en unknown
- 2012-04-02 US US14/008,789 patent/US9269002B2/en active Active
- 2012-04-02 KR KR1020137019062A patent/KR20140017527A/en not_active Application Discontinuation
- 2012-04-02 EP EP12767610.4A patent/EP2695143B1/en active Active
- 2012-04-02 RU RU2013143978/08A patent/RU2013143978A/en not_active Application Discontinuation
- 2012-04-02 WO PCT/JP2012/002269 patent/WO2012137463A1/en active Application Filing
- 2012-04-02 BR BR112013025416A patent/BR112013025416A2/en not_active IP Right Cessation
- 2012-04-06 CN CN201220143819XU patent/CN202976122U/en not_active Expired - Lifetime
- 2012-04-06 CN CN201210100149.8A patent/CN102867169B/en active Active
-
2016
- 2016-02-04 US US15/015,926 patent/US9582938B2/en active Active
- 2016-12-14 US US15/378,444 patent/US9836887B2/en active Active
-
2017
- 2017-11-09 US US15/807,680 patent/US10395429B2/en active Active
-
2019
- 2019-07-15 US US16/511,216 patent/US10930083B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009289035A (en) * | 2008-05-29 | 2009-12-10 | Jiro Makino | Image display system, portable display, server computer, and archaeological sightseeing system |
JP2010231459A (en) * | 2009-03-26 | 2010-10-14 | Kyocera Corp | Image synthesizing apparatus |
Non-Patent Citations (1)
Title |
---|
See also references of EP2695143A4 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9436874B2 (en) | 2013-01-28 | 2016-09-06 | Huawei Device Co., Ltd. | Method for discovering augmented reality object, and terminal |
EP3070681A4 (en) * | 2013-11-13 | 2017-07-12 | Sony Corporation | Display control device, display control method and program |
US20150346816A1 (en) * | 2014-05-30 | 2015-12-03 | Moriahtown Co., Ltd. | Display device using wearable eyeglasses and method of operating the same |
CN104090973A (en) * | 2014-07-18 | 2014-10-08 | 百度在线网络技术(北京)有限公司 | Information presentation method and device |
Also Published As
Publication number | Publication date |
---|---|
US9582938B2 (en) | 2017-02-28 |
JP2012221250A (en) | 2012-11-12 |
EP2695143B1 (en) | 2020-02-26 |
CN102867169A (en) | 2013-01-09 |
US20160217622A1 (en) | 2016-07-28 |
US9269002B2 (en) | 2016-02-23 |
US20140016825A1 (en) | 2014-01-16 |
US9836887B2 (en) | 2017-12-05 |
US20190340826A1 (en) | 2019-11-07 |
US10930083B2 (en) | 2021-02-23 |
US20180068491A1 (en) | 2018-03-08 |
BR112013025416A2 (en) | 2016-12-27 |
KR20140017527A (en) | 2014-02-11 |
JP5732988B2 (en) | 2015-06-10 |
EP2695143A4 (en) | 2016-01-13 |
CN102867169B (en) | 2017-07-11 |
CN202976122U (en) | 2013-06-05 |
TW201243767A (en) | 2012-11-01 |
US10395429B2 (en) | 2019-08-27 |
US20170157513A1 (en) | 2017-06-08 |
RU2013143978A (en) | 2015-04-10 |
EP2695143A1 (en) | 2014-02-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10930083B2 (en) | Image processing apparatus and display control method | |
US11195307B2 (en) | Image processing apparatus, image processing method, and program | |
US10521944B2 (en) | Repositioning user perspectives in virtual reality environments | |
US9761054B2 (en) | Augmented reality computing with inertial sensors | |
US8811667B2 (en) | Terminal device, object control method, and program | |
EP2777026B1 (en) | Information processing device, information processing method, and program | |
US9823821B2 (en) | Information processing apparatus, display control method, and program for superimposing virtual objects on input image and selecting an interested object | |
CN108885856A (en) | Information processing equipment, information processing method and program | |
CN108885497B (en) | Information processing apparatus, information processing method, and computer readable medium | |
CN112114659B (en) | Method and system for determining a fine gaze point of a user | |
WO2021125190A1 (en) | Information processing device, information processing system, and information processing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12767610 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20137019062 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2013143978 Country of ref document: RU Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14008789 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112013025416 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 112013025416 Country of ref document: BR Kind code of ref document: A2 Effective date: 20131001 |