EP2717227A2 - Image processing program, image processing device, image processing system, and image processing method - Google Patents
Image processing program, image processing device, image processing system, and image processing method Download PDFInfo
- Publication number
- EP2717227A2 EP2717227A2 EP13174670.3A EP13174670A EP2717227A2 EP 2717227 A2 EP2717227 A2 EP 2717227A2 EP 13174670 A EP13174670 A EP 13174670A EP 2717227 A2 EP2717227 A2 EP 2717227A2
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- display
- reference information
- virtual
- virtual space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
Definitions
- the present invention relates to an image processing program, an image processing device, an image processing system, and an image processing method for superimposing various types of information on a real space.
- AR augmented reality
- Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana An Augmented Reality System and its Calibration based on Marker Tracking. Journal of the Virtual Reality Society of Japan, Vol. 4 No. 4, 1999. 606-616 . discloses a method of displaying a virtual object based on a position and a posture of a marker in an image captured by a camera.
- Non-Patent document 1 Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana. An Augmented Reality System and its Calibration based on Marker Tracking. Journal of the Virtual Reality Society of Japan, Vol. 4 Neo. 4,1999. 606-616 .
- a feature in a real space becomes a reference of a position or a posture of a virtual object in a virtual space
- display of the virtual object is restricted by the feature.
- the marker since a virtual object associated with a marker is displayed based on the marker, the marker must be maintained in a predetermined state relative to an imaging device in order to display the virtual object as desired.
- the present invention has been made in consideration of the problems described above, and an object thereof is to ease a restriction imposed on the display of a virtual object by a feature in a real space in augmented reality techniques.
- the present invention adopts the following configuration as a solution to the problems described above.
- the present invention is an image processing program which causes a computer to function as: updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space; image generating means for generating an image of a virtual space including the virtual object; display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; and update interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- the display device may be connected as a peripheral device to a computer that executes the program according to the present invention or may be connected to the computer via a communication network or the like.
- the computer that is mainly responsible for executing the program according to the present invention may be built on a virtual environment such as a so-called cloud.
- display reference information of a virtual object can be determined without being restricted by a position or a posture of a feature such as a marker. Therefore, a restriction imposed on the display of a virtual object by a feature in a real space in augmented reality techniques can be eased.
- application of the present invention is not limited to particular types of augmented reality techniques.
- the present invention can be applied to an augmented reality technique which enables a user to view an image of a virtual space superimposed on a real space by displaying a composite image created by compositing a virtual space image on a captured image and to an augmented reality technique (head-up display: HUD or the like) which enables a user to view an image of a virtual space superimposed on a real space by projecting a virtual space image to a field of view of the user.
- HUD head-up display
- the image processing program may further comprise feature detecting means for detecting a feature in a real space that enables at least any of a position and a posture in the real space to be identified, and the display reference information updating means may update the display reference information stored by the display reference information storing means based on the feature.
- examples of a feature in a real space include a so-called AR marker or a code such as a two-dimensional barcode.
- a feature may be attached to a component such as a card.
- such a feature is not limited to a dedicated marker, a code, or the like. Even an article that is used for another purpose may be used as the feature as long as the article enables a display reference of a virtual object to be acquired.
- the update interrupting means may interrupt an update of the display reference information by the display reference information updating means on the condition that a predetermined operation is performed by a user.
- a state where "a predetermined operation is performed" includes a state where the predetermined operation is currently being performed.
- the image processing program may cause the computer to further function as captured image acquiring means for acquiring a captured image that is captured by an imaging device, and the feature detecting means may detect from the captured image a feature in a space captured in the captured image.
- the display control means may enable a user to view an image of the virtual space superimposed on a real space by causing the display device to display a composite image in which an image of the virtual space is superimposed on the captured image.
- the image processing program may cause the computer to further function as composite image preserving means for preserving the composite image.
- the composite image By enabling the composite image to be preserved in a state where display reference information is fixed, a composite image of a virtual object displayed according to display reference information desired by a user and a captured image can be preserved.
- the composite image that is preserved may be a still image or a moving image.
- the composite image preserving means may preserve the composite image in response to a user operation performed in order to issue an instruction for preserving an image.
- a composite image of a virtual object displayed according to display reference information desired by a user and a captured image in an captured state desired by the user can be preserved.
- the display reference information storing means may store at least any of a position and a posture in the virtual object in the virtual space as the display reference information; the display reference information updating means may update the display reference information based on the feature that enables any of a position and a posture in the real space to be identified; and the image generating means may generate an image of a virtual space including the virtual object having at least any of a position and a posture determined based on the display reference information.
- display reference information is a reference indicating at least any of a position and a posture in a space whose image is captured.
- the display reference information may be, for example, a coordinate system having an origin and three axes thereof determined in accordance with a position and a posture of a marker, or position and posture information of a marker relative to an imaging device.
- the present invention can be considered as being an invention described as follows.
- the present invention is an image processing program which causes a computer to function as: updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space; image generating means for generating an image of a virtual space including the virtual object; display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; and update interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- the present invention can also be considered as being an information processing device, an information processing system having one or more information processing devices, a method that is executed by a computer, or a program to be executed on a computer.
- the present invention may be embodied by a recording medium which is readable by a device such as a computer, a machine, or the like and on which such a program is recorded.
- a recording medium that is readable by a computer or the like refers to a recording medium which accumulates information such as data and programs by an electric action, a magnetic action, an optical action, a mechanical action, or a chemical action and which can be read by a computer or the like.
- a restriction imposed on a display of a virtual object by a feature in a real space in augmented reality techniques can be eased.
- the present invention can be applied to an information processing program to be executed on a computer of a portable information processing device, an information processing device, an information processing system having one or more information processing devices, an information processing method, and the like.
- FIG. 1 is a diagram showing a configuration of a system 100 according to the present embodiment.
- the system 100 comprises an information processing device 1 and a plurality of cards 2a and 2b (however, when types of cards are not distinguished, simply referred to as a "card 2").
- the information processing device 1 is an information processing device to which a central processing unit (CPU) 11, a random access memory (RAM) 12, a read only memory (ROM) 13, an auxiliary storage device 14, an imaging device 15, a display 16, and an input device 17 constituted by various buttons, a touch screen, or the like are electrically connected. Moreover, components may be omitted, replaced, or added as appropriate according to each embodiment in a specific hardware configuration of the information processing device 1.
- the CPU 11 is a central processing unit and controls the respective components of the information processing device 1 including the RAM 12 and the auxiliary storage device 14 by processing commands and data deployed on the RAM 12, the ROM 13, and the like.
- the RAM 12 is a primary storage device controlled by the CPU 11, and various instructions and data are written onto and read from the RAM 12.
- the CPU 11, the RAM 12, and the ROM 13 constitute a controller of the information processing device 1.
- the auxiliary storage device 14 is a non-volatile storage device. Information that is desirably retained even after shutting down the information processing device 1 including an operating system (OS) of the information processing device 1 that is loaded onto the RAM 12, various programs for executing the processes described later, and various data to be used by the information processing device 1 are mainly written into and read from the auxiliary storage device 14.
- OS operating system
- EEPROM electrically erasable programmable ROM
- HDD hard disk drive
- a portable medium that is detachably mounted to the information processing device 1 may be used as the auxiliary storage device 14.
- Examples of a portable medium include a memory card constituted by an EEPROM or the like, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray Disc (BD). Furthermore, a combination of an auxiliary storage device 14 constituted by a portable medium and a non-portable auxiliary storage device 14 can also be used.
- Markers 3a and 3b (however, when types of markers are not distinguished, simply referred to as a "marker 3") which differ from each other in printing methods or the like are attached to the cards 2a and 2b.
- the marker 3 is associated with a virtual object displayed by the information processing device 1 and is an indicator representing a reference of a position and a posture when the virtual object associated with the marker 3 is displayed. While two cards 2 are shown in FIG. 1 , there may be three or more cards 2. In addition, while markers 3a and 3b which differ from one another are attached to the cards 2a to 2b to display different virtual objects, cards 2 attached with a same marker may also be used.
- a virtual object is composited and displayed at a predetermined position relative to the marker 3 associated with the virtual object on the display 16 of the information processing device 1.
- a virtual object has upward and downward, forward and backward, and leftward and rightward directions. Therefore, the marker 3 may enable a display posture of a virtual object to be identified.
- the marker 3 may be a symbol, a character, a graphic, a picture, or a combination thereof which enables a position and a posture relative to the imaging device 15 to be identified by being captured with the imaging device 15.
- the information processing device 1 is an information processing device having a so-called AR function.
- the information processing device 1 has a function of compositing a virtual object in a virtual space rendered using a virtual camera on a captured image of a real space captured using the imaging device 15, and displaying the virtual object composited on the captured image on the display 16.
- a virtual object is three-dimensional image data.
- a virtual object may alternatively be two-dimensional image data.
- FIG. 2 is a diagram showing an outline of a functional configuration of the information processing device 1 according to the present embodiment. Due to the CPU 11 interpreting and executing various programs deployed on the RAM 12, the information processing device 1 according to the present embodiment functions as an information processing device comprising a captured image acquiring unit 21, a feature detecting unit 22, a display reference information updating unit 23, a display reference information storing unit 24, an image generating unit 25, a display control unit 26, an update interrupting unit 27, and a composite image preserving unit 28. Although an example in which all of these functions are executed by a general-purpose CPU 11 is explained in the present embodiment, a part of or all of these functions may be realized by one or a plurality of dedicated processors.
- the captured image acquiring unit 21 acquires a captured image that is captured by the imaging device 15.
- the feature detecting unit 22 is capable of detecting a marker 3 included in the image by performing image processing such as pattern matching. For example, detection of the marker 3 is performed using an image recognition engine.
- the display reference information updating unit 23 acquires information that becomes a reference of a position and a posture in a space captured in the captured image and updates display reference information.
- display reference information stored by the display reference information storing unit 24 is updated according to a latest marker position and a latest marker posture relative to the camera.
- the display reference information storing unit 24 stores display reference information for determining a position and a posture of a virtual object arranged in a virtual space.
- display reference information refers to a reference used to represent a position and a posture of a virtual object in a virtual space.
- display reference information may be a reference used to represent only one of a position and a posture of a virtual object in a virtual space.
- a marker coordinate system which has a center point of the marker 3 as an origin and which uses three axes perpendicular to one another is acquired as display reference information for each marker 3.
- information other than a marker coordinate system such as a captured image itself may alternatively be used as display reference information.
- a plurality of markers 3 can share one marker coordinate system. Defining a coordinate system of a virtual space using a marker 3 arranged in a real space as a reference enables the real space and the virtual space to be associated with each other. Moreover, methods other than using a marker coordinate system may be adopted to associate a real space and a virtual space with each other.
- a virtual object arranged in a virtual space is arranged in a marker coordinate system of the marker 3 with which the virtual object is associated.
- a marker coordinate system can be acquired by calculating a position and a posture of the marker 3 relative to the imaging device 15 based on an appearance of the marker 3 included in the captured image.
- a position and a posture of a virtual camera in a marker coordinate system are conformed to a position and a posture of the imaging device 15 in a real space. Therefore, when a virtual space is defined based on the marker 3 and a position or a capturing direction of the imaging device 15 is varied in the virtual space, an image of the virtual space displayed on the display 16 also varies.
- the image generating unit 25 renders a virtual space image by arranging a virtual object whose position and posture are determined according to display reference information stored by the display reference information storing unit 24 in a virtual space and generating an image of a virtual space as seen from a virtual camera. Subsequently, due to the AR function described earlier, the information processing device 1 according to the present embodiment generates a composite image in which a captured image acquired by the captured image acquiring unit 21 and a virtual space image which includes a virtual object and which is generated by the image generating unit 25 are superimposed.
- the display control unit 26 causes the generated composite image to be displayed by the display 16 that is a display device. Accordingly, a user can get a feeling that the virtual object actually exists in the real space.
- the update interrupting unit 27 interrupts an update of display reference information by the display reference information updating unit 23. Moreover, enabling/disabling update of display reference information can be switched based on a predetermined operation performed by the user. In addition, when a plurality of markers is detected and a plurality of pieces of display reference information is stored by the display reference information storing unit 24, none of the pieces of display reference information is updated as long as update of display reference information is disabled.
- the composite image preserving unit 28 preserves the composite image in response to a user operation performed in order to issue an instruction for preserving the image.
- examples of a "predetermined operation” include pressing down on a predetermined button (in this case, update of display reference information remains disabled until a release operation is performed) and a predetermined button being pressed down (in this case, update of display reference information remains disabled as long as the predetermined button is pressed down).
- the information processing device 1 In addition to display reference information stored by the display reference information storing unit 24 described above, the information processing device 1 retains marker information and object information.
- Marker information is information related to the marker 3.
- marker information includes a marker ID for identifying the marker 3, a marker image, a marker size, an associated object ID, a position and a posture of a virtual object, and a display size of the object.
- a marker image is an image showing an exterior of the marker 3.
- a marker size is information indicating a size of the marker 3 such as vertical and horizontal lengths of the marker 3.
- the display reference information updating unit 23 of the information processing device 1 is capable of acquiring a distance between the imaging device 15 and the marker 3, a posture of the marker 3, and the like or, in other words, position and posture information of the marker 3 and a marker coordinate system from an appearance of the marker 3 included in a captured image.
- the associated object ID is an identification number of a virtual object displayed at a position corresponding to the marker 3.
- marker information includes an object ID of a virtual object managed by a relevant marker coordinate system.
- a position and a posture of a virtual object are represented by a position (coordinate value) and a posture (vector) in a marker coordinate system. Marker information exists for each marker 3 used in the system 100.
- Object information is information related to a virtual object displayed at a position corresponding to the marker 3.
- object information includes an object ID for identifying the virtual object and object data.
- Object information exists for each object used in the system 100.
- FIG. 3 is a flow chart showing a flow of image processing according to the present embodiment.
- the image processing shown in the present flow chart is started when a user operation for activating an imaging function using the imaging device 15 is accepted by the information processing device 1.
- Information stored by the display reference information storing unit 24 is initialized upon activation of the imaging function. Therefore, the display reference information storing unit 24 does not store display reference information upon activation of the imaging function.
- processing according to the present embodiment is repetitively executed for each frame divided at 60 frames/second.
- steps S101 and S102 a captured image is acquired and a judgment is made on whether or not update of display reference information is enabled.
- the captured image acquiring unit 21 acquires a captured image captured by the imaging device 15 (step S101).
- the update interrupting unit 27 judges whether or not update of display reference information is enabled.
- the processing skips steps S103 and S104 and advances to step S105.
- the processing advances to step S103.
- step S103 a marker 3 is detected from the captured image.
- the feature detecting unit 22 detects all markers 3 corresponding to a marker image included in marker information from the captured image as features of a captured space. Detection of the markers 3 can be performed using a general image recognition engine. Subsequently, processing proceeds to step S104.
- step S104 a reference acquiring process is executed for each marker 3.
- the display reference information updating unit 23 acquires position and posture information of the marker 3 in a real space for each marker 3 and updates display reference information of each marker 3. More specifically, the display reference information updating unit 23 acquires a position and a posture of the marker 3 in a real space based on a position of the marker 3 in the captured image, a result of a comparison between a marker size included in marker information and a size of the marker 3 included in the captured image, and a distortion of the marker 3 in the captured image with respect to a marker image included in the marker information.
- the display reference information updating unit 23 updates display reference information with the position and posture information of the marker in a real space acquired in this manner. Subsequently, processing proceeds to step S105.
- step S105 an image of a virtual space is generated.
- the image generating unit 25 renders an image of a virtual space including one or more virtual objects of which at least any of a position and a posture has been determined according to display reference information and which is arranged in a marker coordinate system from a perspective of a virtual camera arranged in the marker coordinate system at a same position as the imaging device 15.
- Object data for rendering the virtual object is acquired from object information.
- the virtual object may be animated by varying the virtual object every frame or every few frames. Examples of the animation include a character of the virtual object changing a facial expression or moving around. Subsequently, processing proceeds to step S106.
- step S106 a display process is executed.
- the display control unit 26 generates a composite image in which the image of the virtual space is superimposed on the captured image, and outputs the composite image to the display 16 to have the composite image displayed. Subsequently, processing proceeds to step S107.
- the composite image is preserved according to a composite image preserving instruction.
- the composite image preserving unit 28 preserves the composite image (step S108).
- the composite image may be preserved as a still image or as a moving image.
- the composite image is not preserved (step S107). Subsequently, processing proceeds to step S109.
- step S109 the processing shown in the present flow chart is executed for each frame. Therefore, the processing shown in the present flow chart is repetitively executed starting from step S101 on a regular basis until the imaging function is terminated based on a user operation or the like (step S109). Due to the execution of the image processing described above, update of display reference information of a virtual object which is updated based on the marker 3 can be disabled once the display reference information reaches a state that is desirable to a user, a capturing position or an angle with respect to a real space can be adjusted while update is being disabled, and a composite image of the virtual object in a desired state and the real space in a desired state can be obtained.
- FIGS. 4 and 5 are diagrams showing examples of a display screen of the display 16 when a composite image is acquired by the image processing according to the present embodiment. More specifically, FIG. 4 is a diagram showing an example of a display screen when update of display reference information is enabled, and FIG. 5 is a diagram showing an example of a display screen when update of display reference information is disabled.
- the display reference information storing unit 24 does not store display reference information upon activation of the imaging function. Therefore, when a marker is not included in a captured image immediately after the imaging function of the information processing device 1 is activated in response to a user operation, a virtual object is not arranged in a virtual space and the captured image obtained by the imaging device 15 is displayed on the display 16.
- the display reference information updating unit 23 updates display reference information based on the marker 3 detected by the feature detecting unit 22 and the image generating unit 25 renders a virtual object at a position and in a posture corresponding to the marker 3. Therefore, a composite image in which the virtual object is superimposed on the marker 3 is displayed on the display 16 (refer to FIG. 4 ).
- update of display reference information is enabled in an initial state, a user can adjust the imaging device 15 or the marker 3 in this state to put a position and a posture of the virtual object displayed on the display 16 in a desired state.
- the user disables update of display reference information by performing a predetermined operation.
- update of display reference information is disabled, the virtual object is displayed according to last updated display reference information. Even if a position or a posture of a marker in the captured image varies or a marker is no longer included in the captured image, the virtual object is maintained in the state desired by the user (refer to FIG. 5 ).
- the user can adjust a position or an angle of the imaging device 15 to place a captured image in a desired state.
- the user can preserve a desired composite image. For example, FIG. 5 shows a state where a character of a virtual object is on top of the user's head.
- images that cannot be obtained by conventional devices in which a virtual object is always displayed on a marker can be readily obtained such as a composite image showing a character of a virtual object standing upside down on a ceiling or a composite image showing characters of a plurality of virtual objects lined up on a cake.
- the display 16 included in the information processing device 1 may be a display device which can be stereoscopically viewed with the naked eye.
- a display device adopting a lenticular system or a parallax barrier system is used so that a left-eye image and a right-eye image, which are alternately displayed in a horizontal direction, are respectively viewed separately by the left eye and the right eye.
- the display 16 may be a display device capable of switching between a stereoscopic display mode which displays stereoscopically viewable images and a planar display mode which displays images in a planar view (which displays planar view images).
- the display modes may be switched using a 3D adjustment switch (not shown).
- an imaging device capable of stereo photography is used as the imaging device 15.
- the image generating unit 25 generates a left-eye virtual image and a right-eye virtual image using two virtual cameras corresponding to the stereo camera of the imaging device 15.
- the display control unit generates a left-eye composite image and a right-eye composite image, and outputs the composite images to a stereoscopically viewable display 16.
- the imaging device 15 may be an imaging device capable of capturing images of the front and the rear of the display 16 by including an outer imaging unit provided so as to be capable of capturing an image along a line of sight of a user viewing the display 16 and an inner imaging unit provided so as to be capable of capturing an image of the user viewing the display 16.
- an outer imaging unit provided so as to be capable of capturing an image along a line of sight of a user viewing the display 16
- an inner imaging unit provided so as to be capable of capturing an image of the user viewing the display 16.
- a configuration may be adopted which enables a display posture of a virtual object to be adjusted based on a posture of the imaging device 15 when update of display reference information is disabled.
- a position and a posture of the imaging device 15 can be detected using an acceleration sensor, an angular velocity sensor, or the like.
- the information processing device 1 calculates a posture of the imaging device 15 based on an output from the acceleration sensor, the angular velocity sensor, or the like to adjust a display posture of the virtual object.
- a posture of an imaging device varies while update of display reference information is being disabled, a posture of the virtual object can be maintained "with respect to a real space".
- the display reference information may be any information as long as the information is obtained from a real space and can be used as a reference of at least any of a position and a posture of a virtual object in a virtual space.
- the display reference information may be a captured image itself.
- the image generating unit 25 extracts a display reference of a virtual object from a captured image stored as display reference information for each frame.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Architecture (AREA)
- Processing Or Creating Images (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- The present invention relates to an image processing program, an image processing device, an image processing system, and an image processing method for superimposing various types of information on a real space.
- Conventionally, research is being conducted regarding reference coordinate determination for displaying a virtual object in the field of augmented reality (AR) which involves techniques for superimposing various types of information on a real space. For example, Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana. An Augmented Reality System and its Calibration based on Marker Tracking. Journal of the Virtual Reality Society of Japan, Vol. 4 No. 4, 1999. 606-616. discloses a method of displaying a virtual object based on a position and a posture of a marker in an image captured by a camera.
- [Non-Patent document 1] Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana. An Augmented Reality System and its Calibration based on Marker Tracking. Journal of the Virtual Reality Society of Japan, Vol. 4 Neo. 4,1999. 606-616.
- However, in augmented reality techniques, since a feature in a real space becomes a reference of a position or a posture of a virtual object in a virtual space, display of the virtual object is restricted by the feature. For example, since a virtual object associated with a marker is displayed based on the marker, the marker must be maintained in a predetermined state relative to an imaging device in order to display the virtual object as desired. In addition, it is difficult to display a virtual object without including a marker.
- The present invention has been made in consideration of the problems described above, and an object thereof is to ease a restriction imposed on the display of a virtual object by a feature in a real space in augmented reality techniques.
- The present invention adopts the following configuration as a solution to the problems described above. Specifically, the present invention is an image processing program which causes a computer to function as: updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space; image generating means for generating an image of a virtual space including the virtual object; display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; and update interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- In this case, the display device may be connected as a peripheral device to a computer that executes the program according to the present invention or may be connected to the computer via a communication network or the like. In addition, the computer that is mainly responsible for executing the program according to the present invention may be built on a virtual environment such as a so-called cloud.
- According to the image processing program described above, since an update of the display reference information is interrupted when predetermined conditions are met, display reference information of a virtual object can be determined without being restricted by a position or a posture of a feature such as a marker. Therefore, a restriction imposed on the display of a virtual object by a feature in a real space in augmented reality techniques can be eased.
- Moreover, application of the present invention is not limited to particular types of augmented reality techniques. For example, the present invention can be applied to an augmented reality technique which enables a user to view an image of a virtual space superimposed on a real space by displaying a composite image created by compositing a virtual space image on a captured image and to an augmented reality technique (head-up display: HUD or the like) which enables a user to view an image of a virtual space superimposed on a real space by projecting a virtual space image to a field of view of the user.
- In addition, the image processing program may further comprise feature detecting means for detecting a feature in a real space that enables at least any of a position and a posture in the real space to be identified, and the display reference information updating means may update the display reference information stored by the display reference information storing means based on the feature.
- In this case, examples of a feature in a real space include a so-called AR marker or a code such as a two-dimensional barcode. Such a feature may be attached to a component such as a card. In addition, such a feature is not limited to a dedicated marker, a code, or the like. Even an article that is used for another purpose may be used as the feature as long as the article enables a display reference of a virtual object to be acquired.
- In addition, the update interrupting means may interrupt an update of the display reference information by the display reference information updating means on the condition that a predetermined operation is performed by a user.
- Accordingly, the user can specify a timing where the display reference information is fixed and a virtual object can be fixed with display reference information desired by the user. Moreover, in this case, a state where "a predetermined operation is performed" includes a state where the predetermined operation is currently being performed.
- In addition, the image processing program may cause the computer to further function as captured image acquiring means for acquiring a captured image that is captured by an imaging device, and the feature detecting means may detect from the captured image a feature in a space captured in the captured image.
- Furthermore, the display control means may enable a user to view an image of the virtual space superimposed on a real space by causing the display device to display a composite image in which an image of the virtual space is superimposed on the captured image.
- In addition, the image processing program may cause the computer to further function as composite image preserving means for preserving the composite image.
- By enabling the composite image to be preserved in a state where display reference information is fixed, a composite image of a virtual object displayed according to display reference information desired by a user and a captured image can be preserved. In this case, the composite image that is preserved may be a still image or a moving image.
- Furthermore, the composite image preserving means may preserve the composite image in response to a user operation performed in order to issue an instruction for preserving an image.
- By preserving a composite image in response to a user operation in a state where display reference information is fixed, a composite image of a virtual object displayed according to display reference information desired by a user and a captured image in an captured state desired by the user can be preserved.
- In addition, the display reference information storing means may store at least any of a position and a posture in the virtual object in the virtual space as the display reference information; the display reference information updating means may update the display reference information based on the feature that enables any of a position and a posture in the real space to be identified; and the image generating means may generate an image of a virtual space including the virtual object having at least any of a position and a posture determined based on the display reference information.
- For example, display reference information is a reference indicating at least any of a position and a posture in a space whose image is captured. The display reference information may be, for example, a coordinate system having an origin and three axes thereof determined in accordance with a position and a posture of a marker, or position and posture information of a marker relative to an imaging device.
- Moreover, the present invention can be considered as being an invention described as follows. Specifically, the present invention is an image processing program which causes a computer to function as: updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space; image generating means for generating an image of a virtual space including the virtual object; display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; and update interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- In addition, the present invention can also be considered as being an information processing device, an information processing system having one or more information processing devices, a method that is executed by a computer, or a program to be executed on a computer. Furthermore, the present invention may be embodied by a recording medium which is readable by a device such as a computer, a machine, or the like and on which such a program is recorded. In this case, a recording medium that is readable by a computer or the like refers to a recording medium which accumulates information such as data and programs by an electric action, a magnetic action, an optical action, a mechanical action, or a chemical action and which can be read by a computer or the like.
- According to the present invention, a restriction imposed on a display of a virtual object by a feature in a real space in augmented reality techniques can be eased.
-
-
FIG. 1 is a diagram showing an outline of a system including an information processing device according to an embodiment; -
FIG. 2 is a diagram showing an outline of a functional configuration of an information processing device according to an embodiment; -
FIG. 3 is a flow chart showing a flow of image processing according to an embodiment; -
FIG. 4 is a diagram showing an example of a display screen when update of display reference information is enabled according to an embodiment; and -
FIG. 5 is a diagram showing an example of a display screen when update of display reference information is disabled according to an embodiment. - Hereinafter, an embodiment of the present invention will be described with reference to the drawings. It should be noted that the embodiment described below merely represents an example of implementing the present invention and is not intended to limit the present invention to the specific configuration described below. When implementing the present invention, a specific configuration may be adopted as appropriate in accordance with each embodiment. For example, the present invention can be applied to an information processing program to be executed on a computer of a portable information processing device, an information processing device, an information processing system having one or more information processing devices, an information processing method, and the like.
-
FIG. 1 is a diagram showing a configuration of asystem 100 according to the present embodiment. Thesystem 100 comprises aninformation processing device 1 and a plurality ofcards card 2"). - The
information processing device 1 is an information processing device to which a central processing unit (CPU) 11, a random access memory (RAM) 12, a read only memory (ROM) 13, anauxiliary storage device 14, animaging device 15, adisplay 16, and aninput device 17 constituted by various buttons, a touch screen, or the like are electrically connected. Moreover, components may be omitted, replaced, or added as appropriate according to each embodiment in a specific hardware configuration of theinformation processing device 1. - The
CPU 11 is a central processing unit and controls the respective components of theinformation processing device 1 including theRAM 12 and theauxiliary storage device 14 by processing commands and data deployed on theRAM 12, theROM 13, and the like. In addition, theRAM 12 is a primary storage device controlled by theCPU 11, and various instructions and data are written onto and read from theRAM 12. In other words, theCPU 11, theRAM 12, and theROM 13 constitute a controller of theinformation processing device 1. - The
auxiliary storage device 14 is a non-volatile storage device. Information that is desirably retained even after shutting down theinformation processing device 1 including an operating system (OS) of theinformation processing device 1 that is loaded onto theRAM 12, various programs for executing the processes described later, and various data to be used by theinformation processing device 1 are mainly written into and read from theauxiliary storage device 14. For example, an electrically erasable programmable ROM (EEPROM) or a hard disk drive (HDD) can be used as theauxiliary storage device 14. Alternatively, a portable medium that is detachably mounted to theinformation processing device 1 may be used as theauxiliary storage device 14. Examples of a portable medium include a memory card constituted by an EEPROM or the like, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray Disc (BD). Furthermore, a combination of anauxiliary storage device 14 constituted by a portable medium and a non-portableauxiliary storage device 14 can also be used. -
Markers marker 3") which differ from each other in printing methods or the like are attached to thecards marker 3 is associated with a virtual object displayed by theinformation processing device 1 and is an indicator representing a reference of a position and a posture when the virtual object associated with themarker 3 is displayed. While twocards 2 are shown inFIG. 1 , there may be three ormore cards 2. In addition, whilemarkers cards 2a to 2b to display different virtual objects,cards 2 attached with a same marker may also be used. - In the present embodiment, a virtual object is composited and displayed at a predetermined position relative to the
marker 3 associated with the virtual object on thedisplay 16 of theinformation processing device 1. In addition, a virtual object has upward and downward, forward and backward, and leftward and rightward directions. Therefore, themarker 3 may enable a display posture of a virtual object to be identified. In other words, themarker 3 may be a symbol, a character, a graphic, a picture, or a combination thereof which enables a position and a posture relative to theimaging device 15 to be identified by being captured with theimaging device 15. - Next, functions possessed by the
information processing device 1 according to the present embodiment will be described. Theinformation processing device 1 according to the present embodiment is an information processing device having a so-called AR function. Theinformation processing device 1 has a function of compositing a virtual object in a virtual space rendered using a virtual camera on a captured image of a real space captured using theimaging device 15, and displaying the virtual object composited on the captured image on thedisplay 16. In the present embodiment, a virtual object is three-dimensional image data. However, a virtual object may alternatively be two-dimensional image data. -
FIG. 2 is a diagram showing an outline of a functional configuration of theinformation processing device 1 according to the present embodiment. Due to theCPU 11 interpreting and executing various programs deployed on theRAM 12, theinformation processing device 1 according to the present embodiment functions as an information processing device comprising a capturedimage acquiring unit 21, afeature detecting unit 22, a display referenceinformation updating unit 23, a display referenceinformation storing unit 24, animage generating unit 25, adisplay control unit 26, anupdate interrupting unit 27, and a compositeimage preserving unit 28. Although an example in which all of these functions are executed by a general-purpose CPU 11 is explained in the present embodiment, a part of or all of these functions may be realized by one or a plurality of dedicated processors. - The captured
image acquiring unit 21 acquires a captured image that is captured by theimaging device 15. In addition, with respect to an image captured by theimaging device 15, thefeature detecting unit 22 is capable of detecting amarker 3 included in the image by performing image processing such as pattern matching. For example, detection of themarker 3 is performed using an image recognition engine. - Based on the detected
marker 3, the display referenceinformation updating unit 23 acquires information that becomes a reference of a position and a posture in a space captured in the captured image and updates display reference information. In the present embodiment, even when a camera moves or themarker 3 is moved, display reference information stored by the display referenceinformation storing unit 24 is updated according to a latest marker position and a latest marker posture relative to the camera. - The display reference
information storing unit 24 stores display reference information for determining a position and a posture of a virtual object arranged in a virtual space. In the present embodiment, display reference information refers to a reference used to represent a position and a posture of a virtual object in a virtual space. However, display reference information may be a reference used to represent only one of a position and a posture of a virtual object in a virtual space. In a reference acquiring process according to the present embodiment, a marker coordinate system which has a center point of themarker 3 as an origin and which uses three axes perpendicular to one another is acquired as display reference information for eachmarker 3. However, information other than a marker coordinate system such as a captured image itself may alternatively be used as display reference information. In addition, a plurality ofmarkers 3 can share one marker coordinate system. Defining a coordinate system of a virtual space using amarker 3 arranged in a real space as a reference enables the real space and the virtual space to be associated with each other. Moreover, methods other than using a marker coordinate system may be adopted to associate a real space and a virtual space with each other. - In the present embodiment, a virtual object arranged in a virtual space is arranged in a marker coordinate system of the
marker 3 with which the virtual object is associated. A marker coordinate system can be acquired by calculating a position and a posture of themarker 3 relative to theimaging device 15 based on an appearance of themarker 3 included in the captured image. A position and a posture of a virtual camera in a marker coordinate system are conformed to a position and a posture of theimaging device 15 in a real space. Therefore, when a virtual space is defined based on themarker 3 and a position or a capturing direction of theimaging device 15 is varied in the virtual space, an image of the virtual space displayed on thedisplay 16 also varies. - The
image generating unit 25 renders a virtual space image by arranging a virtual object whose position and posture are determined according to display reference information stored by the display referenceinformation storing unit 24 in a virtual space and generating an image of a virtual space as seen from a virtual camera. Subsequently, due to the AR function described earlier, theinformation processing device 1 according to the present embodiment generates a composite image in which a captured image acquired by the capturedimage acquiring unit 21 and a virtual space image which includes a virtual object and which is generated by theimage generating unit 25 are superimposed. - The
display control unit 26 causes the generated composite image to be displayed by thedisplay 16 that is a display device. Accordingly, a user can get a feeling that the virtual object actually exists in the real space. - When updating of display reference information is disabled, the
update interrupting unit 27 interrupts an update of display reference information by the display referenceinformation updating unit 23. Moreover, enabling/disabling update of display reference information can be switched based on a predetermined operation performed by the user. In addition, when a plurality of markers is detected and a plurality of pieces of display reference information is stored by the display referenceinformation storing unit 24, none of the pieces of display reference information is updated as long as update of display reference information is disabled. Having such a function enables theinformation processing device 1 according to the present embodiment to disable update of display reference information of a virtual object which is updated based on themarker 3 once the display reference information reaches a state that is desirable to the user, adjust a capturing position or an angle with respect to a real space while update is being disabled, and obtain a composite image of the virtual object in a desired state and the real space in a desired state. The compositeimage preserving unit 28 preserves the composite image in response to a user operation performed in order to issue an instruction for preserving the image. - In this case, examples of a "predetermined operation" include pressing down on a predetermined button (in this case, update of display reference information remains disabled until a release operation is performed) and a predetermined button being pressed down (in this case, update of display reference information remains disabled as long as the predetermined button is pressed down).
- Next, information retained by the
information processing device 1 according to the present embodiment will be described. In addition to display reference information stored by the display referenceinformation storing unit 24 described above, theinformation processing device 1 retains marker information and object information. - Marker information is information related to the
marker 3. For example, marker information includes a marker ID for identifying themarker 3, a marker image, a marker size, an associated object ID, a position and a posture of a virtual object, and a display size of the object. A marker image is an image showing an exterior of themarker 3. In addition, a marker size is information indicating a size of themarker 3 such as vertical and horizontal lengths of themarker 3. Based on a marker image and a marker size, the display referenceinformation updating unit 23 of theinformation processing device 1 is capable of acquiring a distance between theimaging device 15 and themarker 3, a posture of themarker 3, and the like or, in other words, position and posture information of themarker 3 and a marker coordinate system from an appearance of themarker 3 included in a captured image. The associated object ID is an identification number of a virtual object displayed at a position corresponding to themarker 3. Moreover, two or more virtual objects may be associated with onemarker 3. In the present embodiment, marker information includes an object ID of a virtual object managed by a relevant marker coordinate system. A position and a posture of a virtual object are represented by a position (coordinate value) and a posture (vector) in a marker coordinate system. Marker information exists for eachmarker 3 used in thesystem 100. - Object information is information related to a virtual object displayed at a position corresponding to the
marker 3. For example, object information includes an object ID for identifying the virtual object and object data. Object information exists for each object used in thesystem 100. - Next, a flow of processing executed in the present embodiment will be described. Moreover, specific contents and a processing sequence of processing shown in a flow chart according to the present embodiment merely represent an example of implementing the present invention. Specific processing contents and a processing sequence may be appropriately selected for each embodiment of the present invention.
-
FIG. 3 is a flow chart showing a flow of image processing according to the present embodiment. The image processing shown in the present flow chart is started when a user operation for activating an imaging function using theimaging device 15 is accepted by theinformation processing device 1. Information stored by the display referenceinformation storing unit 24 is initialized upon activation of the imaging function. Therefore, the display referenceinformation storing unit 24 does not store display reference information upon activation of the imaging function. Moreover, processing according to the present embodiment is repetitively executed for each frame divided at 60 frames/second. - In steps S101 and S102, a captured image is acquired and a judgment is made on whether or not update of display reference information is enabled. The captured
image acquiring unit 21 acquires a captured image captured by the imaging device 15 (step S101). Subsequently, theupdate interrupting unit 27 judges whether or not update of display reference information is enabled. When it is judged that update of display reference information is not enabled (update of display reference information is disabled), in order to interrupt update of display reference information by the display referenceinformation updating unit 23, the processing skips steps S103 and S104 and advances to step S105. On the other hand, when it is judged that update of display reference information is enabled, in order to perform update of display reference information by the display referenceinformation updating unit 23, the processing advances to step S103. - In step S103, a
marker 3 is detected from the captured image. When a captured image is acquired, thefeature detecting unit 22 detects allmarkers 3 corresponding to a marker image included in marker information from the captured image as features of a captured space. Detection of themarkers 3 can be performed using a general image recognition engine. Subsequently, processing proceeds to step S104. - In step S104, a reference acquiring process is executed for each
marker 3. Based on the detectedmarker 3, the display referenceinformation updating unit 23 acquires position and posture information of themarker 3 in a real space for eachmarker 3 and updates display reference information of eachmarker 3. More specifically, the display referenceinformation updating unit 23 acquires a position and a posture of themarker 3 in a real space based on a position of themarker 3 in the captured image, a result of a comparison between a marker size included in marker information and a size of themarker 3 included in the captured image, and a distortion of themarker 3 in the captured image with respect to a marker image included in the marker information. The display referenceinformation updating unit 23 updates display reference information with the position and posture information of the marker in a real space acquired in this manner. Subsequently, processing proceeds to step S105. - In step S105, an image of a virtual space is generated. The
image generating unit 25 renders an image of a virtual space including one or more virtual objects of which at least any of a position and a posture has been determined according to display reference information and which is arranged in a marker coordinate system from a perspective of a virtual camera arranged in the marker coordinate system at a same position as theimaging device 15. Object data for rendering the virtual object is acquired from object information. Moreover, the virtual object may be animated by varying the virtual object every frame or every few frames. Examples of the animation include a character of the virtual object changing a facial expression or moving around. Subsequently, processing proceeds to step S106. - In step S106, a display process is executed. The
display control unit 26 generates a composite image in which the image of the virtual space is superimposed on the captured image, and outputs the composite image to thedisplay 16 to have the composite image displayed. Subsequently, processing proceeds to step S107. - In steps S107 and S108, the composite image is preserved according to a composite image preserving instruction. In response to a user operation performed in order to issue an instruction for preserving an image (step S107), the composite
image preserving unit 28 preserves the composite image (step S108). At this point, the composite image may be preserved as a still image or as a moving image. On the other hand, if a user operation performed in order to issue an instruction for preserving the image has not been accepted, the composite image is not preserved (step S107). Subsequently, processing proceeds to step S109. - As described earlier, the processing shown in the present flow chart is executed for each frame. Therefore, the processing shown in the present flow chart is repetitively executed starting from step S101 on a regular basis until the imaging function is terminated based on a user operation or the like (step S109). Due to the execution of the image processing described above, update of display reference information of a virtual object which is updated based on the
marker 3 can be disabled once the display reference information reaches a state that is desirable to a user, a capturing position or an angle with respect to a real space can be adjusted while update is being disabled, and a composite image of the virtual object in a desired state and the real space in a desired state can be obtained. -
FIGS. 4 and5 are diagrams showing examples of a display screen of thedisplay 16 when a composite image is acquired by the image processing according to the present embodiment. More specifically,FIG. 4 is a diagram showing an example of a display screen when update of display reference information is enabled, andFIG. 5 is a diagram showing an example of a display screen when update of display reference information is disabled. - As described earlier, information stored by the display reference
information storing unit 24 is initialized upon activation of the imaging function. Accordingly, the display referenceinformation storing unit 24 does not store display reference information upon activation of the imaging function. Therefore, when a marker is not included in a captured image immediately after the imaging function of theinformation processing device 1 is activated in response to a user operation, a virtual object is not arranged in a virtual space and the captured image obtained by theimaging device 15 is displayed on thedisplay 16. - When a
marker 3 enters an imaging range of theimaging device 15 and themarker 3 is included in a captured image, the display referenceinformation updating unit 23 updates display reference information based on themarker 3 detected by thefeature detecting unit 22 and theimage generating unit 25 renders a virtual object at a position and in a posture corresponding to themarker 3. Therefore, a composite image in which the virtual object is superimposed on themarker 3 is displayed on the display 16 (refer toFIG. 4 ). - Since update of display reference information is enabled in an initial state, a user can adjust the
imaging device 15 or themarker 3 in this state to put a position and a posture of the virtual object displayed on thedisplay 16 in a desired state. In addition, once the position and the posture of the virtual object are in the desired state, the user disables update of display reference information by performing a predetermined operation. When update of display reference information is disabled, the virtual object is displayed according to last updated display reference information. Even if a position or a posture of a marker in the captured image varies or a marker is no longer included in the captured image, the virtual object is maintained in the state desired by the user (refer toFIG. 5 ). - When update of display reference information is disabled, the user can adjust a position or an angle of the
imaging device 15 to place a captured image in a desired state. In addition, by performing an operation to preserve the composite image at a timing where both a state of the virtual object and a state of the captured image reach desired states, the user can preserve a desired composite image. For example,FIG. 5 shows a state where a character of a virtual object is on top of the user's head. Besides the above, with theinformation processing device 1 according to the present embodiment, images that cannot be obtained by conventional devices in which a virtual object is always displayed on a marker can be readily obtained such as a composite image showing a character of a virtual object standing upside down on a ceiling or a composite image showing characters of a plurality of virtual objects lined up on a cake. - Variations applicable to the embodiment described above will now be described.
- The
display 16 included in theinformation processing device 1 may be a display device which can be stereoscopically viewed with the naked eye. For example, a display device adopting a lenticular system or a parallax barrier system is used so that a left-eye image and a right-eye image, which are alternately displayed in a horizontal direction, are respectively viewed separately by the left eye and the right eye. Alternatively, thedisplay 16 may be a display device capable of switching between a stereoscopic display mode which displays stereoscopically viewable images and a planar display mode which displays images in a planar view (which displays planar view images). For example, the display modes may be switched using a 3D adjustment switch (not shown). - In this case, in order to obtain a left-eye captured image and a right-eye captured image to enable stereoscopic view, an imaging device capable of stereo photography is used as the
imaging device 15. In addition, in order to also make a virtual space image a stereoscopically viewable image, theimage generating unit 25 generates a left-eye virtual image and a right-eye virtual image using two virtual cameras corresponding to the stereo camera of theimaging device 15. Furthermore, the display control unit generates a left-eye composite image and a right-eye composite image, and outputs the composite images to a stereoscopicallyviewable display 16. - Furthermore, the
imaging device 15 may be an imaging device capable of capturing images of the front and the rear of thedisplay 16 by including an outer imaging unit provided so as to be capable of capturing an image along a line of sight of a user viewing thedisplay 16 and an inner imaging unit provided so as to be capable of capturing an image of the user viewing thedisplay 16. By providing the outer imaging unit and the inner imaging unit described above, a composite image in which a virtual object and the user viewing thedisplay 16 are shown together can be generated by adjusting display reference information of the virtual object to a desired state using the outer imaging unit and themarker 3 and then switching to the inner imaging unit in a state where update of the display reference information has been disabled. - In addition, a configuration may be adopted which enables a display posture of a virtual object to be adjusted based on a posture of the
imaging device 15 when update of display reference information is disabled. In this case, a position and a posture of theimaging device 15 can be detected using an acceleration sensor, an angular velocity sensor, or the like. Theinformation processing device 1 calculates a posture of theimaging device 15 based on an output from the acceleration sensor, the angular velocity sensor, or the like to adjust a display posture of the virtual object. - For example, when display reference information is determined by placing a marker horizontally on a table, even if update of the display reference information is disabled in a state where a virtual object of a character is stood upright, a change in an imaging angle of the
imaging device 15 results in the character becoming inclined relative to a real space (the character is no longer upright). In this case, if a display posture of the virtual object is to be adjusted based on a posture of theimaging device 15, a display in which the character is stood upright can be obtained while keeping a position of the character in a virtual space fixed. - Accordingly, even in a case where, after positioning a virtual object, a posture of an imaging device varies while update of display reference information is being disabled, a posture of the virtual object can be maintained "with respect to a real space".
- In addition, while an example in which a marker coordinate system is used as display reference information has been described in the embodiment above, the display reference information may be any information as long as the information is obtained from a real space and can be used as a reference of at least any of a position and a posture of a virtual object in a virtual space. For example, the display reference information may be a captured image itself. When a captured image is used as display reference information, the
image generating unit 25 extracts a display reference of a virtual object from a captured image stored as display reference information for each frame. - While certain example systems, methods, devices and apparatuses have been described herein, it is to be understood that the appended claims are not to be limited to the systems, methods, devices and apparatuses disclosed, but on the contrary, are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (14)
- An image processing program which causes a computer to function as:updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space;image generating means for generating an image of a virtual space including the virtual object;display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; andupdate interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- The image processing program according to claim 1, further causing a computer to function as
display reference information storing means for storing display reference information that is used as a reference for at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space, wherein
the updating means updates at least any of a position and a posture of the virtual object in the virtual space by updating the display reference information stored by the display reference information storing means based on information obtained from the real space,
the image generating means generates an image of a virtual space including a virtual object arranged according to the display reference information, and
the update interrupting means interrupts an update of at least any of a position and a posture, in the virtual space, of the virtual object by interrupting an update of the display reference information by the updating means when the predetermined conditions are met. - The image processing program according to claim 2, further causing the computer to function as
feature detecting means for detecting a feature in a real space that enables at least any of a position and a posture in the real space to be identified, wherein
the updating means updates the display reference information stored by the display reference information storing means based on the feature. - The image processing program according to claim 3, further causing the computer to function as
captured image acquiring means for acquiring a captured image that is captured by an imaging device, wherein
the feature detecting means detects from the captured image a feature in a space captured in the captured image. - The image processing program according to claim 4, wherein
the display control means enables a user to view an image of the virtual space superimposed on a real space by causing the display device to display a composite image in which an image of the virtual space is superimposed on the captured image. - The image processing program according to claim 5, further causing the computer to function as
composite image preserving means for preserving the composite image. - The image processing program according to claim 6, wherein
the composite image preserving means preserves the composite image in response to a user operation performed in order to issue an instruction for preserving an image. - The image processing program according to any one of claims 1 to 7, wherein the update interrupting means interrupts an update of the display reference information by the updating means on a condition that a predetermined operation is performed by a user.
- An image processing device comprising:updating means for updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space;image generating means for generating an image of a virtual space including the virtual object;display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; andupdate interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- The image processing device according to claim 9, further comprising display reference information storing means for storing display reference information that is used as a reference for at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space, wherein
the updating means updates at least any of a position and a posture of the virtual object in the virtual space by updating the display reference information stored by the display reference information storing means based on information obtained from the real space,
the image generating means generates an image of a virtual space including a virtual object arranged according to the display reference information, and
the update interrupting means interrupts an update of at least any of a position and a posture, in the virtual space, of the virtual object by interrupting an update of the display reference information by the updating means when the predetermined conditions are met. - An image processing system comprising:a component attached with a feature that enables a position and a posture relative to an imaging device to be identified by being captured by the imaging device; andan image processing device connected to the imaging device,
whereinthe image processing device includes:updating means for updating at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space based on the component which is arranged in a real space and which is captured by the imaging device;image generating means for generating an image of a virtual space including the virtual object;display control means for causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; andupdate interrupting means for interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met. - The image processing system according to claim 11, wherein the image processing device further includes
display reference information storing means for storing display reference information that is used as a reference for at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space,
the updating means updates at least any of a position and a posture of the virtual object in the virtual space by updating the display reference information stored by the display reference information storing means based on information obtained from the real space,
the image generating means generates an image of a virtual space including a virtual object arranged according to the display reference information, and
the update interrupting means interrupts an update of at least any of a position and a posture, in the virtual space, of the virtual object by interrupting an update of the display reference information by the updating means when the predetermined conditions are met. - An image processing method in which a computer executes:an updating step of updating, based on information obtained from a real space, at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space;an image generating step of generating an image of a virtual space including the virtual object;a display control step of causing a display device to display an image so that the image of the virtual space is superimposed on the real space so as to be viewed by a user; andan update interrupting step of interrupting an update of at least any of a position and a posture, in the virtual space, of the virtual object when predetermined conditions are met.
- The image processing method according to claim 13, wherein a computer further executes a display reference information storing step of storing display reference information that is used as a reference for at least any of a position and a posture, in a virtual space, of a virtual object arranged in the virtual space,
in the updating step, at least any of a position and a posture of the virtual object in the virtual space is updated by updating the display reference information stored in the display reference information storing step based on information obtained from the real space,
in the image generating step, an image of a virtual space including a virtual object arranged according to the display reference information is generated, and
in the update interrupting step, an update of at least any of a position and a posture, in the virtual space, of the virtual object is interrupted by interrupting an update of the display reference information in the updating step when the predetermined conditions are met.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012220353A JP6021568B2 (en) | 2012-10-02 | 2012-10-02 | Image processing program, image processing apparatus, image processing system, and image processing method |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2717227A2 true EP2717227A2 (en) | 2014-04-09 |
EP2717227A3 EP2717227A3 (en) | 2018-02-07 |
Family
ID=48703255
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP13174670.3A Pending EP2717227A3 (en) | 2012-10-02 | 2013-07-02 | Image processing program, image processing device, image processing system, and image processing method |
Country Status (3)
Country | Link |
---|---|
US (1) | US9478068B2 (en) |
EP (1) | EP2717227A3 (en) |
JP (1) | JP6021568B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015182134A1 (en) * | 2014-05-28 | 2015-12-03 | Sony Corporation | Improved setting of virtual illumination environment |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170061700A1 (en) * | 2015-02-13 | 2017-03-02 | Julian Michael Urbach | Intercommunication between a head mounted display and a real world object |
JP6242473B1 (en) * | 2016-12-22 | 2017-12-06 | 株式会社コロプラ | Method for providing virtual space, program for causing computer to execute the method, and information processing apparatus for executing the program |
JP7079287B2 (en) | 2019-11-07 | 2022-06-01 | 株式会社スクウェア・エニックス | Viewing system, model configuration device, control method, program and recording medium |
JP2022079362A (en) | 2020-11-16 | 2022-05-26 | 任天堂株式会社 | Information processing system, information processing program, information processor, and method for processing information |
JP7257370B2 (en) | 2020-11-18 | 2023-04-13 | 任天堂株式会社 | Information processing program, information processing device, information processing system, and information processing method |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH1188730A (en) * | 1997-09-09 | 1999-03-30 | Olympus Optical Co Ltd | Electronic camera |
JP4613449B2 (en) * | 2001-06-01 | 2011-01-19 | セイコーエプソン株式会社 | Output service providing system, virtual object management terminal, moving object, virtual object management terminal program, moving object program, and output service providing method |
JP4401727B2 (en) * | 2003-09-30 | 2010-01-20 | キヤノン株式会社 | Image display apparatus and method |
JP4835243B2 (en) * | 2006-04-13 | 2011-12-14 | ブラザー工業株式会社 | Image composition apparatus and image composition program |
JP5111210B2 (en) * | 2008-04-09 | 2013-01-09 | キヤノン株式会社 | Image processing apparatus and image processing method |
US20110173550A1 (en) * | 2010-01-13 | 2011-07-14 | Microsoft Corporation | Data driven user interface animation |
JP5149939B2 (en) * | 2010-06-15 | 2013-02-20 | 任天堂株式会社 | Information processing program, information processing apparatus, information processing system, and information processing method |
JP4869430B1 (en) * | 2010-09-24 | 2012-02-08 | 任天堂株式会社 | Image processing program, image processing apparatus, image processing system, and image processing method |
US8854356B2 (en) * | 2010-09-28 | 2014-10-07 | Nintendo Co., Ltd. | Storage medium having stored therein image processing program, image processing apparatus, image processing system, and image processing method |
JP5827007B2 (en) * | 2010-10-15 | 2015-12-02 | 任天堂株式会社 | Game program, image processing apparatus, image processing system, and image processing method |
JP5480777B2 (en) * | 2010-11-08 | 2014-04-23 | 株式会社Nttドコモ | Object display device and object display method |
JP5697487B2 (en) * | 2011-02-25 | 2015-04-08 | 任天堂株式会社 | Image processing system, image processing method, image processing apparatus, and image processing program |
-
2012
- 2012-10-02 JP JP2012220353A patent/JP6021568B2/en active Active
-
2013
- 2013-07-02 EP EP13174670.3A patent/EP2717227A3/en active Pending
- 2013-07-03 US US13/934,736 patent/US9478068B2/en active Active
Non-Patent Citations (1)
Title |
---|
HIROKAZU KATO; MARK BILLINGHURST; KOICHI ASANO; KEIHACHIRO TACHIBANA, AN AUGMENTED REALITY SYSTEM AND ITS CALIBRATION BASED ON MARKER TRACKING. JOURNAL OF THE VIRTUAL REALITY SOCIETY OF JAPAN, vol. 4, no. 4, 1999, pages 606 - 616 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015182134A1 (en) * | 2014-05-28 | 2015-12-03 | Sony Corporation | Improved setting of virtual illumination environment |
US10204404B2 (en) | 2014-05-28 | 2019-02-12 | Sony Corporation | Image processing device and image processing method |
Also Published As
Publication number | Publication date |
---|---|
US20140092133A1 (en) | 2014-04-03 |
US9478068B2 (en) | 2016-10-25 |
JP6021568B2 (en) | 2016-11-09 |
EP2717227A3 (en) | 2018-02-07 |
JP2014071877A (en) | 2014-04-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9602740B2 (en) | Computer-readable recording medium recording information processing program, information processing apparatus, information processing system, and information processing method for superimposing a virtual image on a captured image of real space | |
JP5697487B2 (en) | Image processing system, image processing method, image processing apparatus, and image processing program | |
US9478068B2 (en) | Computer-readable medium, image processing device, image processing system, and image processing method | |
US10062211B2 (en) | Computer-readable recording medium recording information processing program, information processing apparatus, information processing system, and information processing method | |
JP6618681B2 (en) | Information processing apparatus, control method and program therefor, and information processing system | |
US9424689B2 (en) | System,method,apparatus and computer readable non-transitory storage medium storing information processing program for providing an augmented reality technique | |
US9916687B2 (en) | Computer-readable recording medium recording information processing program, information processing apparatus, information processing system, and information processing method | |
EP2477160A1 (en) | Apparatus and method for providing augmented reality perceived through a window | |
US20140125701A1 (en) | Computer-readable medium, information processing apparatus, information processing system and information processing method | |
US11244145B2 (en) | Information processing apparatus, information processing method, and recording medium | |
KR102450236B1 (en) | Electronic apparatus, method for controlling thereof and the computer readable recording medium | |
EP3906527A1 (en) | Image bounding shape using 3d environment representation | |
JP2016122392A (en) | Information processing apparatus, information processing system, control method and program of the same | |
CN112513785A (en) | Augmented reality viewer with automatic surface selection placement and content orientation placement | |
KR101308184B1 (en) | Augmented reality apparatus and method of windows form | |
EP3346375B1 (en) | Program, recording medium, content provision device, and control method | |
US20200211275A1 (en) | Information processing device, information processing method, and recording medium | |
US9158984B2 (en) | Computer-readable recording medium recording program for image processing, information processing apparatus, information processing system, and image processing method | |
JP2019032713A (en) | Information processing device, information processing method, and program | |
CN115698923A (en) | Information processing apparatus, information processing method, and program | |
JP6514386B1 (en) | PROGRAM, RECORDING MEDIUM, AND IMAGE GENERATION METHOD | |
CN117475112A (en) | Stereoscopic depth marking | |
CN117940963A (en) | Display device, control method for display device, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 19/00 20110101AFI20171221BHEP |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20180806 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20210705 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |