WO2014002346A1 - 映像処理装置、映像処理方法、および映像処理システム - Google Patents

映像処理装置、映像処理方法、および映像処理システム Download PDF

Info

Publication number
WO2014002346A1
WO2014002346A1 PCT/JP2013/002605 JP2013002605W WO2014002346A1 WO 2014002346 A1 WO2014002346 A1 WO 2014002346A1 JP 2013002605 W JP2013002605 W JP 2013002605W WO 2014002346 A1 WO2014002346 A1 WO 2014002346A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
marker
proximity
interaction
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2013/002605
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
良徳 大橋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Computer Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Computer Entertainment Inc filed Critical Sony Computer Entertainment Inc
Priority to CN201380032960.XA priority Critical patent/CN104380347B/zh
Priority to US14/406,919 priority patent/US9639989B2/en
Priority to EP13810089.6A priority patent/EP2869274A4/en
Publication of WO2014002346A1 publication Critical patent/WO2014002346A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor
    • H04N5/7475—Constructional details of television projection apparatus
    • H04N5/7491—Constructional details of television projection apparatus of head mounted projectors

Definitions

  • the present invention relates to a video processing device, a video processing method, and a video processing system.
  • HMD head-mounted displays
  • an AR Augmented Reality
  • CG Computer Graphics
  • Technology is also entering the practical stage.
  • specific information such as a barcode is recognized and an image is generated in association with the information.
  • the present invention has been made in view of these problems, and an object of the present invention is to provide a technique for producing an interaction between images generated in association with a specific object.
  • an aspect of the present invention is a video processing apparatus.
  • This apparatus includes a field of view of a user wearing an optically transmissive HMD provided in an optically transmissive HMD that presents an image observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space.
  • An AR image generation unit that generates an AR image to be presented to the optically transmissive HMD using a subject imaged by an imaging element that captures a subject in an area including the image as a marker, and a first AR image associated with the first marker
  • the proximity acquisition unit that acquires the proximity in the virtual three-dimensional space between the second marker or the second AR image linked to the second marker and the proximity acquisition unit
  • An interaction rendering unit that calculates an interaction between the first AR image and the second AR image based on the proximity.
  • the AR image generation unit changes at least one of the first AR image and the second AR image according to the interaction calculated by the interaction rendering unit.
  • Another aspect of the present invention is a video processing method.
  • This method includes an optically transmissive HMD that presents an image observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space, and a region that includes a visual field of a user wearing the optically transmissive HMD.
  • Still another aspect of the present invention is a video processing system.
  • the system includes an optically transmissive HMD that presents an image observed when a three-dimensional image in a virtual three-dimensional space is projected into a real space, and the optically transmissive HMD.
  • An image pickup device that picks up an image of a subject in an area including the field of view of the user who wears the image, and an AR image generation that generates an AR (Augmented Reality) image that is presented on the optically transmissive HMD using the subject imaged by the image pickup device as a marker
  • the degree of proximity in a virtual three-dimensional space between the first AR image linked to the first marker and the second AR image linked to the second marker or the second marker Based on the proximity acquired by the proximity acquisition unit and the proximity acquisition unit, the interaction between the first AR image or the second AR image is calculated. And an interaction for the directing department.
  • the AR image generation unit changes at least one of the first AR image and the second AR image according to the interaction calculated by the interaction rendering unit.
  • Still another aspect of the present invention is a program for causing a computer to implement each step of the above method.
  • This program may be provided as part of the firmware incorporated in the device in order to perform basic control of hardware resources such as video and audio decoders.
  • This firmware is stored in a semiconductor memory such as a ROM (Read Only Memory) or a flash memory in the device.
  • a computer-readable recording medium storing the program may be provided, and the program may be transmitted through a communication line.
  • the embodiment of the present invention produces a virtual interaction between 3D images based on the proximity when a 3D image moved by a user operation approaches another 3D image. Reflect in the 3D image. Further, the interaction is, for example, contact or collision between the three-dimensional images, and the three-dimensional image is changed and presented according to attributes such as virtual speed and acceleration of the three-dimensional image before and after the interaction. .
  • FIG. 1 is a diagram schematically showing an overall configuration of a video presentation system 100 according to an embodiment.
  • the video presentation system 100 according to the embodiment includes a stereoscopic video observation device 200, a three-dimensional monitor 300, and an information processing apparatus 400.
  • the stereoscopic image observation device 200 is an optical transmission type HMD.
  • the stereoscopic image observation device 200 may include an optical shutter (not shown) for observing a three-dimensional monitor 300 described later.
  • the optical shutter opens and closes the left and right shutters in synchronization with the parallax image switching of the three-dimensional monitor 300. More specifically, when the left-eye parallax image is displayed on the 3D monitor 300, the right-eye shutter is closed and the left-eye shutter is opened, so that the user wearing the stereoscopic image observation device 200 can use the left-eye. Presents a parallax image.
  • the left-eye shutter is closed and the right-eye shutter is opened, and the right-eye parallax image is presented to the user.
  • the optical shutter can be realized using, for example, a known liquid crystal shutter.
  • the stereoscopic image observation device 200 receives a synchronization signal for shutter switching.
  • the synchronization signal is wirelessly transmitted from a signal transmission unit (not shown) provided in the three-dimensional monitor 300 or the information processing apparatus 400 using, for example, infrared light.
  • the 3D monitor 300 displays a stereoscopic image by a frame sequential method. Since the left and right eyes of a human are about 6 cm apart, parallax occurs between the image seen from the left eye and the image seen from the right eye. The human brain is said to use parallax images perceived by the left and right eyes as one piece of information for recognizing depth. For this reason, when a parallax image perceived by the left eye and a parallax image perceived by the right eye are projected onto the respective eyes, it is recognized as a video having a depth by humans.
  • the three-dimensional monitor 300 alternately displays the left-eye parallax image and the right-eye parallax image in a time-sharing manner.
  • the three-dimensional monitor 300 can be realized using a known presentation device such as a liquid crystal television, a plasma display, or an organic EL monitor.
  • the information processing apparatus 400 acquires a stereoscopic video to be presented by the video presentation system 100 and the synchronization signal described above.
  • Examples of the information processing apparatus 400 include a stationary game machine and a portable game machine.
  • the information processing apparatus 400 generates a stereoscopic video and a synchronization signal using a built-in processor, and acquires a stereoscopic video from another information processing apparatus such as a server via a network interface (not shown).
  • FIG. 2 is a diagram schematically illustrating an example of the appearance of the stereoscopic image observation device 200 according to the embodiment.
  • the stereoscopic video observation device 200 includes a presentation unit 202 that presents a stereoscopic video, an imaging element 204, and a housing 206 that houses various modules.
  • the presentation unit 202 includes an optically transmissive HMD that presents a stereoscopic image to the user's eyes, and a liquid crystal shutter that changes the transmittance of external light that passes through the optically transmissive HMD.
  • the imaging element 204 images a subject in an area including the field of view of the user wearing the stereoscopic video observation device 200. For this reason, the image sensor 204 is installed so as to be arranged around the user's eyebrows when the user wears the stereoscopic image observation device 200.
  • the image sensor 204 can be realized by using a known solid-state image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor.
  • the housing 206 plays a role of a frame in the glasses-shaped stereoscopic image observation device 200 and houses various modules (not shown) used by the stereoscopic image observation device 200.
  • the modules used by the stereoscopic image observation device 200 include an optical engine including a hologram light guide plate for realizing an optically transmissive HMD, a driver and a synchronization signal receiver for driving a liquid crystal shutter, and other Wi-Fi (registered trademark).
  • a communication module such as a module, an electronic compass, an acceleration sensor, a tilt sensor, a GPS (Global Positioning System) sensor, and an illuminance sensor. These modules are examples, and the stereoscopic image observation device 200 does not necessarily need to be equipped with all of these modules. Which module is mounted may be determined according to the usage scene assumed by the stereoscopic video observation device 200.
  • FIG. 2 is a diagram illustrating a glasses-type stereoscopic image observation device 200.
  • the shape of the stereoscopic image observation device 200 may be various variations such as a hat shape, a belt shape that is fixed around the user's head, and a helmet shape that covers the entire user's head. Those skilled in the art can easily understand that the stereoscopic image observation device 200 having a shape is also included in the embodiment of the present invention.
  • FIG. 3 is a diagram schematically showing an internal configuration of the video processing apparatus 500 according to the embodiment.
  • the video processing apparatus 500 according to the embodiment is realized as a part of the information processing apparatus 400 described above.
  • it may be realized in a server that generates a stereoscopic video to be transmitted to the information processing apparatus 400 via a network such as the Internet, or may be incorporated in the stereoscopic video observation device 200 or the three-dimensional monitor 300.
  • the video processing device 500 may be an independent device. In the following description, it is assumed that the video processing apparatus 500 according to the embodiment is realized as a part of the information processing apparatus 400 described above.
  • the video processing apparatus 500 includes a content execution unit 510, a video generation unit 520, an output unit 530, an attribute management unit 540, a marker identification unit 550, a proximity acquisition unit 560, and an interaction effect production unit 570.
  • the content execution unit 510 executes content including video to be processed by the video processing device 500, such as a game application or an application that provides map information.
  • the content execution unit 510 sets a three-dimensional object to be displayed on the stereoscopic video observation device 200 in a virtual three-dimensional space by executing the content.
  • the “object” in this specification is a set of polygons that are rendering elements in the three-dimensional CG, and each polygon has a common coordinate axis and has a meaning as a group. Specifically, it is a collection of polygons representing objects such as trees, houses, cars, etc., and a collection of polygons representing people and creatures such as characters to be operated by the user. Since a set of polygons constituting the “object” has a common coordinate axis, the position and orientation can be specified in a virtual three-dimensional space.
  • each coordinate of the polygon constituting the “tree” object can be obtained by calculating a coordinate transformation of rotation about the central axis.
  • the video generation unit 520 generates a stereoscopic video to be displayed on the stereoscopic video observation device 200 based on the three-dimensional object set by the content execution unit 510 in the virtual three-dimensional space. Specifically, the content execution unit 510 generates a left-eye parallax image and a right-eye parallax image of a three-dimensional object set in a virtual three-dimensional space.
  • the marker specifying unit 550 acquires a video taken by the image pickup device 204 provided in the stereoscopic video observation device 200 and specifies a marker in the video.
  • the “marker” is information used by the object acquisition unit 502 that generates an object, for example, and is information that can specify the position of an image to be generated in a virtual three-dimensional space.
  • Specific examples of the marker include, for example, an image or a moving image displayed on a monitor, an image printed on a card or paper, an object having a specific shape such as a circle or a star, a specific color, a silhouette or face of a person or an animal, GPS For example, the position indicated by the specific position information.
  • the “orientation” of the marker in the three-dimensional space can be specified.
  • the position where the image is to be generated but also the direction of the image to be generated can be specified.
  • the “AR image” is an object associated with a marker, and an object whose position, orientation, and inclination change in conjunction with a change in the position, orientation, or inclination of the marker.
  • an AR image of a “ball” moving with a velocity vector v 1 is associated with a barcode printed on paper as a marker.
  • the video generation unit 520 includes an object generation unit 524, an AR image generation unit 526, and a control unit 522.
  • the control unit 522 controls the operations of the object generation unit 524 and the AR image generation unit 526 in an integrated manner.
  • the control unit 522 acquires information on the three-dimensional object set in the virtual three-dimensional space by the content execution unit 510 and the line-of-sight direction for generating the video to be presented to the user, and sends the parallax image to the object generation unit 524. Is generated.
  • the object generation unit 524 generates a parallax image under the control of the control unit 522 and outputs the parallax image to the output unit 530 described later.
  • FIG. 4 is a diagram illustrating an image including a marker 710 and an AR image 712 linked to the marker.
  • FIG. 4 illustrates a marker 710 made up of a rod-like gripping part gripped by the user's arm 702 and a spherical object. Of course, the user's arm 702 is real.
  • the marker 710 is also an actual object.
  • an orthogonal coordinate system 602 with an x-axis, a y-axis, and a z-axis is set in the three-dimensional space.
  • the orthogonal coordinate system 602 matches the coordinate system set by the object generation unit 524 and the AR image generation unit 526 in a virtual three-dimensional space.
  • the orthogonal coordinate system 602 it is preferable to set the orthogonal coordinate system 602 so that the xy plane where the x-axis and the y-axis of the orthogonal coordinate system 602 are stretched and the display area of the three-dimensional monitor 300 are parallel to each other. More specifically, the origin O of the orthogonal coordinate system 602 is preferably set so that the display area of the three-dimensional monitor 300 overlaps the xy plane where the x axis and the y axis of the orthogonal coordinate system 602 are stretched.
  • the z-axis of the orthogonal coordinate system 602 indicates that the viewpoint side of the user who wears the stereoscopic image observation device 200 has a negative z coordinate value with respect to the display area of the three-dimensional monitor 300 and the viewpoint with respect to the display area of the three-dimensional monitor 300 It is preferable to set the opposite side to a positive z coordinate value.
  • the AR image 712 is an object expressing flame. Since the AR image 712 generated by the AR image generation unit 526 in association with the marker 710 is also an object, the position coordinates are determined.
  • the AR image generation unit 526 generates the AR image 712 according to the coordinate system 704 set with the point determined according to the position of the marker 710 associated with the AR image 712 as an origin instead of the orthogonal coordinate system 602 described above. For this reason, when the user changes the position, orientation, and inclination of the marker 710, the coordinate system 704 that the AR image 712 uses as a reference also changes. When the coordinate system 704 is changed, the position and orientation of the AR image 712 are also changed accordingly. It should be noted that the origin of the coordinate system 704 on which the AR image 712 is a reference does not necessarily have to overlap with the related marker.
  • the proximity acquisition unit 560 calculates and acquires the proximity between two different AR images, that is, the distance between the two AR images.
  • FIG. 5 is a diagram for explaining the proximity between two different AR images, and is a diagram illustrating an example of an image presented to a user wearing the stereoscopic image observation device 200.
  • FIG. 5 shows three types of markers 710, 714, and 300 and three types of AR images 716, 718, and 720 associated with them.
  • the marker 714 is an image printed on paper
  • the marker 300 has a display area on the three-dimensional monitor 300 as a marker.
  • the AR image 716 associated with the marker 710 is a brilliant virtual blade
  • the AR image 718 associated with the marker 714 is a conical object.
  • the AR image 720 associated with the marker 300 is a spherical object that moves from the 3D monitor 300 and moves away from the 3D monitor 300.
  • Each AR image has a distance measurement reference point determined for measuring a distance between the AR images.
  • the distance measurement reference point of the AR image 716 is indicated by reference numeral 722.
  • distance measurement reference points of the AR image 718 and the AR image 720 are denoted by reference numerals 724 and 726, respectively.
  • the distance measurement reference point may be any point as long as it exists on the surface of the polygon constituting the AR image 716 or in the vicinity of the AR image 716.
  • the proximity acquisition unit 560 includes a plurality of distance measurement reference points. A distance measurement reference point is set at the center of gravity of the polygon. As a result, even if there are a plurality of polygons constituting the object, the position coordinates of the object can be represented by one coordinate, so that it can be expected to reduce the calculation cost and improve the processing speed.
  • Each AR image also has a collision determination reference radius determined for determining a collision between the AR images.
  • the collision determination reference radius of the AR image 716 is r 1 and the collision determination reference radius of the AR image 720 is r 2 .
  • the coordinates of the distance measurement reference point of the AR image 716 are (x 1 , y 1 , z 1 ), and the coordinates of the distance measurement reference point of the AR image 720 are (x 2 , y 2 , z 2 ).
  • the Euclidean distance L between the distance measurement reference point of the AR image 716 and the distance measurement reference point of the AR image 720 is equal to the collision determination reference radius r 1 of the AR image 716 and AR. It means equal to the sum of the image 720 collision determination reference radius r 2.
  • the proximity acquisition unit 560 considers that the AR image 716 and the AR image 720 are “in contact”.
  • the proximity acquisition unit 560 considers that the AR image 716 and the AR image 720 are “separated”.
  • the proximity acquisition unit 560 regards the AR image 716 and the AR image 720 as “overlapping”.
  • the value of E acquired by the proximity acquisition unit 560 can be used as a value expressing the proximity between the two AR images.
  • the collision determination reference radius of each AR image may be a value that reflects the size of the AR image in the three-dimensional space. For example, based on the distance from the distance measurement reference point to the polygon constituting the AR image. Should be set. More specifically, the average value of the distance from the distance measurement reference point to each polygon constituting the AR image, the maximum value or the minimum value of the distance, and the like may be adopted as the collision determination reference radius of the AR image.
  • the interaction effect unit 570 calculates an interaction that occurs between two different AR images based on the proximity acquired by the proximity acquisition unit 560.
  • the “interaction” is a concept including a virtual physical or chemical interaction generated between the AR images and a visual effect (Visual FX; VFX) using CG.
  • the former are the conservation of momentum in the collision between rigid bodies, the phenomenon of both combining if there is a collision between soft bodies, or the other AR image that was not burned when one AR image is burning. It is a phenomenon that the flame moves. In the latter example, when one AR image is a blade that cuts everything, the effect that another AR image touching the AR image is cut, and the attribute that one AR image is transparent to everything When added, it has the effect of overlapping without interfering with the other AR image.
  • the attribute management unit 540 manages attributes including information indicating virtual physical characteristics and visual effects given to each AR image generated by the AR image generation unit 526.
  • the attribute managed by the attribute management unit 540 changes in accordance with the interaction calculated by the interaction rendering unit 570. Specifically, whether the physical characteristics included in the attributes managed by the attribute management unit 540 are the position coordinates, velocity vector, acceleration vector, mass, AR image rigidity, color, temperature, and combustion in the three-dimensional space of the AR image. Whether or not light is emitted, the distance measurement reference point and the collision reference radius described above are included. These physical characteristics are examples, and the attribute management unit 540 does not necessarily have to manage all of these physical characteristics. What physical characteristics are to be managed may be determined according to the usage scene assumed by the content reproduced by the content execution unit 510.
  • the interaction rendering unit 570 changes the physical characteristics of the two AR images based on the proximity acquired by the proximity acquisition unit 560. More specifically, the interaction effect unit 570 calculates an interaction based on the proximity acquired by the proximity acquisition unit 560, and the attribute management unit 540 determines the interaction calculated by the interaction effect unit 570. Accordingly, the attribute assigned to the AR image is changed.
  • the AR image generation unit 526 changes the image generated when the attribute of the AR image changes.
  • an AR image 720 that is a spherical object that moves from the 3D monitor 300 and moves away from the 3D monitor 300 is a conical object that is stationary in the 3D space.
  • the proximity acquisition unit 560 calculates the proximity between the AR image 720 and the AR image 718 and determines that they are in contact with each other.
  • the interaction rendering unit 570 applies the momentum conservation law and the kinetic energy conservation law to the AR image 720 based on the mass and velocity vectors that are the attributes and the mass and velocity vectors that are the attributes of the AR image 718. Then, the velocity vectors of both after the collision are calculated.
  • Attribute management unit 540 rewrites the velocity vector in the attributes of AR image 718 and AR image 720 with the velocity vector calculated by interaction rendering unit 570.
  • the AR image generation unit 526 changes the AR image 718 and the AR image 720 so that the AR image 718 and the AR image 720 move according to the velocity vector rewritten by the attribute management unit 540.
  • FIG. 6 is a diagram illustrating an interaction between two different AR images, and is a diagram illustrating an example of an image presented to a user wearing the stereoscopic image observation device 200.
  • the AR image 712 representing the flame associated with the marker 710 is in close proximity to the lead of the AR image 730 representing the bomb associated with the paper marker 728.
  • the fire guide line of the AR image 730 is not lit, but when the user moves the marker 710, if the AR image 712 is sufficiently close to the fire guide line of the AR image 730, the proximity acquisition unit 560 Assume that you have determined.
  • the interaction effect unit 570 considers that the lead wire of the AR image 730 has been ignited, and the attribute management unit 540 changes the attribute of the AR image 730 from “not burning” to “burning”. As a result, the AR image generation unit 526 changes the AR image 730 so that the fire guide wire of the AR image 730 is lit and becomes shorter with time.
  • the AR image generation unit 526 generates the AR image 730 representing the bomb as if it was divided into several fragments by the explosion.
  • the AR image generation unit 526 may newly generate another AR image from one AR image.
  • newly generating another AR image from one AR image there is a case of generating an image in which a bullet is fired from an AR image representing a pistol.
  • the user moves the marker 710 at a high speed in the example shown in FIG. At this time, the movement vector and the acceleration vector of the AR image 712 change according to the movement of the marker 710.
  • the AR image generation unit 526 May generate a new AR image different from the AR image 712. As a result, it is possible to produce an effect that the flame splits and pops out in accordance with the user's operation.
  • the newly generated AR image may be associated with the original marker, or may be another marker using position information. Good.
  • FIG. 7 is a flowchart showing the flow of interaction effect processing by the video processing apparatus 500 according to the embodiment. The processing in this flowchart starts when the video processing apparatus 500 is turned on, for example.
  • the proximity acquisition unit 560 acquires the position coordinates of the first AR image, the distance measurement reference point, and the collision reference radius associated with the first marker specified by the marker specifying unit 550 (S2). The proximity acquisition unit 560 further acquires the position coordinates of the second AR image, the distance measurement reference point, and the collision reference radius associated with the second marker specified by the marker specifying unit 550 (S4).
  • the proximity acquisition unit 560 acquires the proximity between the first AR image and the second AR image according to the above-described equation (1) (S6).
  • the interaction effect unit 570 calculates the interaction between the first AR image and the second AR image based on the proximity acquired by the proximity acquisition unit 560 (S8).
  • the attribute management unit 540 changes the attributes of the first AR image and the second AR image based on the interaction calculated by the interaction rendering unit 570 (S10).
  • the AR image generation unit 526 reflects the attribute change made by the attribute management unit 540 in the first AR image and the second AR image (S12). When the AR image generation unit 526 reflects the attribute change in the image, the processing in this flowchart ends.
  • the usage scenes of the video presentation system 100 configured as described above are as follows.
  • a user who wears the stereoscopic video observation device 200 and uses the video presentation system 100 moves the AR image associated with the marker by moving the marker and brings it closer to another AR image.
  • the AR image generation unit 526 reflects the interaction by the interaction effect unit 570 in the AR image.
  • the content of the content reproduced by the AR image or the content execution unit 510 is changed according to the proximity of the AR image to a different marker, the proximity of the collision, and the attributes such as the speed and acceleration of the AR image at that time. It can be reflected.
  • the acceleration of the marker it is possible to express the whip AR image associated with the marker according to the movement of the marker. It is also possible to produce a special visual effect added to the AR image.
  • the video presentation system 100 it is possible to provide a technique for producing an interaction between images generated in association with a specific object. More specifically, it is possible to reflect an application in which a plurality of AR images influence each other and the movement of the marker in the AR image.
  • the distance measurement reference point is not limited to the coordinates of the center of gravity of the polygon.
  • the position coordinates of one polygon among the polygons constituting the object may be used as a representative.
  • the polygon used as a representative may be adaptively changed according to the positional relationship between the objects.
  • the position coordinates of the polygon with the shortest distance from another object may be adopted as the object coordinates.
  • the interaction when the AR images are in contact with each other has been mainly described.
  • the interaction is calculated even when the AR images are not in contact with each other.
  • the AR image representing the fan and the AR image representing the piece of paper are sufficiently close to each other even if they do not touch each other.
  • the interaction that the AR image representing a piece of paper blows away is produced.
  • the interaction is determined according to the proximity of the AR images.
  • the proximity acquisition unit 560 describes the case of acquiring the proximity between two different AR images.
  • the proximity calculated by the proximity acquisition unit 560 is not limited to the proximity between two different AR images.
  • the degree of proximity between an AR image and a marker may be obtained. Hereinafter, this case will be described.
  • FIG. 8 is a diagram for explaining the proximity between a certain AR image and a marker.
  • the user manipulates the marker 710 and tries to hit the AR 302 associated with the marker 710 against the character 302 displayed in the display area of the three-dimensional monitor 300.
  • the example shown in FIG. 8 is an example of a game application, and it is assumed that the user hits the AR image 732 representing lightning against the character 302 that is an enemy character.
  • the display area of the three-dimensional monitor 300 is also a marker.
  • the proximity acquisition unit 560 acquires the proximity between the AR image 732 and the character 302 that is a partial image displayed in the display area of the three-dimensional monitor 300 that is a marker. That is, the proximity acquisition unit 560 calculates the proximity between the AR image and the marker.
  • the content reproduced by the content execution unit 510 is a conventional two-dimensional content
  • the video displayed on the display area of the three-dimensional monitor 300 is a conventional two-dimensional video
  • 100 video presentation system 200 stereoscopic image observation device, 202 presentation unit, 204 image sensor, 206 housing, 300 3D monitor, 302 character, 400 information processing device, 500 video processing device, 502 object acquisition unit, 510 content execution unit , 520 video generation unit, 522 control unit, 524 object generation unit, 526 AR image generation unit, 530 output unit, 540 attribute management unit, 550 marker identification unit, 560 proximity acquisition unit, 570 interaction directing unit, 602 orthogonal coordinates System, 704 coordinate system.
  • the present invention can be used for a video processing device, a video processing method, and a video processing system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Optics & Photonics (AREA)
  • Processing Or Creating Images (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Controls And Circuits For Display Device (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2013/002605 2012-06-29 2013-04-17 映像処理装置、映像処理方法、および映像処理システム Ceased WO2014002346A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201380032960.XA CN104380347B (zh) 2012-06-29 2013-04-17 视频处理设备、视频处理方法和视频处理系统
US14/406,919 US9639989B2 (en) 2012-06-29 2013-04-17 Video processing device, video processing method, and video processing system
EP13810089.6A EP2869274A4 (en) 2012-06-29 2013-04-17 VIDEO PROCESSING DEVICE, VIDEO PROCESSING METHOD AND VIDEO PROCESSING SYSTEM

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2012147344A JP5891125B2 (ja) 2012-06-29 2012-06-29 映像処理装置、映像処理方法、および映像処理システム
JP2012-147344 2012-06-29

Publications (1)

Publication Number Publication Date
WO2014002346A1 true WO2014002346A1 (ja) 2014-01-03

Family

ID=49782553

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2013/002605 Ceased WO2014002346A1 (ja) 2012-06-29 2013-04-17 映像処理装置、映像処理方法、および映像処理システム

Country Status (5)

Country Link
US (1) US9639989B2 (enExample)
EP (1) EP2869274A4 (enExample)
JP (1) JP5891125B2 (enExample)
CN (2) CN104380347B (enExample)
WO (1) WO2014002346A1 (enExample)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105872526A (zh) * 2015-01-21 2016-08-17 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5891125B2 (ja) * 2012-06-29 2016-03-22 株式会社ソニー・コンピュータエンタテインメント 映像処理装置、映像処理方法、および映像処理システム
KR101483054B1 (ko) * 2014-05-31 2015-01-16 연세대학교 산학협력단 상호작용을 지원하는 모바일 기반 증강현실 제작 시스템 및 방법
KR101618004B1 (ko) * 2015-01-27 2016-05-09 가톨릭대학교 산학협력단 가상 현실 기반의 인터랙티브 컨텐츠 제공 장치 및 그 방법
JP6336930B2 (ja) 2015-02-16 2018-06-06 富士フイルム株式会社 仮想オブジェクト表示装置、方法、プログラムおよびシステム
GB2535727A (en) * 2015-02-25 2016-08-31 Bae Systems Plc Interactive information system
US20160253891A1 (en) * 2015-02-27 2016-09-01 Elwha Llc Device that determines that a subject may contact a sensed object and that warns of the potential contact
US10038838B2 (en) 2015-05-29 2018-07-31 Hover Inc. Directed image capture
EP3286619B1 (en) * 2015-06-29 2024-08-21 Essilor International A scene image analysis module
US9922463B2 (en) 2015-08-07 2018-03-20 Microsoft Technology Licensing, Llc Virtually visualizing energy
US20170039986A1 (en) * 2015-08-07 2017-02-09 Microsoft Technology Licensing, Llc Mixed Reality Social Interactions
US20170038829A1 (en) * 2015-08-07 2017-02-09 Microsoft Technology Licensing, Llc Social interaction for remote communication
EP3342189B1 (en) 2015-08-24 2020-10-14 PCMS Holdings, Inc. Systems and methods for enhancing augmented reality experience with dynamic output mapping
US10545717B2 (en) 2015-10-08 2020-01-28 Pcms Holdings, Inc. Methods and systems of automatic calibration for dynamic display configurations
US10788966B2 (en) * 2016-02-10 2020-09-29 Disney Enterprises, Inc. Systems and methods for interacting with a virtual interface
US10616662B2 (en) 2016-02-10 2020-04-07 Disney Enterprises, Inc. Systems and methods to provide video and control signals over an internet protocol communications network
US10163198B2 (en) 2016-02-26 2018-12-25 Samsung Electronics Co., Ltd. Portable image device for simulating interaction with electronic device
US10169918B2 (en) * 2016-06-24 2019-01-01 Microsoft Technology Licensing, Llc Relational rendering of holographic objects
US11014001B2 (en) * 2018-03-05 2021-05-25 Sony Interactive Entertainment LLC Building virtual reality (VR) gaming environments using real-world virtual reality maps
US10695667B2 (en) * 2018-03-14 2020-06-30 Sony Interactive Entertainment LLC Pro gaming AR visor and method for parsing context specific HUD content from a video stream
US10777012B2 (en) * 2018-09-27 2020-09-15 Universal City Studios Llc Display systems in an entertainment environment
US10776954B2 (en) 2018-10-08 2020-09-15 Microsoft Technology Licensing, Llc Real-world anchor in a virtual-reality environment
WO2020112900A2 (en) 2018-11-30 2020-06-04 Pcms Holdings, Inc. Method for mirroring 3d objects to light field displays
JP7443014B2 (ja) * 2019-10-08 2024-03-05 大豊精機株式会社 ロボットアーム試験装置
EP4499251A1 (en) * 2022-03-30 2025-02-05 Universal City Studios LLC Systems and methods for producing responses to interactions within an interactive environment
JP2024052410A (ja) * 2022-09-30 2024-04-11 富士フイルム株式会社 プロセッサ、情報処理方法、及び情報処理プログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1188913A (ja) * 1997-09-01 1999-03-30 M R Syst Kenkyusho:Kk 姿勢検出装置、姿勢検出方法、記憶媒体および複合現実感提示装置
JP2000353248A (ja) * 1999-06-11 2000-12-19 Mr System Kenkyusho:Kk 複合現実感装置及び複合現実感提示方法
JP2000350865A (ja) * 1999-06-11 2000-12-19 Mr System Kenkyusho:Kk 複合現実空間のゲーム装置、その画像処理方法およびプログラム記憶媒体
JP2003103052A (ja) * 2001-09-28 2003-04-08 Canon Inc 映像体験システム、情報処理方法およびプログラム
JP2006318094A (ja) * 2005-05-11 2006-11-24 Canon Inc 情報処理方法、情報処理装置

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3401897B2 (ja) 1994-02-16 2003-04-28 株式会社セガ 衝突判定処理システムおよびこれを用いた画像処理装置
CN1161087A (zh) * 1994-08-10 1997-10-01 实质(Ip)有限公司 安置在头上的光学系统
US6522312B2 (en) * 1997-09-01 2003-02-18 Canon Kabushiki Kaisha Apparatus for presenting mixed reality shared among operators
ATE543546T1 (de) * 1999-06-11 2012-02-15 Canon Kk Vorrichtung und verfahren zur darstellung eines von mehreren benutzern geteilten raumes, wo wirklichkeit zum teil einbezogen wird, entsprechende spielvorrichtung und entsprechendes schnittstellenverfahren
US6771841B1 (en) * 1999-12-29 2004-08-03 Intel Corporation Determining a bounding shape for a collection of points
US20030062675A1 (en) 2001-09-28 2003-04-03 Canon Kabushiki Kaisha Image experiencing system and information processing method
CN2667920Y (zh) 2003-12-26 2005-01-05 沈阳工业学院 一种光学双视场三维头盔显示装置
JP3844482B2 (ja) 2004-09-01 2006-11-15 株式会社ソニー・コンピュータエンタテインメント 画像処理装置
WO2006025137A1 (ja) * 2004-09-01 2006-03-09 Sony Computer Entertainment Inc. 画像処理装置、ゲーム装置および画像処理方法
JP4335160B2 (ja) 2005-03-02 2009-09-30 任天堂株式会社 衝突判定プログラムおよび衝突判定装置
ES2325374T3 (es) 2005-05-03 2009-09-02 Seac02 S.R.L. Sistema de realidad aumentada con identificacion de objeto marcador real.
US7773098B2 (en) 2005-05-11 2010-08-10 Canon Kabushiki Kaisha Virtual reality presentation apparatus and method
CN100399835C (zh) 2005-09-29 2008-07-02 北京理工大学 用于现场数字三维重建的增强现实定点观察系统
KR100809479B1 (ko) 2006-07-27 2008-03-03 한국전자통신연구원 혼합 현실 환경을 위한 얼굴 착용형 디스플레이 장치
JP2010026818A (ja) 2008-07-18 2010-02-04 Geisha Tokyo Entertainment Inc 画像処理プログラム、画像処理装置及び画像処理方法
CN101539804A (zh) * 2009-03-11 2009-09-23 上海大学 基于增强虚拟现实与异型屏的实时人机互动方法及系统
JP5627973B2 (ja) 2010-09-24 2014-11-19 任天堂株式会社 ゲーム処理をするためのプログラム、装置、システムおよび方法
US8625200B2 (en) * 2010-10-21 2014-01-07 Lockheed Martin Corporation Head-mounted display apparatus employing one or more reflective optical surfaces
JP5472056B2 (ja) * 2010-11-19 2014-04-16 コニカミノルタ株式会社 表示システム、表示処理装置、表示方法、および表示プログラム
US8189263B1 (en) * 2011-04-01 2012-05-29 Google Inc. Image waveguide with mirror arrays
JP5891125B2 (ja) * 2012-06-29 2016-03-22 株式会社ソニー・コンピュータエンタテインメント 映像処理装置、映像処理方法、および映像処理システム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH1188913A (ja) * 1997-09-01 1999-03-30 M R Syst Kenkyusho:Kk 姿勢検出装置、姿勢検出方法、記憶媒体および複合現実感提示装置
JP2000353248A (ja) * 1999-06-11 2000-12-19 Mr System Kenkyusho:Kk 複合現実感装置及び複合現実感提示方法
JP2000350865A (ja) * 1999-06-11 2000-12-19 Mr System Kenkyusho:Kk 複合現実空間のゲーム装置、その画像処理方法およびプログラム記憶媒体
JP2003103052A (ja) * 2001-09-28 2003-04-08 Canon Inc 映像体験システム、情報処理方法およびプログラム
JP2006318094A (ja) * 2005-05-11 2006-11-24 Canon Inc 情報処理方法、情報処理装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105872526A (zh) * 2015-01-21 2016-08-17 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法
CN105872526B (zh) * 2015-01-21 2017-10-31 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法

Also Published As

Publication number Publication date
CN104380347B (zh) 2017-12-05
US9639989B2 (en) 2017-05-02
CN104380347A (zh) 2015-02-25
EP2869274A4 (en) 2016-01-27
JP2014010664A (ja) 2014-01-20
CN107844196B (zh) 2020-12-01
US20150170419A1 (en) 2015-06-18
JP5891125B2 (ja) 2016-03-22
EP2869274A1 (en) 2015-05-06
CN107844196A (zh) 2018-03-27

Similar Documents

Publication Publication Date Title
JP5891125B2 (ja) 映像処理装置、映像処理方法、および映像処理システム
JP7445720B2 (ja) 拡張現実のためのシステムおよび方法
JP5483761B2 (ja) 映像出力装置、立体映像観察デバイス、映像提示システム、および映像出力方法
US9122053B2 (en) Realistic occlusion for a head mounted augmented reality display
US10643389B2 (en) Mechanism to give holographic objects saliency in multiple spaces
JP6813501B2 (ja) 拡張現実システムに結合されるプライバシーセンシティブ消費者カメラ
US10453248B2 (en) Method of providing virtual space and system for executing the same
US20180189549A1 (en) Method for communication via virtual space, program for executing the method on computer, and information processing apparatus for executing the program
US20140368537A1 (en) Shared and private holographic objects
US20180357817A1 (en) Information processing method, program, and computer
TW201447375A (zh) 擴增實境之頭戴式電子裝置及方法
WO2013155217A1 (en) Realistic occlusion for a head mounted augmented reality display
CN104995583A (zh) 用于混合现实环境的直接交互系统
CN104956252A (zh) 用于近眼显示设备的外围显示器
WO2014190106A1 (en) Hologram anchoring and dynamic positioning
JP6161749B2 (ja) 映像処理装置、映像処理方法、および映像処理システム
US20180190010A1 (en) Method for providing virtual space, program for executing the method on computer, and information processing apparatus for executing the program
US20180189555A1 (en) Method executed on computer for communicating via virtual space, program for executing the method on computer, and computer apparatus therefor
HK1190481B (en) Realistic occlusion for a head mounted augmented reality display
HK1190481A (en) Realistic occlusion for a head mounted augmented reality display

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13810089

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2013810089

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 14406919

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE