Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating 3D models or images for computer graphics
  • G06T19/006—Mixed reality
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
  • H04S7/30—Control circuits for electronic adaptation of the sound field
  • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
  • H04S7/303—Tracking of listener position or orientation
  • H04S7/304—For headphones
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S1/00—Two-channel systems
  • H04S1/002—Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution
  • H04S1/005—For headphones
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
  • H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
  • H04S2400/13—Aspects of volume control, not necessarily automatic, in stereophonic sound systems

Definitions

• the present invention relates to a technique for presenting an image obtained by superposing a physical space and virtual space to the user.
• MR mixed reality
• MR presentation apparatus comprises a video display unit, physical video capturing unit, virtual video generation unit, position and orientation detection unit, and video composition unit which composites physical and virtual video images.
• the physical video capturing unit is, for example, a compact camera attached to a head mounted display (HMD) , and captures a scenery in front of the HMD.
• HMD head mounted display
• the HMD as a physical video image.
• the captured physical video image is recorded in a memory of a computer as data .
• the position and orientation detection unit is, for example, a position and orientation sensor, which detects the position and orientation of the physical video capturing unit.
• the position and orientation of the physical video capturing unit can be calculated by a method using magnetism or a method using image processing.
• the virtual video generation unit generates a virtual video image by laying out CG images that have undergone three-dimensional (3D) modeling on a virtual space having the same scale as a physical space, and rendering the scene of that virtual space from the same position and orientation as those of the physical video capturing unit.
• 3D three-dimensional
• the video composition unit generates an MR video image by superposing the virtual video image obtained by the virtual video generation unit on the physical video image obtained by the physical video capturing unit.
• An operation example of the video composition unit includes a control operation for writing a physical video image captured by the physical video capturing unit on a video memory of the computer, and controlling the virtual video generation unit to write a virtual video image on the written physical video image.
• the position and orientation detection unit measures the viewpoint position and orientation of the HMD.
• the video composition unit outputs a virtual video image to the HMD.
• 3D sound reproduction can be executed according to the position of the virtual object using a 3D sound reproduction technique as a related art (patent reference 1) .
• a sound generated in a scene on the virtual space is presented as a 3D sound, or a virtual sound is modified in consideration of a physical sound environment as if it were sounding on the physical space.
• it is difficult to change a physical sound from a physical sound source by changing the layout of the virtual object and to present the changed physical sound to the viewer.
• the viewer cannot use a virtual object as a shield on a physical object serving as a sound source so as to shield a physical sound from that sound source.
• the present invention has been made in consideration of the aforementioned problems, and has - A -
• an image processing apparatus for compositing an image of a physical space and an image of a virtual object, comprises: a unit which acquires a position of a sound source on the physical space and a position of the virtual object; and a change unit which changes a sound based on the sound source in accordance with the position of the sound source and the position of the virtual object.
• an image processing method to be executed by an image processing apparatus for compositing an image of a physical space and an image of a virtual object comprises: a step of acquiring a position of a sound source on the physical space and a position of the virtual object; and a step of changing a sound based on the sound source in accordance with the position of the sound source and the position of the virtual object.
• an image processing apparatus which comprises: a unit which generates an image of a virtual space configured by a virtual object, the image of the virtual space being adapted to be superposed on a physical space on which a physical object serving as a sound source is laid out, a unit which outputs the image of the virtual space, an acquisition unit which acquires a sound produced by the physical object as sound data, and an output unit which generates a sound signal based on the sound data acquired by the acquisition unit, and outputs the generated sound signal to a sound output device
• the apparatus comprises: a unit which acquires position information of the physical object; a unit which acquires position information of the virtual object; a unit which acquires position information of a viewpoint of a user; a determination unit which calculates a positional relationship among the physical object, the virtual object, and the viewpoint using the position information of the physical object, the position information of the virtual object, and the position information of the viewpoint, and determines whether or not the calculated positional relationship satisfies
• an image processing method to be executed by an image processing apparatus which comprises a unit which generates an image of a virtual space configured by a virtual object, the image of the virtual space being to be superposed on a physical space on which a physical object serving as a sound source is laid out, a unit which outputs the image of the virtual space, an acquisition unit which acquires a sound produced by the physical object as sound data, and an output unit which generates a sound signal based on the sound data acquired by the acquisition unit, and outputs the generated sound signal to a sound output device
• the method comprises: a step of acquiring position information of the physical object; a step of acquiring position information of the virtual object; a step of acquiring position information of a viewpoint of a user; a determination step of calculating a positional relationship among the physical object, the virtual object, and the viewpoint using the position information of the physical object, the position information of the virtual object, and the position information of the viewpoint, and determining whether or not the calculated positional
• FIG. 1 is a block diagram showing an example of the hardware arrangement of a system according to the first embodiment of the present invention
• FIG. 2 is a flowchart of main processing executed by a computer 100;
• Fig. 3 is a flowchart showing details of the processing in step S205;
• Fig. 4 is a flowchart showing details of the processing in step S302.
• Fig. 5 is a view showing a state of a physical space assumed upon execution of the processing according to the flowchart of Fig. 4.
• Fig. 1 is a block diagram showing an example of the hardware arrangement of a system according to this embodiment. As shown in Fig. 1, the system according to this embodiment comprises a computer 100, microphone 110, headphone 109, sensor controller 105, position and orientation sensors 106a to 106c, HMD 104, and video camera 103.
• the microphone 110 will be described first. As is well known, the microphone 110 is used to collect a surrounding sound, and a signal indicating the collected sound is converted into sound data and is input to the computer 100.
• the microphone 110 may be laid out at a predetermined position on a physical space or may be laid out on a "physical object that produces a sound (a physical object serving as a sound source) " (on the physical object) laid out on the physical space.
• the headphone 109 is a sound output device which covers the ears of the user and supplies a sound to the ears.
• the headphone 109 is not particularly limited as long as it can supply not a sound on the physical space but only a sound according to sound data supplied from the computer 100.
• a headphone having a known noise cancel function may be used.
• the noise cancel function prevents the user who wears the headphone from hearing any sound on the physical noise, and can realize shielding of a sound better than that obtained by simple sound isolation.
• a sound input from the microphone 110 to the computer 100 is normally output intact to the headphone 109.
• the computer 100 adjusts a sound collected by the microphone 110, and outputs the adjusted sound to the headphone 109.
• the HMD 104 will be described below.
• the video camera 103 and the position and orientation sensor 106a are attached to the HMD 104.
• the video camera 103 is used to capture a movie of the physical space, and sequentially outputs captured frame images (physical space images) to the computer 100.
• the video cameras 103 may be attached one each to the right and left positions on the HMD 104.
• the position and orientation sensor 106a is used to measure the position and orientation of itself, and outputs the measurement results to the sensor controller 105 as signals.
• the sensor controller 105 calculates position and orientation information of the position and orientation sensor 106a based on the signals received from the position and orientation sensor 106a, and outputs the calculated position and orientation information to the computer 100.
• the position and orientation sensors 106b and 106c are further connected to the sensor controller 105.
• the position and orientation sensor 106b is attached to the physical object that produces a sound (the physical object serving as the sound source) , and the position and orientation sensor 106c is laid out at a predetermined position on the physical space or is held by the hand of the user.
• the position and orientation sensors 106b and 106c measure the positions and orientations of themselves as in the position and orientation sensor 106a.
• the position and orientation sensors 106b and 106c respectively output the measurement results to the sensor controller 105 as signals.
• the sensor controller 105 calculates position and orientation information of the position and orientation sensors 106b and 106c based on the signals received from the position and orientation sensors 106b and 106c, and outputs the calculated position and orientation information to the computer 100.
• a sensor system configured by the position and orientation sensors 106a to 106c and the sensor controller 105 can use various sensor systems such as a magnetic sensor, optical sensor, and the like. Since the technique for acquiring the position and orientation information of a target object using a sensor is known to those who are skilled in the art, a description thereof will not be given.
• the HMD 104 has a display screen, which is located in front of the eyes of the user who wears the HMD 104 on the head.
• the computer 100 has a CPU 101 and memories 107 and
• Fig. 1 are those used in the following description, and the computer 100 is not configured by only these components .
• the CPU 101 executes respective processes as those to be implemented by the computer 100 using programs 111 to 114 stored in the memory 107 and data
• the memory 107 stores the programs 111 to
• the memory 108 stores the data 122 to 129, which are to be processed by the CPU 101.
• each of these memories 107 and 108 is not limited to this, and given information described in the following description, and information which would be naturally used by those who are skilled in the art and require no special explanation are stored. Allocations of information to be stored in the memories 107 and 108 are not limited to those shown in Fig. 1.
• 107 and 108 need not be used as independent memories but they may be used as a single memory.
• I/Fs interfaces
• Figs. 2 to 4 that show the flowcharts of the processing. Note that a main body that executes the processing according to these flowcharts is the CPU 101 unless otherwise specified in the following description.
• Fig. 2 is a flowchart of main processing executed by the computer 100.
• the CPU 101 acquires a physical space image (physical video image) output from the video camera 103, and stores it as physical space image data 122 in the memory 108 in step S201.
• a physical space image physical video image
• step S202 the CPU 101 acquires the position and orientation information of the position and orientation sensor 106a, which is output from the sensor controller 105.
• the CPU 101 calculates position and orientation information of the video camera 103 (viewpoint) by adding relationship information indicating the position and orientation relationship between the video camera 103 and position and orientation sensor 106a to the acquired position and orientation information.
• the CPU 101 stores the calculated position and orientation information of the viewpoint in the memory 108 as camera position and orientation data 123.
• step S203 the CPU 101 executes a physical sound source position acquisition program 111 stored in the memory 107.
• the CPU 101 acquires the position and orientation information of the position and orientation sensor 106b, which is output from the sensor controller 105, i.e., that of a physical object serving as a sound source.
• the CPU 101 stores the acquired position and orientation information of the physical object serving as the sound source in the memory 108 as physical sound source position and orientation data 124.
• step S204 the CPU 101 reads out virtual scene data 126 stored in the memory 108, and creates a virtual space based on the readout virtual scene data 126.
• the virtual scene data 126 includes data of layout positions and orientations (position information and orientation information) of virtual objects which form the virtual space, the types of light sources laid out on the virtual space, the irradiation directions of light, colors of light, and the like.
• the virtual scene data 126 includes shape information of the virtual objects. For example, when each virtual object is configured by polygons, the shape information includes normal vector data of the polygons, attributes and colors of the polygons, coordinate value data of vertices that configure the polygons, texture map data, and the like.
• step S205 the CPU 101 executes a physical sound acquisition program 113 stored in the memory 107. As a result, the CPU 101 acquires sound data output from the microphone 110. [0047] The CPU 101 then executes a physical sound modification program 112.
• the CPU 101 calculates the positional relationship among the physical object, virtual objects, and viewpoint using the pieces of position information of the physical object, virtual objects, and viewpoint.
• the CPU 101 determines whether or not the calculated positional relationship satisfies a predetermined condition. If it is determined that the positional relationship satisfies the predetermined condition, the CPU 101 adjusts the sound data acquired in step S205. That is, the CPU 101 manipulates the sound volume and quality of a sound indicated by that sound data based on these pieces of position information.
• the CPU 101 stores the adjusted sound data in the memory 108 as physical sound reproduction setting data 127.
• the CPU 101 executes a sound reproduction program 114. As a result, the CPU 101 outputs a sound signal based on the physical sound reproduction setting data 127 stored in the memory 108 to the headphone 109. Details of the processing in step S205 will be described later.
• step S206 the CPU 101 lays out the viewpoint having the position and orientation indicated by the camera position and orientation data 123 stored in the memory 108 in step S202 on the virtual space created in step S204.
• the CPU 101 then generates an image of the virtual space (virtual space image) viewable from that viewpoint.
• the CPU 101 stores the generated virtual space image in the memory 108 as CG image data 128.
• step S207 the CPU 101 superposes the virtual space image indicated by the CG image data 128 stored in the memory 108 in step S206 on the physical space image indicated by the physical space image data 122 stored in the memory 108 in step S201.
• various techniques for superposing a virtual space image on a physical space image are available, and any of such techniques may be used in this embodiment.
• the CPU 101 stores the generated composite image (a superposed image generated by superposing the virtual space image on the physical space image) in the memory 108 as MR image data 129.
• step S208 the CPU 101 outputs the MR image data 129 stored in the memory 108 in step S207 to the HMD 104 as a video signal. As a result, the composite image is displayed in front of the eyes of the user who wears the HMD 104 on the head.
• the CPU 101 detects an instruction to end this processing input from an operation unit (not shown) or detects that a condition required to end this processing is satisfied, it ends the processing via step S209. On the other hand, if the CPU 101 does not detect anything, the process returns to step S201 via step S209, and the CPU 101 executes the processes in step S201 and subsequent steps so as to present a composite image of the next frame to the user.
• the processing in step S205 will be described below.
• Fig. 3 is a flowchart showing details of the processing in step S205.
• step S301 the CPU 101 executes the physical sound acquisition program 113 stored in the memory 107. As a result, the CPU 101 acquires sound data output from the microphone 110.
• the microphone 110 may be laid out on the "physical object that produces a sound (the physical object serving as the sound source) " (on the physical object) .
• the microphone 110 is preferably attached to a neighboring position of the position and orientation sensor 106b, so that the position and orientation of the microphone 110 become nearly the same as those measured by the position and orientation sensor 106b.
• the microphone 110 may be attached to the user such as the ear of the user who wears the HMD 104 on the head.
• step S302 the CPU 101 executes the physical sound modification program 112.
• the CPU 101 calculates the positional relationship among the physical object, virtual objects, and viewpoint using the pieces of position information of the physical object serving as the sound source, the virtual object, and the viewpoint.
• the CPU 101 determines whether or not the calculated positional relationship satisfies a predetermined condition. If it is determined that the positional relationship satisfies the predetermined condition, the CPU 101 adjusts the sound data acquired in step S301. That is, the CPU 101 manipulates the sound volume and quality of a sound indicated by that sound data based on these pieces of position information.
• the CPU 101 stores the adjusted sound data in the memory 108 as the physical sound reproduction setting data 127. Details of the processing in step S302 will be described later. [0056] In step S303, the CPU 101 executes the sound reproduction program 114. As a result, the CPU 101 outputs a sound signal based on the physical sound reproduction setting data 127 stored in the memory 108 in step S302 to the headphone 109. When other sounds are to be produced (e.g., a virtual object produces a sound) , the CPU 101 generates sound signals based on data of these sounds, and outputs a mixed signal obtained by mixing the generated sound signals and that based on the physical sound reproduction setting data 127 to the headphone 109. [0057] The CPU 101 ends the processing according to the flowchart shown in Fig. 3, and returns to step S206 shown in Fig. 2.
• step S302 Details of the processing in step S302 will be described below.
• Fig. 4 is a flowchart showing details of the processing in step S302.
• the processing of the flowchart shown in Fig. 4 is an example of a series of processes for determining whether or not the positional relationship among the physical object serving as the sound source, virtual objects, and viewpoint satisfies the predetermined relationship, and adjusting sound data when it is determined that the positional relationship satisfies the predetermined condition. That is, in the processing of the flowchart shown in Fig. 4, the CPU 101 determines whether or not one or more intersections between a line segment that couples the position of the physical object serving as the sound source and that of the viewpoint, and the virtual objects exist.
• Fig. 5 is a view showing the physical space assumed upon execution of the processing according to the flowchart of Fig. 4.
• the position and orientation sensor 106b is laid out on a physical object 502 serving as a sound source. Therefore, the position and orientation measured by the position and orientation sensor 106b are those of the position and orientation sensor 106b itself, and are also those of the physical object 502.
• the microphone 110 is laid out at a predetermined position (where it can collect a sound generated by the physical object 502) on the physical space. Of course, the microphone 110 may be laid out on the physical object 502. [0061] A user 501 holds the position and orientation sensor 106c in hand.
• Reference numeral 503 denotes a planar virtual object, which is laid out at the position and orientation measured by the position and orientation sensor 106c (Fig. 5 illustrates the position and orientation sensor 106c and virtual object 503 to deviate from each other so as to illustrate both the virtual object 503 and position and orientation sensor 106c) . That is, when the user moves the hand that holds the position and orientation sensor 106c, the position and orientation of the position and orientation sensor 106c also change, and those of the virtual object 503 change accordingly. As a result, the user 501 can manipulate the position and orientation of the virtual object 503.
• the computer 100 determines that a sound generated by the physical object 502 is shielded by the virtual object 503.
• the computer 100 then adjusts sound data to lower the volume (sound volume) of the sound data acquired from the microphone 110.
• the computer 100 outputs a sound signal based on the adjusted sound data to the headphone 109.
• the user 501 who wears the headphone 109 can experience "the sensation of the volume of the audible sound lowering as a sound given from the physical object 502 is shielded by the virtual object 503".
• step S401 the CPU 101 acquires position information from the position and orientation information of the physical object serving as the sound source acquired in step S203. Furthermore, the CPU 101 acquires position information from the position and orientation information of the viewpoint acquired in step S202.
• the CPU 101 then calculates a line segment that couples a position indicated by the position information of the physical object serving as the sound source, and a position indicated by the position information of the viewpoint. [0066]
• the CPU 101 checks in step S402 if the line segment calculated in step S401 intersects with each of one or more virtual objects laid out in step S204, so as to determine the presence/absence of intersections with the line segment. In this embodiment, assume that the number of virtual objects to be laid out on the virtual space is one, for the sake of simplicity. [0067] As a result of the process in step S402, if the virtual object laid out on the virtual space intersects with the line segment calculated in step S401, the process advances to step S404.
• step S403 the CPU 101 may convert the sound data acquired from the microphone 110 into a sound signal intact without adjusting it, and may output the sound signal to the headphone 109. However, in Fig. 4, the CPU 101 adjusts this sound data to set the volume of a sound indicated by the sound data acquired from the microphone , 110 to that of a prescribed value. Since a technique for increasing or decreasing the volume by adjusting sound data is known to those who are skilled in the art, a description thereof will not be given. The process then returns to step S303 in Fig. 3.
• step S404 the CPU 101 adjusts this sound data so as to lower the volume (sound volume) of a sound indicated by the sound data acquired from the microphone 110 by a predetermined amount. The process then returns to step S303 in Fig. 3. As a result, a sound signal can be generated based on the adjusted sound data, and that sound signal can be output to the headphone 109.
• step S403 determines whether or not an intersection exists.
• the amount of lowering the volume may be varied in accordance with the position of the intersection on the virtual object.
• the surface of the virtual object is divided into a plurality of regions, and amounts of lowering the volume are set for the respective divided regions. Then, by specifying which of the divided regions the intersection is located, the volume is lowered by an amount corresponding to the specified divided region. Also, the amount of lowering the volume may be changed depending on whether or not the region of the virtual object includes the physical object serving as the sound source.
• material information indicating the material of the virtual object may be referred to, and the amount of lowering the volume may be varied based on the material information which is referred to. For example, when the material information at the intersection assumes a numerical value indicating high hardness of the material, the amount of lowering the volume is increased. Conversely, when the material information at the intersection assumes a numerical value indicating low hardness of the material, the amount of lowering the volume is decreased.
• the volume of a sound indicated by sound data is manipulated as an example of adjustment of sound data.
• other elements of a sound may be changed.
• a sound indicated by sound data acquired from the microphone 110 may be filtered (equalized) in association with its frequency. For example, only low- frequency components may be reduced, or only high- frequency components may be reduced.
• material information indicating the material of the virtual object may be referred to, and the sound data may be adjusted to change the sound quality of a sound indicated by that sound data based on the material information, which is referred to.
• This embodiment has exemplified the case in which the virtual object shields a sound generated by the physical object serving as the sound source.
• the volume of a sound indicated by the sound data may be increased.
• the HMD 104 of the video see-through type is used.
• an HMD of an optical see-through type may be used.
• transmission of a sound signal to the HMD 104 remains the same, but that of an image to the HMD 104 is slightly different from the above description. That is, when the HMD 104 is of the optical see-through type, only a virtual space image is transmitted to the HMD 104.
• a method other than the position and orientation acquisition method using the sensor system may be used. For example, a method of laying out indices on the physical space, and calculating the position and orientation information of the video camera 103 using an image obtained by capturing that physical space by the video camera 103 may be used. This method is a state-of-the-art technique.
• the position information of the physical object serving as the sound source may be acquired using a microphone array in place of the position and orientation sensor attached to the physical object.
• the number of physical objects serving as sound sources is one. However, even when a plurality of physical objects serving as sound sources are laid out on the physical space, the first embodiment can be applied to each individual physical object.
• microphones 110 and position and orientation sensors 106c are provided to the respective physical objects serving as sound sources.
• the computer 100 executes the processing described in the first embodiment for each physical object, and finally mixes sounds collected from the respective physical objects, thus outputting the mixed sound to the headphone 109.
• a recording medium (or storage medium) that records program codes of software required to implement the functions of the aforementioned embodiments is supplied to a system or apparatus.
• That storage medium is a computer-readable storage medium, needless to say.
• a computer or a CPU or MPU of that system or apparatus reads out and executes the program codes stored in the recording medium.
• the program codes themselves read out from the recording medium implement the functions of the aforementioned embodiments, and the recording medium that records the program codes constitutes the present invention.
• an operating system (OS) or the like, which runs on the computer, executes some or all of actual processes based on instructions of these program codes.
• OS operating system
• the present invention also includes a case in which the functions of the aforementioned embodiments are implemented by these processes.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Computer Hardware Design (AREA)
• Computer Graphics (AREA)
• Theoretical Computer Science (AREA)
• Acoustics & Sound (AREA)
• Signal Processing (AREA)
• Stereophonic System (AREA)
• Processing Or Creating Images (AREA)
• Studio Devices (AREA)
• User Interface Of Digital Computer (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=40625863

Family Applications (1)

Country Status (3)

Cited By (1)

Families Citing this family (23)

Citations (3)

Family Cites Families (3)

• 2007
  • 2007-11-07 JP JP2007289965A patent/JP4926916B2/ja not_active Expired - Fee Related
• 2008
  • 2008-11-05 US US12/741,344 patent/US20100265164A1/en not_active Abandoned
  • 2008-11-05 WO PCT/JP2008/070540 patent/WO2009060981A1/en not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events