JP2007087030A - Program, information storage medium, and image generation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a program, an information storage medium, and an image generation system for expressing the movement of an object vibrating in response to a sound in reality. <P>SOLUTION: When sound data sampled by a sampling rate which is higher than a frame rate are reproduced and output, an image viewed from a predetermined point of view in an object space is generated by performing vertex shading to move vertexes configuring an object on the basis of data corresponding to the sound data, that is, vibration data sampled by the frame rate. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a program, an information storage medium, and an image generation system.

  Conventionally, an image generation system (game system) that generates an image that can be viewed from a virtual camera (a given viewpoint) in an object space (virtual three-dimensional space) in which an object such as a character is set is known. It is very popular for experiencing so-called virtual reality.

In such an image generation system, in order to enhance the player's virtual reality, for example, a sound that is actually reproduced and output also with respect to the movement of an object that performs complex vibration (pulsation) such as a speaker diaphragm (cone). It is desired to be able to express realistically in correspondence with.
JP 2001-269483 A

  The present invention has been made in view of the above circumstances, and provides a program, an information storage medium, and an image generation system capable of realistically expressing the motion of an object that vibrates in response to sound.

  The present invention is an image generation system for generating an image that can be viewed from a given viewpoint in an object space, the sound data storage unit storing sound data sampled at a sampling rate higher than a frame rate, and the sound data A vibration data storage unit that stores vibration data sampled at the frame rate, and when the sound data is reproduced and output, the vibration data corresponding to the reproduced frame is acquired. , And an image generation unit that performs vertex shading for moving the vertices constituting the object based on the vibration data. The present invention also relates to a program that causes a computer to function as each of the above-described units. The present invention also relates to an information storage medium that can be read by a computer, and stores (records) a program that causes the computer to function as each of the above-described units.

  According to the present invention, an object is deformed by moving vertices constituting the object using vibration data corresponding to a reproduction frame of sound data. This makes it possible to realistically represent how the object vibrates in accordance with the change in the reproduced sound.

  The image generation system of the present invention further includes a vibration data generation unit that acquires the sound data and generates the vibration data corresponding to a reproduction frame of the sound data, and the vibration data generation unit includes the Nth reproduction. The vibration data corresponding to the Nth reproduction frame may be generated by filtering a plurality of sampling data of the sound data reproduced and output in the frame. In the program and information storage medium of the present invention, a computer may function as the vibration data generation unit. In this way, vibration data suitable for deforming an object in the reproduction frame can be obtained from a plurality of sampling data included in the sound data reproduced and output in the same reproduction frame.

  In the image generation system, the program, and the information storage medium of the present invention, the vibration data generation unit performs processing for averaging a plurality of sampling data of the sound data reproduced and output in the Nth reproduction frame. You may make it carry out as. In this way, the low-frequency component of the sound data can be extracted as vibration data, and for example, it is possible to realistically represent how the woofer diaphragm vibrates.

  In the image generation system, program, and information storage medium of the present invention, the vibration data generation unit has a maximum or minimum amplitude among a plurality of sampling data of the sound data reproduced and output in the Nth reproduction frame. You may make it perform the process which extracts sampling data as said filtering process. If it does in this way, vibration data can be generated from sound data by simple processing. For example, if sampling data with the maximum amplitude and sampling data with the minimum amplitude are extracted alternately in a plurality of reproduction frames, vibration data can be obtained.

  In the image generation system, the program, and the information storage medium of the present invention, the vibration data generation unit acquires sound data obtained by sampling the detection sound of the sound detection unit at a sampling rate higher than the frame rate, and the acquired detection sound The vibration data may be generated based on sound data corresponding to. In this way, it is possible to realistically represent how the object vibrates by deforming the object according to the sound freely input to the microphone or the like.

  In the image generation system, the program, and the information storage medium of the present invention, the vibration data generation unit ends the vibration data corresponding to the sound data for the Nth reproduction frame, and at least the N-1th reproduction frame ends. You may make it produce | generate before. In this way, it is possible to reduce the difference between the timing of reproducing sound data and the state of vibration of the object, and to express the state of vibration of the object that has reacted to the sound in a state close to real time. Become.

  In the image generation system, the program, and the information storage medium of the present invention, the image generation unit moves the vertex of the object based on the vibration data and attribute data set corresponding to the object. May be performed. In this way, it is possible to express that the state of vibration differs according to the difference in hardness and vibration characteristics of the object.

  In the image generation system, the program, and the information storage medium of the present invention, the image generation unit obtains movement information of a plurality of vertices constituting the object by interpolation processing of the vibration data, and each vertex is determined based on the movement information. You may make it perform the vertex shading which moves. In this way, different movement information can be obtained from a single piece of vibration data for a plurality of vertices, and various vibration states of the object can be expressed.

  In the image generation system, the program, and the information storage medium of the present invention, the image generation unit moves the vertices based on the vibration data and a movement vector set in advance for each vertex constituting the object. Information may be obtained, and vertex shading for moving each vertex based on the movement information may be performed. In this way, different movement information can be obtained from a single piece of vibration data for a plurality of vertices, and various vibration states of the object can be expressed.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows an example of a functional block diagram of an image generation system (game system) of the present embodiment. Note that the image generation system of the present embodiment may have a configuration in which some of the components (each unit) in FIG. 1 are omitted.

  The input unit 160 is for a player to input operation data of an object (for example, a player character). The input unit 160 includes an operation unit 162 and a sound detection unit 164. The operation unit 162 can be realized by, for example, a lever, a button, a steering, a touch panel display, or the like. The sound detection unit 164 can be realized by a microphone (microphone), for example. The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM (VRAM) or the like.

  The information storage medium 180 (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (CD, DVD), magneto-optical disk (MO), magnetic disk, hard disk, and magnetic tape. Alternatively, it can be realized by a memory (ROM). The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, the information storage medium 180 stores a program for causing a computer to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit).

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by a CRT, LCD, touch panel display, HMD (head mounted display), or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The portable information storage device 194 stores player personal data, game save data, and the like. Examples of the portable information storage device 194 include a memory card and a portable game device. The communication unit 196 performs various controls for communicating with the outside (for example, a host device or other image generation system), and functions thereof are hardware such as various processors or communication ASICs, programs, and the like. It can be realized by.

  A program (data) for causing a computer to function as each unit of the present embodiment is distributed from the information storage medium of the host device (server) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. May be. Use of the information storage medium of such a host device (server) can also be included in the scope of the present invention.

  The processing unit 100 (processor) performs processing such as game processing, image generation processing, or sound generation processing based on operation data and programs from the operation unit 160. Here, as the game process, a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for placing an object such as a character or a map, a process for displaying an object, and a game result are calculated. There is a process or a process of ending a game when a game end condition is satisfied. The processing unit 100 performs various processes using the main storage unit 172 in the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs.

  The processing unit 100 includes an object space setting unit 110, a movement / motion processing unit 112, a virtual camera control unit 114, a vibration data generation unit 116, an image generation unit 120, and a sound generation unit 130. Note that some of these may be omitted.

  The object space setting unit 110 is composed of various objects (polygons, free-form surfaces, subdivision surfaces, etc.) representing display objects such as characters, buildings, stadiums, cars, trees, pillars, walls, and maps (terrain). (Object) is set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects.

  The movement / motion processing unit 112 performs a movement / motion calculation (movement / motion simulation) of an object (such as a character, a car, or an airplane). That is, based on operation data input by the player through the operation unit 160, a program (movement / motion algorithm), various data (motion data), etc., the model object is moved in the object space, or the object is moved (motion, motion, etc.). Animation). Specifically, a simulation process for sequentially obtaining object movement information (position, rotation angle, speed, or acceleration) and motion information (part object position or rotation angle) every frame (1/60 second). Do. A frame is a unit of time for performing object movement / motion processing (simulation processing) and image generation processing.

  The virtual camera control unit 114 performs a virtual camera (viewpoint) control process for generating an image viewed from a given (arbitrary) viewpoint in the object space. Specifically, a process for controlling the position (X, Y, Z) or the rotation angle (rotation angle about the X, Y, Z axes) of the virtual camera (process for controlling the viewpoint position and the line-of-sight direction) is performed.

  For example, when an object (eg, character, ball, car) is photographed from behind using a virtual camera, the position or rotation angle of the virtual camera (the direction of the virtual camera is set so that the virtual camera follows changes in the position or rotation of the object. ) To control. In this case, the virtual camera can be controlled based on information such as the position, rotation angle, or speed of the object obtained by the movement / motion processing unit 112. Alternatively, the virtual camera may be controlled to rotate at a predetermined rotation angle or to move along a predetermined movement path. In this case, the virtual camera is controlled based on the virtual camera data for specifying the position (movement path) or rotation angle of the virtual camera. When there are a plurality of virtual cameras (viewpoints), the above control process is performed for each virtual camera.

  The vibration data generation unit 116 performs processing for acquiring sound data and generating vibration data corresponding to a reproduction frame of the sound data. The generated vibration data is stored in the vibration data storage unit 178. The sound data is acquired from the sound data storage unit 176. The sound data storage unit 176 stores sound data corresponding to the sound detected by the sound detection unit 164 and sound data read from the information storage medium 180. This sound data is data sampled at a sampling rate sufficiently higher than a frame rate (for example, 1/60 second) which is a unit period of the drawing process. For example, taking CDM or DVD PCM data that can be the information storage medium 180 as an example, the sampling rate is 1/444100 seconds (44.1 kHz). In addition, when acquiring sound from the sound detection unit 164 such as a microphone, the vibration data generation unit 116 detects sound when generating vibration data from sound data corresponding to the sound detected by the sound detection unit 164. Sound data obtained by sampling the detection sound of the unit 164 at a sampling rate higher than the frame rate is buffered in the sound data storage unit 176, and sound data corresponding to the detection sound is acquired. In this way, it is possible to realistically represent how the object vibrates by deforming the object according to the sound freely input to the microphone or the like.

  The vibration data generation unit 116 generates vibration data by filtering a plurality of sampling data included in the acquired sound data. As the filtering process, for example, the sampling data of the sound data reproduced within one frame, the sampling data with the maximum amplitude among the sampling data of the sound data reproduced within one frame, and the amplitude There is a process of alternately extracting the minimum sampling data.

  The image generation unit 120 performs drawing processing based on the results of various processes (game processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. When generating a so-called three-dimensional game image, first, a vertex list including vertex data (vertex position coordinates, texture coordinates, color data, normal vector, α value, etc.) of each vertex of the object is input and input. Based on the vertex data included in the vertex list, vertex shading (vertex processing in a broad sense) is performed. When performing vertex shading, vertex generation processing (tessellation, curved surface division, polygon division) for subdividing a polygon can be performed as necessary. In vertex shading, according to the vertex shader program (first shader program in a broad sense), vertex movement processing, coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, or geometric processing such as perspective transformation is performed. On the basis of the processing result, the vertex data given to the vertex group constituting the object is changed (updated or adjusted). Then, rasterization (scan conversion) is performed based on the vertex data after vertex shading, and pixel shading (pixel processing and fragment processing in a broad sense) that draws pixels (fragments that configure the display screen) that make up the frame image Is done. In pixel shading, according to the pixel shader program (second shader program), various processes such as texture reading (texture mapping), color data setting / changing, translucent composition, anti-aliasing, etc. A specific drawing color is determined, and the drawing color of the perspective-transformed object is output (drawn) to a drawing buffer 174 (a buffer capable of storing image information in units of pixels; VRAM). That is, in pixel shading, per-pixel processing for setting or changing image information (color, normal, luminance, α value, etc.) of a frame image in units of pixels is performed. As a result, an image (frame image) that can be seen from the virtual camera (given viewpoint) in the object space is generated. In addition, when there are a plurality of virtual cameras (viewpoints), a frame image can be generated so that an image seen from each virtual camera can be displayed on one screen as a divided image.

  The image generation unit 120 includes a vertex shader unit 122 and a pixel shader unit 124. Note that some of these may be omitted.

  The vertex shader unit 122 is a type of programmable shader that performs vertex shading, which is processing performed in units of vertices. In particular, when the sound data is reproduced and output from the sound output unit 192, the vertex shader unit 122 acquires vibration data corresponding to the reproduction frame from the vibration data storage unit 178, and based on the acquired vibration data. Vertex shading is performed to move (displace) the vertices that make up the object. Specifically, the position coordinates of each vertex after movement (after displacement) are obtained based on the movement information of each vertex including vibration data and other parameters set for each vertex as necessary. A process of moving each vertex to the position coordinate is performed.

  The movement information is information that gives the movement amount (movement distance) and movement direction of the vertex when the position coordinates after the movement of the vertex are obtained. In this movement information, the amount of movement of the vertex can be given by vibration data. At this time, one or a plurality of vertices constituting the object can be set in advance as representative vertices (reference vertices), and the movement amounts of the representative vertices and the surrounding vertices can be obtained by interpolation processing of vibration data. A coefficient (movement amount coefficient) for determining the movement amount of each vertex is set in the vertex data in advance, and the movement amount of each vertex can be obtained by multiplying the coefficient and vibration data. . Further, a reference movement vector for determining the movement amount and movement direction of each vertex is set in the vertex data in advance, and the movement amount and movement direction of each vertex can be obtained from the vector data and vibration data. it can. The reference movement vector may be one whose size is normalized (unit vectorization). In this case, only the movement direction of the vertex is given by the reference movement vector. In addition to the above, attribute data is included as a parameter included in the movement information. The attribute data is data for giving the ease of vibration of the object such as the natural frequency (resonance frequency) of the object or the elastic coefficient (stiffness, softness) of the object. For example, attribute data can be used as a parameter for changing the state of vibration according to the material of the object, such as an object made of metal that hardly vibrates and an object such as paper easily vibrates.

  The pixel shader unit 124 can perform texture mapping, hidden surface removal processing, α blending, and the like.

  Texture mapping is a process for mapping a texture (texel value) stored in a texture area (not shown) in the storage unit 170 to an object. Specifically, the texture (surface properties such as color and α value) is read from the texture area of the storage unit 170 using texture coordinates or the like set (given) to the vertex of the object. Then, a texture that is a two-dimensional image is mapped to an object. In this case, processing for associating pixels and texels, bilinear interpolation (texel interpolation), and the like are performed.

  As the hidden surface removal processing, hidden surface removal processing can be performed by a Z buffer method (depth comparison method, Z test) using a Z buffer (depth buffer) in which Z values (depth information) of drawing pixels are stored. . That is, when drawing pixels corresponding to the primitive of the object are drawn, the Z value stored in the Z buffer is referred to. Then, the Z value of the referenced Z buffer is compared with the Z value at the drawing pixel of the primitive, and the Z value at the drawing pixel is a Z value (for example, a small Z value) on the near side when viewed from the virtual camera. In some cases, the drawing process of the drawing pixel is performed and the Z value of the Z buffer is updated to a new Z value.

  α blending (α synthesis) is a translucent synthesis process (usually α blending, addition α blending, subtraction α blending, or the like) based on an α value (A value). For example, in the case of normal α blending, the following formulas (1) to (3) are performed.

R _Q = (1−α) × R ₁ + α × R ₂ (1)
G _Q = (1−α) × G ₁ + α × G ₂ (2)
B _Q = (1−α) × B ₁ + α × B ₂ (3)
In addition, in the case of addition α blending, the following processes (4) to (6) are performed.

R _Q = R ₁ + α × R ₂ (4)
G _Q = G ₁ + α × G ₂ (5)
B _Q = B ₁ + α × B ₂ (6)
Further, in the case of subtraction α blending, the following formulas (7) to (9) are performed.

R _Q = R ₁ −α × R ₂ (7)
G _Q = G ₁ −α × G ₂ (8)
B _Q = B ₁ −α × B ₂ (9)
Here, R ₁ , G ₁ , B ₁ are RGB components of an image (original image) already drawn in the drawing buffer 174, and R ₂ , G ₂ , B ₂ should be drawn in the drawing buffer 174. This is the RGB component of the image. R _Q , G _Q , and B _Q are RGB components of an image obtained by α blending. The α value is information that can be stored in association with each pixel (texel, dot), for example, plus alpha information other than color information. The α value can be used as mask information, translucency (equivalent to transparency and opacity), bump information, and the like.

  Here, the vertex shader unit 122 and the pixel shader unit 124 described above are a kind of so-called programmable shader that can program polygon (primitive) drawing processing by a shader program described in a shading language. Programmable shaders can be programmed with vertex-based processing and pixel-by-pixel processing, so the degree of freedom of polygon rendering processing is high, and the expressive power is greatly improved compared to fixed image generation processing using conventional hardware. Can be improved.

  The sound generation unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates game sounds such as BGM, sound effects, or sounds, and outputs the game sounds to the sound output unit 192.

  Note that the image generation system of the present embodiment may be a system dedicated to the single player mode in which only one player can play, or may be a system having a multiplayer mode in which a plurality of players can play. Further, when a plurality of players play, game images and game sounds to be provided to the plurality of players may be generated using one terminal, or connected via a network (transmission line, communication line) or the like. Alternatively, it may be generated by distributed processing using a plurality of terminals (game machine, mobile phone).

2. 2. Method of this embodiment 2.1 Object deformation method according to sound data In this embodiment, by moving the vertex of an object according to the sound data reproduced and output from the sound output unit, the object is deformed, A method is used to express how an object vibrates.

  For example, in the case of a speaker model object MOB as shown in FIG. 2A, the vibration of the diaphragm synchronized with the sound data reproduced and output by deforming the part object POB of the diaphragm (cone). Can be expressed. For example, as shown in FIGS. 2 (B) and 2 (C), the apex constituting the diaphragm part object POB is displayed on the basis of the vibration data corresponding to the sound data. ) Vibration direction) and a bulging direction (-(negative) vibration direction), the state of vibration of the diaphragm can be expressed.

  For example, when the part object POB is deformed in the direction in which the diaphragm contracts, as shown in FIG. 3A, the vertices constituting the part object POB are moved with the direction of the reference movement vector as the movement direction. Further, for example, even when the part object POB is moved in the direction in which the diaphragm contracts, as shown in FIG. 3B, the apexes constituting the part object POB are moved with the direction of the reference movement vector as the movement direction. . Since the vibration direction differs between the case shown in FIG. 3A and the case shown in FIG. 3B, the reference movement vectors are given in opposite directions. Only one reference movement vector needs to be set for each vertex, and when moving in one vibration direction is a positive direction and moving in the other vibration direction is a negative direction, By reversing the vector set in the positive direction or the negative direction according to the polarity of the vibration direction, the reference movement vector in the negative direction or the positive direction can be given. The setting of the vibration direction will be described separately later.

  In each case shown in FIGS. 3A and 3B, the amount of movement of the vertex can be obtained from the result of multiplying the size of the reference movement vector and the amount of vibration obtained from the vibration data. Here, the size of the reference movement vector represents a movement coefficient for determining the movement amount of the vertex. This movement coefficient may be given by a parameter different from the reference movement vector. The reference movement vector may be a normalized vector (unit vectorized), such as a normal vector. In this case, only the movement direction of each vertex is given by the reference movement vector.

  Specifically, as shown in FIG. 4A, the reference movement vector with respect to the positive (+) vibration direction (first vibration direction in a broad sense) and the negative (−) vibration direction ( In a broad sense, the reference movement vector with respect to the second vibration direction) can be set so as to have a mirror image inverted (flip) relationship with respect to the plane (polygon, primitive) of the original model. In this way, it is possible to suppress the deviation (unevenness) of the portion that greatly changes in expansion relative to the original model. Further, in the case of performing vibration expression with a simple model, as shown in FIG. 4B, a unit vector (normal vector) in the normal direction given to each vertex of the object can be used as a reference movement vector. In this case, the reference movement vector with respect to the positive direction (+ side) vibration direction (first vibration direction in a broad sense) and the negative side (− side) vibration direction (second vibration direction in a broad sense). It is only necessary that the reference movement vector with respect to is simply inverted with respect to the original model surface (polygon, primitive).

  The movement coefficient is a coefficient that represents the degree of influence of the vibration amount obtained from the vibration data. For example, a large movement coefficient (for example, 1) is set for a vertex that requires a large movement amount, and a small movement coefficient (for example, less than one) is set for a vertex that requires a small movement amount of the vertex. Then, the degree of contraction and swelling of the diaphragm can be expressed realistically. In other words, the vertex near the center of the diaphragm can be set to a high movement coefficient, and the vertex at the end can be set to a low movement coefficient. The movement coefficient can be set in an arbitrary value range.

  Thus, if the movement coefficient for determining the movement amount of the vertex is set separately from the vibration amount obtained from the vibration data, the movement amount of a plurality of vertices can be determined from one vibration data. There are other methods for obtaining the movement amount of each vertex from one vibration data without setting a movement coefficient for each vertex. For example, it is a method using interpolation processing. In this case, one or a plurality of vertices whose movement amount needs to be increased are set as representative vertices (reference vertices). In this case, for the representative vertex, the vibration amount obtained from the vibration data is used as the vertex movement amount. For the vertices around the representative vertex, an interpolation vibration amount obtained by interpolating the vibration amount given to the representative vertex according to the distance from the representative vertex can be obtained as the movement amount of the vertex.

  Further, in order to express the state of vibration according to the material (material) of the diaphragm, attribute data can be set separately from the parameters described above. As this attribute data, a natural frequency (resonance frequency), an elastic coefficient, etc. can be considered. For example, by setting the natural frequency (resonance frequency) as attribute data, it is possible to express how the vibration is amplified when sound data in a specific frequency band is reproduced and output. Become.

  Next, FIGS. 5A to 5C show an example in which the vibration of the part object near the diaphragm of the speaker is expressed in detail. FIG. 5A shows when the amount of vibration (amplitude) toward the front side (+ side, positive side) is maximum, and FIG. 5B shows the neutral state (no vibration state, zero amplitude state). FIG. 5C shows a case where the amount of vibration (amplitude) to the back side (− side, negative side) is maximum.

  The part objects near the diaphragm include a cone (diaphragm), a center cap, an edge, and the like. In this model, the vertices constituting the center cap are moved back and forth (front and back) by a movement amount Δd (+ Δd, −Δd) obtained based on the vibration data extracted from the sound data, and the vertices constituting the cone are moved. It is moved. In this case, the reference movement vector for setting the direction in which each vertex constituting the cone and the center cap is moved can be shared. Of course, it may be set individually for each vertex, but by using a common reference movement vector, memory consumption can be saved. Moreover, the movement coefficient which determines the moving amount | distance of each vertex can be given for every vertex. For example, for the vertices constituting the cone, the movement coefficient can be set so that the movement amount of the vertices near the edge is small and the movement amount of the vertices near the center cap is large.

  Further, in the detailed model shown in FIGS. 5A to 5C, the deformation of the cone portion (the movement of the apex) may be in the form shown in FIGS. 3A and 3B. In this way, the visual effect is improved, and it can be expressed so that it can be recognized that the cone is vibrating.

  In order to realize more realistic expression, the edge part may be vibrated, and the texture pattern mapped to the cone part is not a single color, but a pattern that can grasp the expansion and contraction of the polygon of the cone part A texture of (color, line, etc.) may be used.

2.2 Vibration Data Generation Method In this embodiment, a method of generating vibration data corresponding to sound data by filtering a plurality of sampling data of sound data output within a reproduction frame of sound data is adopted. Yes. Note that the vibration data may be generated in real time, or previously generated data may be buffered. When vibration data is generated in real time, it is desirable to buffer sound data for several frames in advance and generate vibration data in a frame before the reproduction frame. In this way, when a playback frame of sound data arrives, vertex movement processing can be performed using vibration data corresponding to the sound data played back and output in that playback frame, and the playback and output is actually performed. The difference between the sound data and the vibration of the object due to the deformation of the object can be reduced.

  FIG. 6A shows an example of sound data. This sound data consists of a sampling data group sampled at a sampling rate higher than the frame rate, and a plurality of sampling data is included in one frame. In this case, the volume information of each sampling data may be considered as vibration data. However, since the object rendering process is performed in units of frame rate, it is extremely difficult to generate an image that faithfully follows this sampling rate. In the case of sound data recorded on a CD, DVD, etc., the sampling rate is 44.1 KHz, which is much higher than the frame rate (60 Hz). The eyes cannot recognize the vibration.

  Therefore, in this embodiment, vibration data for moving the vertex of the object is generated by filtering a plurality of sampling data of the sound data reproduced and output within one reproduction frame.

  Specifically, as shown in FIG. 6B, by extracting a sampling data group of sound data reproduced and output in each of the reproduction frames F1 to F9, a low frequency component of the sound data is extracted, Vibration data can be generated. In other words, the volume information of the sampling data group is cumulatively added and the added value is divided by the number of sampling points to calculate the average volume of the volume information in each reproduction frame F1 to F9, and the average volume is vibrated. It can be obtained as data. At this time, by setting an arbitrary threshold volume, vibration data with an average volume exceeding the threshold volume is treated as data representing the amount of vibration in the positive vibration direction, and the average volume below the threshold is set. The vibration data can be handled as data representing the amount of vibration in the negative vibration direction. That is, the vibration data can be given polarity by subtracting the threshold volume from the average volume. The absolute value of the difference between the average volume and the threshold volume can be used as the vibration amount in each vibration direction. Then, according to the polarity of the vibration data, it can be determined whether the moving direction of the apex should be positive or negative.

  According to this method, by extracting the low-frequency component of the sound data as vibration data, for example, it is possible to realistically represent how the diaphragm of the woofer vibrates.

  As a method for generating vibration data, as shown in FIG. 6C, the sampling data of the maximum volume and the minimum of sampling data groups included in the sound data reproduced and output in each of the reproduction frames F1 to F9. The sampling data of the volume can be detected alternately for each reproduction frame, and vibration data having the difference between the maximum volume and the minimum volume as the amplitude can be generated. That is, in this method, peak filtering for detecting the peak of vibration data is performed. Also in this method, by setting an arbitrary threshold volume, the vibration data can be given a polarity, and the volume information corresponding to the vibration data in each of the reproduction frames F1 to F9 and the threshold volume are set. The difference can be the amount of vibration in each vibration direction. In this way, vibration data having a large amplitude can be generated.

3. Processing of this Embodiment Next, a detailed processing example of this embodiment will be described using the flowcharts of FIGS.

  As shown in FIG. 7, it is first determined whether or not it is a frame update (1/60 second) timing (step S10). If it is the frame update timing (YES in step S10), sampling data for one frame is acquired from the sound data buffered in the sound data storage unit (step S11). Next, vibration data generation processing is performed (step S12). That is, as shown in FIG. 6B or 6C, the vibration data is obtained by performing filtering that performs averaging processing and peak detection processing of a plurality of sampling data.

  Next, the shader program for the vertex shader and the pixel shader is transferred, and various parameters necessary for performing the rendering process by executing the shader program are set and transferred (steps S13 and S14). For example, the reference movement vector, vibration data, and attribute data as necessary are transferred to the drawing processor as parameters. Then, the vertex list of the model object is transferred to the drawing processor (step S15), and the vertex shader program and the pixel shader program transferred in step S13 are sequentially executed (steps S16 and S17).

  When vibration data is generated and buffered before a reproduction frame, processing is performed according to the flowchart shown in FIG.

  First, it is determined whether or not it is a frame update (1/60 second) timing (step S10a). If it is the frame update timing (YES in step S10a), vibration data corresponding to the current frame (main frame) buffered in the vibration data storage unit is acquired (step S11a).

  Next, the shader program for the vertex shader and the pixel shader is transferred, and various parameters necessary for performing the rendering process by executing the shader program are set and transferred (steps S12a and S13a). Then, the vertex list of the model object is transferred to the drawing processor (step S14a), and the vertex shader program and the pixel shader program transferred in step S13a are sequentially executed (steps S15a and S16a).

  Thereafter, vibration data for the next frame is generated. Specifically, sampling data for one frame of the next frame is acquired from the sound data group buffered in the sound data storage unit (step S17a). The acquired sampling data group is filtered to generate vibration data for the next frame (step S18a), the generated vibration data is buffered in the vibration data storage unit, and the process ends.

  By the way, the vertex shader performs vertex shading according to the flowchart shown in FIG. First, it is determined whether or not each vertex included in the transferred vertex list is a vertex subject to vertex movement processing (step S20). If it is a vertex subject to vertex movement processing (YES in step S20). Based on the polarity of the vibration data, the vibration direction of the apex is determined (step S21). That is, as shown in FIGS. 2B and 2C, FIGS. 4A and 4B, or FIGS. 5A and 5C, the vibration direction is positive (+). Or a negative (−) vibration direction to determine the direction of the reference movement vector. Next, the coordinates after movement of the vertex are obtained based on the vibration data, the vibration direction, and the reference movement vector set for each vertex, and the vertex is moved according to the obtained coordinates (step S22). In other words, processing for converting the position coordinates of the vertex is performed by matrix calculation (vector calculation). Next, coordinate system conversion processing is performed by geometry processing such as perspective conversion, and the vertex screen coordinates are calculated (step S23). At this time, texture coordinates are set. Finally, the created vertex data is transferred to the pixel shader (step S24), and the vertex shader program is terminated.

4). Hardware Configuration FIG. 10 shows an example of a hardware configuration capable of realizing this embodiment. The main processor 900 operates based on a program stored in a DVD 982 (information storage medium, which may be a CD), a program downloaded via the communication interface 990, a program stored in the ROM 950, or the like. Perform processing, sound processing, etc. The coprocessor 902 assists the processing of the main processor 900, and executes matrix operation (vector operation) at high speed. For example, when a matrix calculation process is required for a physical simulation for moving or moving an object, a program operating on the main processor 900 instructs (requests) the process to the coprocessor 902.

  The geometry processor 904 performs geometry processing such as coordinate conversion, perspective conversion, light source calculation, and curved surface generation based on an instruction from a program operating on the main processor 900, and executes matrix calculation at high speed. The data decompression processor 906 performs decoding processing of the compressed image data and sound data and accelerates the decoding processing of the main processor 900. Thereby, a moving image compressed by the MPEG method or the like can be displayed on the opening screen or the game screen.

  The drawing processor 910 executes drawing (rendering) processing of an object composed of primitive surfaces such as polygons and curved surfaces. When drawing an object, the main processor 900 uses the DMA controller 970 to pass the drawing data (vertex data and other parameters) to the drawing processor 910 and, if necessary, the texture to the texture storage unit 924. Forward. Then, the drawing processor 910 draws the object in the frame buffer 922 while performing hidden surface removal using a Z buffer or the like based on the drawing data and texture. The drawing processor 910 also performs α blending (translucent processing), depth cueing, mip mapping, fog processing, bilinear filtering, trilinear filtering, anti-aliasing, shading processing, and the like. Programmable shaders such as a vertex shader and a pixel shader are also mounted on the drawing processor 910. According to the shader program that realizes the method of this embodiment, the creation / change (update) of vertex data and the drawing color of pixels (or fragments) are changed. Make a decision. When an image for one frame is written in the frame buffer 922, the image is displayed on the display 912.

  The sound processor 930 includes a multi-channel ADPCM sound source and the like, generates game sounds such as BGM, sound effects, and sounds, and outputs them through the speaker 932. Data from the game controller 942 and the memory card 944 is input via the serial interface 940.

  The ROM 950 stores system programs and the like. In the case of an arcade game system, the ROM 950 functions as an information storage medium, and various programs are stored in the ROM 950. A hard disk may be used instead of the ROM 950. The RAM 960 is a work area for various processors. The DMA controller 970 controls DMA transfer between the processor and the memory. The DVD drive 980 (may be a CD drive) accesses a DVD 982 (may be a CD) in which programs, image data, sound data, and the like are stored. The communication interface 990 performs data transfer with the outside via a network (communication line, high-speed serial bus).

  The processing of each unit (each unit) in this embodiment may be realized entirely by hardware, or may be realized by a program stored in an information storage medium or a program distributed via a communication interface. Also good. Alternatively, it may be realized by both hardware and a program.

  When the processing of each part of this embodiment is realized by both hardware and a program, a program for causing the hardware (computer) to function as each part of this embodiment is stored in the information storage medium. More specifically, the program instructs the processors 902, 904, 906, 910, and 930, which are hardware, and passes data if necessary. Each processor 902, 904, 906, 910, 930 realizes the processing of each unit of the present invention based on the instruction and the passed data.

  The present invention is not limited to that described in the above embodiment, and various modifications can be made. For example, terms cited as broad or synonymous terms in the description in the specification or drawings can be replaced with broad or synonymous terms in other descriptions in the specification or drawings. Further, the vertex moving method or the vibration data generating method is not limited to that described in the present embodiment, and methods equivalent to these methods are also included in the scope of the present invention.

  In the present embodiment, the case of expressing the state of vibration of the diaphragm of the speaker has been described as an example. However, the present invention is not limited to this, and for example, glass with a loud sound output from a speaker or the like. It can also be applied to the case of expressing the state of vibration. Further, in the present embodiment, the case where the vertices of the part objects constituting the model object are moved has been described, but the method of the present embodiment can also be applied to the case where the entire vertices of the model object are moved.

  The present invention can also be applied to various games (such as fighting games, shooting games, robot fighting games, sports games, competitive games, role playing games, music playing games, dance games, etc.). The present invention is also applicable to various image generation systems such as a business game system, a home game system, a large attraction system in which a large number of players participate, a simulator, a multimedia terminal, a system board for generating game images, and a mobile phone. it can.

The functional block diagram of the image generation system of this embodiment. FIG. 2A to FIG. 2C are explanatory diagrams of the method of this embodiment. FIG. 3A and FIG. 3B are explanatory diagrams of the method of this embodiment. FIG. 4A and FIG. 4B are explanatory diagrams of the method of this embodiment. FIG. 5A to FIG. 5C are explanatory diagrams of the method of this embodiment. FIG. 6A to FIG. 6C are explanatory diagrams of the method of this embodiment. The flowchart which shows the specific process example of this embodiment. The flowchart which shows the specific process example of this embodiment. The flowchart which shows the specific process example of this embodiment. Hardware configuration example.

Explanation of symbols

MOB model object, POB part object,
100 processing unit,
110 Object space setting unit, 112 Movement / motion processing unit,
114 virtual camera control unit, 116 vibration data generation unit,
120 image generation units, 122 vertex shader units, 124 pixel shader units,
130 sound generation unit, 160 input unit, 162 operation unit, 164 sound detection unit,
170 storage unit,
172 Main memory unit, 174 drawing buffer,
176 sound data storage unit, 178 vibration data storage unit,
180 information storage medium, 190 display unit, 192 sound output unit,
194 Portable information storage device, 196 communication unit

Claims (11)

A program for generating an image that can be seen from a given viewpoint in an object space,
A sound data storage unit for storing sound data sampled at a sampling rate higher than the frame rate;
A vibration data storage unit for storing vibration data sampled at the frame rate, the data corresponding to the sound data;
When the sound data is reproduced and output, the vibration generation data corresponding to the reproduction frame is acquired, and based on the vibration data, an image generation unit that performs vertex shading that moves the vertices constituting the object,
A program characterized by causing a computer to function. In claim 1,
The sound data is acquired, and the computer is caused to function as a vibration data generation unit that generates the vibration data corresponding to a reproduction frame of the sound data,
The vibration data generation unit
A program characterized in that the vibration data corresponding to the Nth reproduction frame is generated by performing a filtering process on a plurality of sampling data of the sound data reproduced and output in the Nth reproduction frame. In claim 2,
The vibration data generation unit
A program characterized in that a process of averaging a plurality of sampling data of the sound data reproduced and output in the Nth reproduction frame is performed as the filtering process. In claim 2,
The vibration data generation unit
A program for performing, as the filtering process, a process of extracting sampling data having a maximum or minimum amplitude from a plurality of sampling data of the sound data reproduced and output in the Nth reproduction frame. In any one of Claims 2-4,
The vibration data generation unit
A program that acquires sound data obtained by sampling a detection sound of a sound detection unit at a sampling rate higher than the frame rate, and generates the vibration data based on sound data corresponding to the acquired detection sound. In any one of Claims 2-5,
The vibration data generation unit
A program for generating the vibration data corresponding to the sound data for the Nth playback frame at least before the end of the (N-1) th playback frame. In any one of Claims 1-6,
The image generator
A program for performing vertex shading for moving a vertex of an object based on the vibration data and attribute data set corresponding to the object. In any one of Claims 1-7,
The image generator
A program for obtaining movement information of a plurality of vertices constituting the object by interpolation processing of the vibration data, and performing vertex shading for moving each vertex based on the movement information. In any one of Claims 1-7,
The image generator
Obtaining vertex movement information based on the vibration data and a preset movement vector for each vertex constituting the object, and performing vertex shading to move each vertex based on the movement information; A featured program.   An information storage medium readable by a computer, wherein the program according to any one of claims 1 to 9 is stored. An image generation system for generating an image visible from a given viewpoint in an object space,
A sound data storage unit for storing sound data sampled at a sampling rate higher than the frame rate;
A vibration data storage unit for storing vibration data sampled at the frame rate, the data corresponding to the sound data;
When the sound data is reproduced and output, the image generation unit that acquires the vibration data corresponding to the reproduction frame, and performs vertex shading that moves the vertices constituting the object based on the vibration data;
An image generation system comprising:

Images