JP2009005910A - Program, information storage medium, game device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a program capable of eliminating the lag between a moving image and sound when an error such as skipping frames is generated in a processing for generating an image constituting the moving image in real time without giving a viewer a sense of incongruity, an information storage medium and a game device. <P>SOLUTION: Whether the moving image has a time lag in relation to the sound or not is determined based on the value of a delay counter. When the moving image is determined to have a time lag, a processing for setting a reproduction frequency to a compensation frequency which is lower than a reference frequency is executed. Namely, by reducing the reproduction speed of the sound by lowering the reproduction frequency than the reference frequency, the amount of the time lag between the moving image and the sound is gradually decreased, and the synchronization of the moving image and the sound is achieved. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a program, an information storage medium, and a game device.

  2. Description of the Related Art Conventionally, there are game devices that perform processing for generating a moving image in real time and processing for reproducing sound in parallel. In such a game device, a frame drop may occur, and there may be a gap between the moving image and the sound. If the shift is small, the viewer does not feel uncomfortable, but if a large shift occurs due to repeated frame dropping, the viewer will feel uncomfortable.

  Patent Document 1 describes an example in which a process of thinning out sound during fast-forward playback or a process of changing the playback frequency of sound is performed to reduce the uncomfortable feeling of sound during fast-forward playback. However, the invention described in Patent Document 1 cannot eliminate the gap between the moving image and the sound when an unexpected error such as a frame drop occurs.

  Further, Patent Document 2 describes an example in which a delay amount until a video is output is calculated, and the video and sound are synchronized based on the delay amount. However, in the invention described in Patent Document 2, since the timing of synchronizing the video and the sound is abrupt, a discontinuous portion is generated in the sound data, and a sound in which noise is mixed is reproduced. Therefore, there is a possibility that the viewer feels uncomfortable.

Patent Document 3 discloses a method for controlling image generation so as to be synchronized with sound. However, Japanese Patent Laid-Open No. 2004-133867 cannot eliminate the deviation from the delay of the moving image with respect to the sound.
JP 2007-13358 A JP 2006-217006 A JP 2000-107455 A

  The present invention has been made in view of the problems as described above. The object of the present invention is to generate an error such as a frame drop in a process for generating an image constituting a moving image in real time, and to generate a moving image with respect to sound. It is an object of the present invention to provide a program, an information storage medium, and a game device that can eliminate a gap between a moving image and a sound without causing a viewer to feel a sense of discomfort even if the moving image and the sound are shifted due to the delay.

(1) The present invention
A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
An audio processing unit that sequentially reads sound data to be played in accordance with an elapsed time from the start time into a storage unit, and performs a process of sequentially playing the read sound data;
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is not delayed, the reference frequency is set as the reproduction frequency, and the sound data is reproduced according to the reference frequency.
When it is determined that the moving image is delayed, a process of setting a correction frequency lower than the reference frequency as a reproduction frequency based on the value of the delay counter and reproducing sound data using the correction frequency; Process to update the value of the delay counter,
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
The present invention relates to a game apparatus characterized by performing a process of subtracting a value corresponding to a reproduction time of sound data by the correction frequency from a value of a delay counter.

  The present invention also relates to a program that causes a computer to function as each of the above-described units and a computer-readable information storage medium that stores such a program.

  According to the present invention, it is possible to eliminate a shift between a moving image and sound that occurs when an image constituting a moving image is not generated within a scheduled time. That is, according to the present invention, it is determined whether the moving image is delayed by referring to the value of the delay counter, and when it is determined that the moving image is delayed with respect to the sound, The sound data is reproduced with a lower correction frequency, and the sound reproduction speed is reduced. Therefore, the delay of the moving image with respect to the sound can be eliminated.

  According to the present invention, in the process of generating the image constituting the moving image in real time, for example, when it is determined that the image constituting the moving image is not generated within the scheduled time due to an error such as a frame drop, A process corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter. Then, in the process of reproducing the sound, when a delay of the moving image is detected, a process of subtracting a value corresponding to the reproduction time of the sound data by the correction frequency from the value of the delay counter is performed. That is, according to the present invention, the magnitude of the value of the delay counter can be determined as the amount of deviation between the moving image and the sound (the moving image delay amount with respect to the sound).

  Therefore, according to the present invention, when it is determined that the moving image is delayed, an appropriate correction frequency can be set as the reproduction frequency in accordance with the amount of deviation between the moving image and the sound. For example, the larger the value of the delay counter (the greater the difference between the moving image and the sound), the slower the sound reproduction speed can be set by setting the correction frequency to a lower band.

(2) Also, in the game device, program, and information storage medium of the present invention,
The audio processing unit is
When it is determined that the moving image is delayed, the difference between the reproduction frequency being reproduced and the correction frequency is lower than the reference frequency based on the value of the delay counter so that the difference is limited to a predetermined band. The correction frequency may be obtained and the obtained correction frequency may be set as the reproduction frequency.

  According to the present invention, since the reproduction frequency lower than the reference frequency is obtained based on the value of the delay counter so that the difference between the reproduction frequency being reproduced and the correction frequency is limited to a predetermined band, the reproduction frequency is changed. It is possible to prevent the viewer from noticing the accompanying change in pitch. That is, if the difference between the reproduction frequency being reproduced and the correction frequency is large, there is a high possibility that the viewer will be perceived by a change in the pitch, but according to the present invention, the reproduction frequency being reproduced and the correction frequency are corrected. By limiting the difference from the frequency to a predetermined band, it is possible to limit the change in the pitch of the reproduced sound that accompanies the change in the reproduction frequency, so that the viewer cannot perceive the change in the pitch. For example, if the predetermined band is limited to a band that many viewers cannot perceive, it is possible to prevent the viewer from feeling uncomfortable with the change in pitch.

(3) The present invention also provides:
A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
An audio processing unit that sequentially reads sound data to be played in accordance with an elapsed time from the start time into a storage unit, and performs a process of sequentially playing the read sound data;
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is delayed, a process of inserting dummy sound data as a reproduction target in the storage unit and a process of updating the value of the delay counter are performed.
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
The present invention relates to a game apparatus characterized by performing a process of subtracting a value corresponding to an insertion amount of the dummy sound data from a value of a delay counter.

  The present invention also relates to a program that causes a computer to function as each of the above-described units and a computer-readable information storage medium that stores such a program.

  According to the present invention, it is possible to eliminate a shift between a moving image and sound that occurs when an image constituting a moving image is not generated within a scheduled time. That is, according to the present invention, it is determined whether or not the moving image is delayed by referring to the value of the delay counter, and when it is determined that the moving image is delayed with respect to the sound, Since the dummy sound data is inserted as a reproduction target and the sound data including the inserted dummy sound data is reproduced, the deviation caused by the delay of the moving image with respect to the sound can be eliminated.

  Further, according to the present invention, in the process of generating the image constituting the moving image in real time, for example, when it is determined that the image constituting the moving image is not generated within the scheduled time due to an error such as a frame drop, A process corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter. Then, in the process of reproducing the sound, a process of subtracting a value corresponding to the insertion amount of dummy sound data from the value of the delay counter is performed. Therefore, according to the present invention, the magnitude of the value of the delay counter can be determined as the shift amount between the moving image and the sound (the moving image delay amount with respect to the sound).

(4) In the game device, program, and information storage medium of the present invention,
The audio processing unit is
The size of the dummy sound data may be obtained according to the value of the delay counter.

  According to the present invention, an appropriate size of dummy sound data can be obtained according to the value of the delay counter indicating the amount of deviation between the moving image and the sound. For example, when the value of the delay counter is large (when the difference between the moving image and the sound is large), the size is larger than when the value of the delay counter is small (when the difference between the moving image and the sound is small). The dummy sound data is inserted. In this way, even when the gap between the moving image and the sound is large, the synchronization between the moving image and the sound can be achieved at an early stage.

(5) Moreover, in the game device, program and information storage medium of the present invention,
The audio processing unit is
In a sound data string composed of a plurality of sound data read into the storage unit, a reproduction position where the amplitude of the reproduction sound is lower than a predetermined value is detected, and the dummy sound data is inserted into the detected reproduction position. May be.

  According to the present invention, a playback position in which the amplitude of the playback sound is lower than a predetermined value is detected from a sound data string composed of a plurality of sound data, and dummy sound data is inserted into the detected playback position. Therefore, since the dummy sound data is inserted at the reproduction position where the volume is low, it is possible to prevent the viewer from noticing the reproduction sound of the dummy sound data.

(6) In the game device, program, and information storage medium of the present invention,
The audio processing unit is
In a sound data sequence composed of a plurality of sound data read into the storage unit, a reproduction position with a low amplitude change rate of the reproduced sound may be detected, and the dummy sound data may be inserted into the detected reproduction position.

  According to the present invention, since the reproduction position where the amplitude change rate of the reproduced sound is low is detected and the dummy sound data is inserted into the detected reproduction position in the sound data string composed of a plurality of sound data, the sound data and the dummy sound data The discontinuous step of the data can be reduced. That is, since it is possible to prevent the occurrence of noise (noise) caused by discontinuous steps in the data, it is possible to insert dummy sound data without causing the viewer to feel uncomfortable.

(7) Also, in the game device, program and information storage medium of the present invention,
The audio processing unit is
In a sound data string composed of a plurality of sound data read into the storage unit, a reproduction position where the amount of frequency components contained in the reproduction sound is larger than a predetermined amount is detected, and the dummy sound data is inserted into the detected reproduction position. You may do it.

  According to the present invention, since the dummy sound data is inserted into the reproduction position where the amount of the frequency component included in the reproduced sound is larger than the predetermined amount in the sound data string composed of a plurality of sound data, the inserted dummy sound data Noise is masked. That is, the reproduction sound of the dummy sound data can be made inconspicuous, and the difference between the moving image and the sound can be eliminated without giving the viewer a sense of incongruity.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration Hereinafter, the present embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

  FIG. 1 shows an example of a functional block diagram of the game apparatus (image generation system) of the present embodiment. Note that the game device of this embodiment may have a configuration in which some of the components (each unit) in FIG. 1 are omitted.

  The operation unit 160 is used by the player to input operation data of the moving body, and the function thereof can be realized by a lever, a button, a steering wheel, a microphone, a touch panel display, a housing, or the like.

  The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM, a VRAM, or the like. The storage unit 170 of the present embodiment includes a main storage unit 172 serving as a work area of the processing unit 100, a drawing buffer 174 that stores color information and the like (image information) of pixels constituting the image, and a sound to be reproduced. A sound buffer 176 in which data is stored, and a delay counter 178 indicating the amount of delay of the moving image with respect to the sound (the amount of deviation between the moving image and the sound) are included.

  In particular, among the sound samples (sound data) stored in the information storage medium 180, the sound buffer to be reproduced is loaded into the sound buffer 176 of this embodiment. That is, sound samples stored in the information storage medium 180 are sequentially read into the sound buffer 176. Here, a sample is data of a sound signal divided in time, and can be a value at a time point discrete in time obtained by sampling (sampling) that converts an analog signal into a digital signal. .

  Further, the sound buffer 176 of this embodiment may store a sound sample including a dummy sample by inserting a dummy sample (dummy sound data) into the sound sample.

  Here, the dummy sample is a sample that is added (inserted) to the sound sample loaded from the information storage medium 180 to the sound buffer 176 in order to eliminate the difference between the moving image and the sound caused by the delay of the moving image. is there. For example, the dummy sample can be a sample consisting of silence. The dummy sample may be a sample obtained by making the sample at the position detected by the sound data control unit 135 redundant.

  The sound buffer 176 may reserve a plurality of sound buffers. For example, the sound data control unit 135 detects the insertion position of the dummy sample in the sound sample in which the sound sample stored in the information storage medium 180 is sequentially read into the first sound buffer, and the detected position Insert a dummy sample into Then, the inserted sound sample (sound sample including a dummy sample) may be stored in the second sound buffer. In such a case, the sound reproduction unit 131 performs a process of reproducing the sound sample stored in the first sound buffer when there is no video delay, and the video delay occurs. If so, a sample of the sound stored in the second sound buffer may be reproduced.

  The delay counter 178 of the present embodiment is data indicating the amount of delay of the moving image with respect to the sound (the amount of deviation between the sound and the moving image), and can be, for example, data in units of samples, seconds, and frames. . In addition, the value of the delay counter 178 of the present embodiment is updated based on the determination result by the image generation processing determination unit 114 of the moving image processing unit 110. More specifically, the value of the delay counter 178 exceeds the scheduled time when the image generation processing determining unit 114 determines that the image constituting the moving image is not generated within the scheduled time. A value corresponding to the excess time is added.

  Further, the delay counter 178 of the present embodiment is updated in the audio processing unit 130 based on the determination result by the delay determination unit 132. More specifically, a value corresponding to the reproduction time of the sound data with the correction frequency is subtracted from the value of the delay counter. Further, a value corresponding to the insertion amount (number of samples) of dummy samples is subtracted from the value of the delay counter.

  Note that the delay counter 178 of the present embodiment can perform processing for updating both the moving image processing unit 110 and the audio processing unit 130. Therefore, exclusive control is performed when updating the value of the delay counter 178 in each of the moving image processing unit 110 and the audio processing unit 130. That is, when the moving image processing unit 110 updates the value of the delay counter 178, the audio processing unit 130 locks the delay counter value so that the delay counter value cannot be updated. After the update, a process to release the lock is performed. On the other hand, when the value of the delay counter 178 is updated in the audio processing unit 130, the moving image processing unit 110 is locked so that the delay counter value cannot be updated, and the audio processing unit 130 sets the value of the delay counter value. After the update, a process to release the lock is performed.

  The information storage medium 180 (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (CD, DVD), magneto-optical disk (MO), magnetic disk, hard disk, and magnetic tape. Alternatively, it can be realized by a memory (ROM), a memory card, or the like. The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, the information storage medium 180 stores a program for causing a computer to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit). The information storage medium 180 may store player personal data, game save data, and the like. In addition, in the information storage medium 180 of the present embodiment, a sample of sound (sound data) to be loaded into the sound buffer 176 for performing a process of sequentially reproducing sound data in the audio processing unit 130 can be stored. .

  Note that a program (data) for causing a computer to function as each unit of the present embodiment is distributed from the information storage medium of the host device (server) to the information storage medium 180 (storage unit 170) via the network and communication unit 196. May be. Use of the information storage medium of such a host device (server) can also be included in the scope of the present invention.

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by a CRT, LCD, touch panel display, HMD (head mounted display), or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The processing unit 100 (processor) performs game processing, image generation processing for generating images constituting a moving image, processing for reproducing sound, and the like based on operation data and programs from the operation unit 160. The processing unit 100 performs various processes using the main storage unit 172 in the storage unit 170 as a work area, and the function of the processing unit 100 is hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.) It can be realized by a program.

  The processing unit 100 includes a moving image processing unit 110 and an audio processing unit 130. The moving image processing unit 110 includes an object space setting unit 111, a movement / motion processing unit 112, a virtual camera control unit 113, and an image generation processing determination unit 114. The delay counter control unit 115 is included. The audio processing unit 130 includes a sound reproduction unit 131, a delay determination unit 132, a reproduction frequency setting unit 133, a delay counter control unit 134, and a sound data control unit 135. Note that some of these may be omitted.

  The moving image processing unit 110 starts at the same time as the start of the audio processing unit, sequentially generates images constituting the moving image, and performs processing to sequentially output the generated images to the display unit.

  The object space setting unit 111 represents a display object such as a moving object, an object, a particle, dead leaves, snow, rain, a character, a building, a stadium, a car, an airplane, a ship, a submarine, a tree, a pillar, a wall, and a map (terrain). Various objects (objects composed of primitives such as polygons, free-form surfaces or subdivision surfaces) are arranged and set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects.

  The movement / motion processing unit 112 performs a movement / motion calculation (movement / motion simulation) of an object (car, train or airplane, dead leaves, snow, rain, etc.). That is, based on operation data input by the player through the operation unit 160, a program (movement / motion algorithm), various data (motion data), or the like, the object is moved in the object space, or the object is moved (motion, animation). ) Process. Specifically, a simulation process for sequentially obtaining object movement information (position, rotation angle, speed, or acceleration) and motion information (position or rotation angle of each part constituting the object) according to the frame rate is performed. . For example, when the frame rate is 60 fps, simulation processing is sequentially performed every frame (1/60 seconds). The frame is a unit of time for performing object movement / motion processing (simulation processing) and image generation processing.

  The virtual camera control unit 113 performs a virtual camera (viewpoint) control process for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, processing for controlling the position (X, Y, Z) or rotation angle (rotation angle about the X, Y, Z axis) of the virtual camera (processing for controlling the viewpoint position, the line-of-sight direction or the angle of view) I do.

  For example, when an object (eg, character, ball, car) is photographed from behind using a virtual camera, the position or rotation angle of the virtual camera (the direction of the virtual camera is set so that the virtual camera follows changes in the position or rotation of the object. ) To control. In this case, the virtual camera can be controlled based on information such as the position, rotation angle, or speed of the object obtained by the movement / motion processing unit 112. Alternatively, control for rotating the virtual camera at a predetermined rotation angle or control for moving the virtual camera along a predetermined movement path may be performed. In this case, the virtual camera is controlled based on the virtual camera data for specifying the position (movement path) or rotation angle of the virtual camera. When there are a plurality of virtual cameras (viewpoints), the above control process is performed for each virtual camera.

  The image generation process determination unit 114 performs a process of determining whether or not an image constituting a moving image has been generated within a scheduled time. For example, when the frame rate is 60 fps (every 1/60 seconds) and the image of each frame is monitored, it is determined whether or not an image of one frame has been generated within 1/60 seconds that is the scheduled time. The process to judge is performed. Here, the process of generating an image of one frame refers to a process from the calculation of object movement / motion in one frame and the vertex calculation of polygons until the final output to the frame buffer.

  If it is determined that the image has not been generated within 1/60 seconds, the image generation processing determination unit 114 determines again whether or not the image has been generated at the next frame timing. For example, when the A image is not generated within 1/60 seconds, it is determined whether or not the A image is generated at the next frame timing (after 1/60 seconds).

  Note that the image generation processing determination unit 114 may determine whether a predetermined number of frames has been generated within a scheduled time. For example, it may be determined whether images for five frames are generated within a scheduled time (for example, within 5/60 seconds in the case of 60 fps).

  Further, the image generation processing determination unit 114 may change the scheduled time when the frame rate is changed. For example, when determining an image for each frame, the scheduled time may be changed from 1/60 seconds to 1/30 seconds when the frame rate is changed from 60 fps to 30 fps.

  The delay counter control unit 115 performs processing for updating the value of the delay counter 178 based on the determination result in the image generation processing determination unit 114. In particular, the delay counter control unit 115 adds a value corresponding to the excess time exceeding the scheduled time to the value of the delay counter when it is determined that the image constituting the moving image is not generated within the scheduled time. Perform the process.

  For example, when the sound is reproduced at the reference frequency (48000 Hz) and the frame rate of the image processing is 60 fps, and it is determined that the image is not generated within the scheduled time (1/60 seconds), the unit of the delay counter is sampled. In the case of handling in (1), processing for adding the excess 800 samples to the value of the delay counter is performed. That is, a process of adding 800 samples corresponding to the number of sound samples for one frame (1/60 second) to the value of the delay counter is performed. When the delay counter is controlled in units of time, a process of adding 1/60 seconds to the value of the delay counter is performed.

  The drawing unit 120 performs drawing processing based on the results of various processing (game processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. When a so-called three-dimensional game image is generated, first, vertex data (vertex position coordinates, texture coordinates, color data, normal vector, α value, etc.) of each vertex defining a display object (object, model) is included. Display object data (object data, model data) is input, and vertex processing is performed based on vertex data included in the input display object data. When performing the vertex processing, vertex generation processing (tessellation, curved surface division, polygon division) for re-dividing the polygon may be performed as necessary. In vertex processing, geometry processing such as vertex movement processing, coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed, and the display object is configured based on the processing result. The given vertex data is changed (updated, adjusted) for the vertex group. Then, rasterization (scan conversion) is performed based on the vertex data after the vertex processing, and the surface of the polygon (primitive) is associated with the pixel. Subsequent to rasterization, pixel processing (fragment processing) for drawing pixels constituting an image (fragments constituting a display screen) is performed. In pixel processing, various processes such as texture reading (texture mapping), color data setting / changing, translucent composition, anti-aliasing, etc. are performed to determine the final drawing color of the pixels that make up the image, and perspective transformation is performed. The drawing color of the object is output (drawn) to the drawing buffer 174 (buffer that can store image information in units of pixels; VRAM, rendering target). That is, in pixel processing, per-pixel processing for setting or changing image information (color, normal, luminance, α value, etc.) in units of pixels is performed. Thereby, an image that can be seen from the virtual camera (given viewpoint) set in the object space is generated. Note that when there are a plurality of virtual cameras (viewpoints), an image can be generated so that an image seen from each virtual camera can be displayed as a divided image on one screen.

  The vertex processing and pixel processing performed by the drawing unit 120 are realized by hardware that enables polygon (primitive) drawing processing to be programmed by a shader program written in a shading language, so-called programmable shaders (vertex shaders and pixel shaders). May be. Programmable shaders can be programmed with vertex-level processing and pixel-level processing, so that the degree of freedom of rendering processing is high, and the expressive power can be greatly improved compared to fixed rendering processing by hardware. .

  The drawing unit 120 performs geometry processing, texture mapping, hidden surface removal processing, α blending, and the like when drawing the display object.

  In the geometry processing, processing such as coordinate conversion, clipping processing, perspective projection conversion, or light source calculation is performed on the display object. Then, display object data after geometric processing (after perspective projection conversion) (position coordinates of vertex of the display object, texture coordinates, color data (luminance data), normal vector, α value, etc.) is stored in the main storage unit 172. Saved.

  Texture mapping is a process for mapping a texture (texel value) stored in the storage unit 170 to a display object. Specifically, the texture (surface properties such as color (RGB) and α value) is read from the storage unit 170 using texture coordinates or the like set (given) at the vertex of the display object. Then, a texture that is a two-dimensional image is mapped to a display object. In this case, processing for associating pixels with texels, bilinear interpolation or the like is performed as texel interpolation.

  As the hidden surface removal processing, hidden surface removal processing can be performed by a Z buffer method (depth comparison method, Z test) using a Z buffer (depth buffer) in which Z values (depth information) of drawing pixels are stored. . That is, when drawing pixels corresponding to the primitive of the object are drawn, the Z value stored in the Z buffer is referred to. Then, the Z value of the referenced Z buffer is compared with the Z value at the drawing pixel of the primitive, and the Z value at the drawing pixel is a Z value (for example, a small Z value) on the near side when viewed from the virtual camera. In some cases, the drawing process of the drawing pixel is performed and the Z value of the Z buffer is updated to a new Z value.

  α blending (α synthesis) is a translucent synthesis process (usually α blending, addition α blending, subtraction α blending, or the like) based on an α value (A value). For example, in normal α blending, a process for obtaining a color obtained by combining two colors by performing linear interpolation with the α value as the strength of synthesis is performed.

  The α value is information that can be stored in association with each pixel (texel, dot), and is, for example, plus alpha information other than color information indicating the luminance of each RGB color component. The α value can be used as mask information, translucency (equivalent to transparency and opacity), bump information, and the like.

  The audio processing unit 130 sequentially reads sound samples (sound data) to be reproduced according to the elapsed time from the start time that starts at the same time as the moving image processing unit 110 into the sound buffer 176, and reads the read sound samples. Performs sequential playback processing.

  The sound reproduction unit 131 performs a process of reproducing the sound sample read into the sound buffer 176 according to the reproduction frequency. For example, the sound reproducing unit 131 may perform a process of reproducing a sound sample with a reference frequency, or may reproduce a sound sample with a correction frequency.

  Here, the reproduction frequency is the number of sound samples reproduced (output) per second. For example, the reference frequency when performing normal reproduction with a DVD is 48000 Hz. In other words, reproducing a sound sample with a reference frequency of 48000 Hz means reproducing a sound of 48000 samples per second.

  Further, the sound reproducing unit 131 may perform a process of reproducing a sound sample in which a dummy sample (dummy sound data) is inserted.

  In addition, the sound reproduction unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates game sounds such as BGM, sound effects, or sounds, and outputs the game sounds to the sound output unit 192. Can do.

  The delay determination unit 132 performs a process of determining whether the moving image is delayed with respect to the sound based on the value of the delay counter 178. For example, for example, zero is set with the value of the delay counter at the start time simultaneously started in the moving image processing unit 110 and the audio processing unit 130 as an initial value. That is, the value of the delay counter when the sound and the moving image are synchronized is set as zero. Then, by referring to the value of the delay counter and determining whether or not the value of the delay counter is greater than zero, it is possible to determine whether or not the moving image is delayed with respect to the sound. When the value of the delay counter is zero (or less than zero), it is determined that there is no delay.

  When the delay determination unit 132 determines that the moving image is not delayed, the reproduction frequency setting unit 133 performs processing for setting the reference frequency as the reproduction frequency. The reproduction frequency setting unit 133 sets a correction frequency lower than the reference frequency as the reproduction frequency based on the value of the delay counter when the delay determination unit 132 determines that the moving image is delayed. I do.

  Further, when it is determined that the moving image is delayed, the reproduction frequency setting unit 133 is based on the value of the delay counter so that the difference between the reproduction frequency being reproduced and the correction frequency is limited to a predetermined band. Thus, a correction frequency lower than the reference frequency can be obtained and the obtained correction frequency can be set as the reproduction frequency. For example, the difference between the reproduction frequency being reproduced and the reproduction frequency to be corrected is limited to 60 Hz. For example, when the reproduction frequency during reproduction is 48000 Hz, a correction frequency of 47940 Hz can be set as the reproduction frequency.

  The delay counter control unit 134 performs processing for updating the value of the delay counter 178 in the audio processing unit 130. More specifically, the delay counter control unit 134 performs a process of subtracting a value corresponding to the reproduction time of the sound data with the correction frequency from the value of the delay counter. For example, when the delay counter is controlled in units of samples, if the correction frequency is 47940 Hz, the amount of deviation between the sound for one sample and the moving image can be eliminated by reproducing 1/60 second. Then, a process of subtracting one sample from the value of the delay counter is performed.

  In addition, when the audio processing unit 130 performs a process of inserting a dummy sample, the delay counter control unit 134 performs a process of subtracting a value corresponding to the insertion amount from the value of the delay counter. For example, when the insertion amount of the inserted dummy sample is 50 samples, a process of subtracting 50 samples from the value of the delay counter is performed.

  When the delay determination unit 132 determines that the moving image is delayed, the sound data control unit 135 performs a process of inserting a dummy sample into the sound buffer 176 as a reproduction target. That is, a process of inserting a dummy sample into the sound sample read into the sound buffer 176 is performed. Note that the sound sample including the inserted dummy sample may be stored again in the sound buffer 176 in which the sound sample before insertion is stored, or may be stored in another sound buffer.

  Here, inserting a dummy sample into a sound sample means inserting a dummy sample at a given position of a sound (waveform) based on a time axis (time domain).

  The sound data control unit 135 may determine the size of the dummy sample according to the value of the delay counter. For example, when the value of the delay counter is 800 samples, dummy samples having the number of samples of 800 samples or less can be inserted. If the size stored in the sound buffer is predetermined, the dummy size is set so that the sum of the size of the already loaded sound sample and the size of the dummy sample is less than the size that can be stored in the sound buffer. Determine the size of the sample.

  In addition, the sound data control unit 135 detects a reproduction position in which the amplitude of the reproduction sound is lower than a predetermined value in the sound sample (a plurality of sound samples) read into the sound buffer, and the detected reproduction position You may make it insert a dummy sample in. Here, the amplitude refers to the amplitude of the sound waveform indicating the time domain. For example, the sound data control unit 135 may detect a sample whose amplitude is lower than 10 dB (decibel) from the sound samples, and insert a dummy sample at the detected position. Note that the sound data control unit 135 may detect a sample whose amplitude is close to zero and insert a dummy sample in which the detected sample is redundant at the position of the detected sample.

  Further, the sound data control unit 135 detects a reproduction position where the amplitude change rate of the reproduced sound is low among a plurality of sound samples read into the sound buffer, and inserts a dummy sample at the detected reproduction position. Also good.

  Here, the low rate of change in amplitude means that the absolute value of the slope (differential coefficient) of the tangent line of the sound waveform is low in the sound waveform indicating the time domain. For example, a sample in which the absolute value of the slope of the tangent line of the sound waveform is zero may be detected, and a dummy sample may be inserted at the detected position.

  Also, the sound data control unit 135 detects a reproduction position in which the amount of frequency components included in the reproduced sound is greater than a predetermined amount from among a plurality of sound samples read into the sound buffer, and a dummy sample is detected at the detected reproduction position. May be inserted. Here, the frequency component included in the reproduced sound can be obtained by performing Fourier transform on the time-domain waveform of the sound sample read into the sound buffer. The sound data control unit 135 according to the present embodiment divides the time domain waveform of the sound sample into predetermined areas, and the area of the waveform in the frequency domain (the horizontal axis is the frequency and the vertical axis is the magnitude of the frequency component) for each area. Seeking. In other words, in this embodiment, the area of the waveform in the frequency domain is regarded as the amount of the frequency component included in the reproduced sound, and a given position in the area where the area of the waveform in the frequency domain is larger than a predetermined amount (predetermined area) is detected. Then, a dummy sample is inserted at the detected position. Note that the area of the waveform in the frequency domain can be obtained by integrating the waveform in the frequency domain. Further, when the time domain waveform is divided into a plurality of areas, it may be divided at predetermined intervals (for example, a basic period).

  Note that the game device of the present embodiment may be a system dedicated to the single player mode in which only one player can play, or may be a system having a multiplayer mode in which a plurality of players can play. Further, when a plurality of players play, game images and game sounds to be provided to the plurality of players may be generated using one terminal, or connected via a network (transmission line, communication line) or the like. Alternatively, it may be generated by distributed processing using a plurality of terminals (game machine, mobile phone).

2. Method according to this embodiment The game device according to this embodiment starts a process for generating a moving image and a process for reproducing a sound at the same time. In other words, in the present embodiment, a process for generating a moving image that sequentially generates and outputs images constituting a moving image in real time, and a sample of a sound to be reproduced according to elapsed time is read into a sound buffer, and the read sound is The process of sequentially reproducing the samples is started at the same time, and the processes are performed in parallel.

  In this embodiment, a sample of a sound to be played stored in an information storage medium such as a DVD is sequentially read into a sound buffer and sequentially played. Since the sound sample is created in advance according to the elapsed time from the start time based on the content of the moving image, the moving image and the reproduced sound are synchronized unless an error occurs in the image generation process. For example, as shown in FIG. 2A, the explosion sound can be reproduced at the timing when the explosion image is displayed.

  However, there are many cases where an image generation process causes an unexpected error. This is because the image generation processing performs coordinate conversion, vertex calculation, and the like of the object in real time, so that the processing load tends to increase and there is a high possibility that an error such as a frame drop will occur. For example, as shown in FIG. 2B, when a frame drop occurs, an explosion image is displayed at timing T2 after explosion sound is reproduced at timing T1.

  Therefore, in this embodiment, in order to eliminate the gap between the moving image and the sound in this way, a method of detecting the delay of the moving image with respect to the sound and reproducing the reproduction frequency with a correction frequency lower than the reference frequency, A method of inserting a dummy sample into this sample is adopted.

2.1 Delay Counter In this embodiment, a delay counter is used to perform processing for generating an image constituting a moving image in real time and processing for reproducing sound. The delay counter indicates a moving image delay amount (deviation amount) with respect to the sound. In this embodiment, the unit of the delay counter is the number of sound samples.

  That is, the magnitude of the value of the delay counter can be grasped as the amount of deviation between the moving image and the sound. Therefore, in this embodiment, it is determined whether the moving image is delayed with respect to the sound by referring to the delay counter. For example, in the present embodiment, the processing for generating an image constituting a moving image in real time and the processing for reproducing sound are started at the same time, and the initial value of the delay counter is set to zero at the start time. When the value of is zero, it can be determined that the moving image and the sound are synchronized. When the value of the delay counter is greater than zero, it can be determined that the moving image is delayed with respect to the sound.

  Note that, as will be described below, in the present embodiment, processing for updating (adding or subtracting) the value of the delay counter is performed in accordance with a change in the amount of deviation between the moving image and the sound.

2.2 Method for detecting a delay of a moving image with respect to sound In the present embodiment, processing for determining whether an image constituting a moving image is generated at a scheduled time and updating the value of a delay counter based on the determination result. I do. For example, if it is determined that an image constituting the video is not generated within the scheduled time, such as when a frame drop error occurs, the delay counter indicates the number of samples corresponding to the excess time exceeding the scheduled time. Processing to add to the value of.

  More specifically, first, the number of frames to be monitored is determined in advance in an image constituting a moving image, and it is determined whether or not an image having the number of frames to be monitored has been generated within a scheduled time. Then, when it is determined that it has not been completed within the scheduled time, a process of adding the number of sound samples corresponding to the excess time exceeding the scheduled time to the value of the delay counter is performed.

  For example, when the frame rate is 60 fps (every 1/60 seconds) and every frame is monitored, whether the image A to be monitored is generated within the scheduled time (within 1/60 seconds). Determine whether. If it is determined that the image A has not been generated within the scheduled time (1/60 seconds), 800 samples, which are the number of sound samples to be reproduced during the excess time (1/60 seconds), are delayed. A process of adding to the counter value is performed.

  On the other hand, when the image A is generated within the scheduled time (1/60 seconds), the process of generating the next image constituting the moving image is performed without updating the value of the delay counter.

  If the image A is not generated within the scheduled time (1/60 seconds), it is determined whether or not the image A is generated within the scheduled time (1/60 seconds) at the next frame timing. For example, when the image A is not generated even at the next frame timing, a process of adding 800 samples, which is the number of sound samples to be reproduced in the excess time (1/60 seconds), to the value of the delay counter is performed. .

  As described above, in this embodiment, when a frame drop occurs in the moving image processing, the number of sound samples corresponding to the delay amount is added to the value of the delay counter. In other words, an increase in the delay counter value indicates that the amount of deviation between the moving image and the sound is large.

  In addition, in the process of generating the image constituting the moving image of the present embodiment in real time, the number of frames to be monitored and the frame rate may be freely changed. In such a case, in order to appropriately detect the delay of the moving image, the scheduled time is appropriately changed according to the changed frame rate and the number of frames to be monitored.

2.3 Method of Setting Playback Frequency as Correction Frequency In the present embodiment, in the process of playing back sound, the value of the delay counter is referred to, and when it is determined that the moving image is delayed with respect to the sound, Set the reproduction frequency to a correction frequency lower than the reference frequency. That is, when the moving image is delayed with respect to the sound, it is possible to achieve the synchronization between the moving image and the sound by adjusting the sound reproduction speed.

  In this embodiment, when the value of the delay counter is zero, the sound and the moving image are synchronized, and when the moving image is larger than zero, the moving image is delayed with respect to the sound. It will be. Therefore, whether or not the moving image is delayed with respect to the sound is determined by determining whether or not the value of the delay counter is greater than zero.

  Note that the timing for determining the delay of the moving image is performed according to the frame rate of the image processing. For example, when the frame rate of the image processing is 60 fps, it is determined whether or not the moving image is delayed every 1/60.

  When it is determined that the moving image is delayed with respect to the sound, processing for setting the reproduction frequency to the correction frequency is performed. That is, the process of reproducing the sound sample with the correction frequency lower than the reference frequency is performed for the sound sample reproduced with the reference frequency (48000 Hz). As described above, by decelerating the sound reproduction speed, it is possible to eliminate the difference between the sound and the moving image.

  For example, as shown in FIG. 3, a frame drop occurs in the process of generating a moving image, and 800 samples are added to the delay counter. Then, in the process of reproducing the sound, it is detected that the moving image is delayed with respect to the sound, and the process of setting the reproduction frequency to the correction frequency is performed. For example, a process of changing the reproduction frequency from the reference frequency of 48000 Hz to the correction frequency of 47940 Hz is performed.

  Thus, in this embodiment, the amount of deviation between the moving image and the sound can be eliminated by reproducing the sound sample with the correction frequency. For example, as shown in FIG. 3, the explosion sound can be reproduced at T2, which is the same timing as the explosion image.

  A method for setting the reproduction frequency to the correction frequency will be described in detail. In the present embodiment, processing for setting the reproduction frequency to the correction frequency is performed based on the value of the delay counter. For example, a larger value of the delay counter means that the amount of delay of the moving image with respect to the sound is larger. Therefore, as the value of the delay counter is larger, the reproduction frequency is lowered (lowered). This is because as the reproduction frequency is lowered, the sound reproduction speed can be reduced and synchronization can be achieved early.

  For example, when the delay counter value is 1 to 800, the reproduction frequency is set to a correction frequency of 47940 Hz, and when the delay counter value is 801 to 1600, the reproduction frequency is set to a correction frequency of 47880 Hz. When the value of the delay counter is 1601 to 2400, the reproduction frequency is set to a correction frequency of 47820 Hz, so that the reproduction frequency is decreased by 60 Hz as the value of the delay counter increases by 800 samples. The correction frequency may be set.

  In the present embodiment, the correction frequency may be limited to a predetermined band regardless of the value of the delay counter so that the reproduction frequency does not become extremely low. This is because if the reproduction frequency is extremely lowered, a viewer with excellent hearing senses a change in the pitch of the reproduced sound. For example, although depending on the human auditory level, the reproduction frequency may be set to the correction frequency so that the reproduction frequency is limited to a band of 47500 to 48000 Hz.

  In this embodiment, a process is performed in which a value corresponding to the reproduction time of the sound sample at the correction frequency is subtracted from the value of the delay counter. This is because the amount of displacement of the moving image with respect to the sound is reduced.

  For example, when the frame rate in the process of generating an image is 60 fps, the video, the sound, and the number of samples that can eliminate the deviation by reproducing the sound sample at the correction frequency every 1/60 seconds are delayed. A process of subtracting from the counter value is performed.

  More specifically, when the correction frequency is 47940 Hz, the delay amount for one sample can be eliminated in 1/60 seconds. Therefore, a process of subtracting one sample from the value of the delay counter is performed every processing time for generating an image of one frame. In other words, when the correction frequency is 47940 Hz, by continuing to reproduce the sound, the delay of 800 samples can be eliminated in the sound reproduction time corresponding to the image generation processing time of 800 frames. And the sound can be synchronized.

  In the present embodiment, the moving image delay is determined every 1/60 seconds, and when it is determined that the moving image delay has occurred, the reproduction frequency is set to the correction frequency and the value of the delay counter is set. Subtract the number of samples that can be resolved from.

  In the present embodiment, when the value of the delay counter becomes zero in the process of reproducing the sound, it is determined that there is no delay of the moving image. Therefore, the sound sample is obtained at the reference frequency (48000 Hz). You can play and keep the video and sound synchronized.

  Next, a method for setting the reproduction frequency to the correction frequency step by step will be described. For example, when the playback frequency is suddenly changed from the reference frequency of 48000 Hz to the correction frequency of 47640 Hz as soon as the delay counter value is added due to the occurrence of a frame drop in the process of generating a moving image, the playback sound May be perceived by the viewer.

  Therefore, in this embodiment, the correction frequency is obtained so that the difference between the reproduction frequency being reproduced and the correction frequency is limited to a predetermined band. For example, the difference between the reproduction frequency during reproduction and the correction frequency is limited to 60 Hz, and the reproduction frequency is changed stepwise every 60 Hz. This is because the change in pitch can be moderated so that the viewer does not feel uncomfortable.

  For example, when a delay of a moving image is detected in the process of reproducing sound, the reproduction frequency is set to 47940 Hz, which is a correction frequency that is 60 Hz lower than the reference frequency of 48000 Hz, and the sound is reproduced by the set reproduction frequency of 47940 Hz. Process to play the sample. Then, after a predetermined playback time has elapsed, the playback frequency is further set to 47880 Hz, which is a correction frequency that is 60 Hz lower than the playback frequency of 47940 Hz being played back, and a sound sample is played back at the set playback frequency of 47880 Hz. Perform the process. In this way, sound samples are reproduced by stepping down by 60 Hz.

  Similarly, when the correction frequency is increased (increased), it is increased stepwise by 60 Hz. For example, when the reproduction frequency being reproduced is 47640 Hz, the reproduction frequency is set to 47700 Hz, which is a correction frequency increased from 47640 Hz to 60 Hz, and a process of reproducing sound samples is performed at the set reproduction frequency of 47700 Hz.

  FIG. 4 shows an example in which the delay of the moving image is eliminated by taking an image processing time (170/60 seconds) for 170 frames when the value of the delay counter is 800 samples.

  That is, FIG. 4 shows the set correction frequency and the “number of samples that can eliminate the deviation between the moving image and the sound in one frame per time (1/60 seconds)” and “correction” according to each correction frequency. The number of frames for which the frequency is maintained, the total number of samples that can eliminate the deviation with the correction frequency duration, the value of the delay counter before subtraction, and the value of the delay counter after subtraction are shown.

  For example, as shown in FIG. 4, when a sound sample is reproduced at a reproduction frequency set to a correction frequency of 47940 Hz, the deviation of one sample can be eliminated in a time per frame (1/60 seconds). it can. In other words, since the deviation of 5 samples can be eliminated by maintaining the correction frequency of 47940 Hz for 5 frames (5/60 seconds), a process of subtracting 5 samples from the value of the delay counter is performed. The value of the delay counter is 795 samples.

  In the example of FIG. 4, the correction frequency is changed step by step every 60 Hz, and the reproduction time (the number of frames that maintain the correction frequency) of the lowest correction frequency of 47640 Hz is made the longest. That is, when the change graph of the sound reproduction time and the reproduction frequency is graphed, it is as shown in FIG. In this way, it is possible to eliminate the shift at an early stage by setting the reproduction frequency stepwise and lengthening the reproduction time at the lowest correction frequency.

  In addition, when sound samples are played back at the playback frequency set as the correction frequency, if the frame loss of the images that make up the movie occurs and the value of the delay counter increases, the correction frequency can be increased. It may be changed to be lower, or the number of sustained frames (reproduction time) of the correction frequency may be changed. That is, the correction frequency may be dynamically set based on the value of the delay counter, and the number of sustained frames of the correction frequency may be determined.

  For example, as shown in FIG. 5B, when it is detected that the value of the delay counter has increased when the reproduction frequency gradually returns to the reference frequency (47800 Hz) stepwise, the reproduction is started from the point of detection. The correction frequency may be set so that the frequency decreases stepwise.

  As described above, in the present embodiment, the difference between the moving image and the sound can be eliminated by reproducing the sound sample with the correction frequency lower than the reference frequency. Further, in this embodiment, since it is possible to grasp the amount of deviation between the moving image and the sound using the delay counter, it is not necessary to sequentially obtain the moving image delay based on the absolute time axis of the game device. It is possible to reduce the processing load for synchronizing the moving image and the sound.

2.4 Technique for Reproducing Sound by Inserting Dummy Sample This embodiment employs a technique for reproducing sound by inserting a dummy sample. In other words, in the present embodiment, when it is determined that the moving image is delayed in the sound reproduction process, the dummy sample is inserted into the sound sample read into the sound buffer, and the inserted dummy sample is included. Performs playback of sound samples. By doing so, the dummy sample is inserted into the sound sample, so that the reproduction time of the sound for the dummy sample is increased, and the deviation between the moving image and the sound can be eliminated.

  In the present embodiment, as described above, whether or not the moving image is delayed with respect to the sound is determined based on the value of the delay counter. For example, if the value of the delay counter is greater than zero, it is determined that a delay has occurred.

  For example, as shown in FIG. 6, when a frame drop occurs in the process of generating a moving image and 800 samples are added to the value of the delay counter, the delay of the moving image is detected in the process of reproducing the sound, and the sound A dummy sample is inserted into the sound sample read into the buffer. In this way, the sound reproduction time can be lengthened according to the inserted dummy sample, and the deviation between the moving image and the sound can be eliminated. Therefore, as shown in FIG. 6, the explosion sound can be reproduced at T2, which is the same timing as the explosion image. In the present embodiment, it is determined whether or not the moving image is delayed according to the frame rate of the images constituting the moving image. For example, when the frame rate of the image processing is 60 fps, it is determined whether or not the moving image is delayed every 1/60.

  Further, the number of dummy samples is determined to be equal to or less than the number of samples of the delay counter value. This is because adding a dummy sample exceeding the number of samples in the delay counter may cause a situation in which the sound is delayed with respect to the moving image. For example, when the value of the delay counter is 40 samples, the number of dummy samples is determined so as to be 40 samples or less. Note that the number of dummy samples may be limited so that the sum of the sound samples and dummy samples already read into the sound buffer fits in the size of the sound buffer.

  And this embodiment detects the position which inserts a dummy sample as follows. For example, among the sound samples stored in the sound buffer, a sample whose sound amplitude (volume) is close to zero is detected. And the dummy sample which made the detected sample redundant is inserted in the position of the detected sample. In this way, even if a dummy sample is added, the silent state only continues, so that it is possible to prevent the viewer from being aware of the dummy sample insertion.

  Further, a reproduction position whose amplitude is lower than a predetermined value among sound samples stored in the sound buffer may be detected, and a dummy sample may be inserted at the detected reproduction position. For example, if the sound sample stored in the sound buffer has a waveform as shown in FIG. 7, the position R having an amplitude lower than the predetermined value d is detected. Then, a dummy sample is inserted at the detected position R. For example, a dummy sample in which the sample at position R is redundant may be inserted at position R.

  In the present embodiment, a position where the amplitude change rate is low among the sound samples stored in the sound buffer may be detected, and a dummy sample may be inserted at the detected position. This is because the discontinuous step between the sample read into the sound buffer and the inserted dummy sample can be reduced, and noise can be prevented. For example, as shown in FIG. 7, since the position R is a position where the amplitude change rate is zero, a dummy sample in which the sample at the position R is redundant may be inserted into the position R.

  In the present embodiment, among the sound samples read into the sound buffer, a reproduction position where the amount of frequency components included in the reproduction sound is larger than a predetermined amount is detected, and a dummy sample is inserted at the detected reproduction position. You may do it. By doing so, the noise of the inserted dummy sample is masked, and as a result, the uncomfortable feeling given to the viewer can be reduced.

  First, in this embodiment, when the waveform of the sound sample read into the sound buffer (the waveform in the time domain) is a sound waveform as shown in FIG. 8, the sound samples are S1, S2, and S3. Dividing into regions, the frequency components of each region are obtained. The frequency component of each region can be obtained by Fourier transforming the time-domain waveform of the sound sample read into the sound buffer for each region.

  Next, in this embodiment, the area of the waveform in the frequency domain (the X axis is the frequency and the Y axis is the magnitude of the frequency component) is determined for each of the areas S1 to S3. This is because the area of the waveform shown in the frequency domain corresponds to the amount of the frequency component. The area of the waveform in the frequency domain can be obtained by integrating the wave in the frequency domain.

  Then, for each of the areas S1 to S3, it is determined whether or not the area of the waveform in the frequency domain is larger than a predetermined amount (predetermined area), and if the area of the waveform in the frequency domain is larger than the predetermined amount A process of inserting a dummy sample at a given position in the determined area is performed.

  As described above, the present embodiment employs a method of simply obtaining the magnitude of the amount of the frequency component by obtaining the area of the waveform in the frequency domain. For example, the frequency domain waveform with a large amount of frequency components such as deep bass and treble, which is thought to increase the masking effect of the reproduced sound of the dummy sample, has a larger area than the frequency domain waveform with a small amount of frequency components. . Therefore, in this embodiment, if the waveform area is large, the amount of frequency components is considered to be large, and a dummy sample is inserted in a region where the waveform area is determined to be larger than a predetermined amount (predetermined area) set in advance. Processing to do.

  In the present embodiment, the area of the waveform in the frequency region at the frequency of the predetermined band is obtained, and the dummy area is determined to be an area determined to be larger than a predetermined amount (predetermined area) corresponding to the frequency of the predetermined band. A sample may be inserted.

  This will be specifically described with reference to FIGS. For example, FIG. 9A shows the frequency distribution in the S1 region, FIG. 9B shows the frequency distribution in the S2 region, and FIG. 9C shows the frequency distribution in the S3 region. First, in this embodiment, the area of the waveform in the frequency region in the frequency band limited to 0 to 5000 Hz in each region of S1 to S3 is obtained. For example, when it is determined that the area of the waveform in the frequency region of the S2 region is larger than a predetermined amount among the S1 to S3 regions, a process of inserting a dummy sample at a given position in the S2 region is performed. . Note that it is desirable to insert a dummy sample at a position where the amplitude is large (a position where the volume is high) among the sound samples in the S2 region. This is because the masking effect of the reproduced sound of the inserted dummy sample is further enhanced.

  In the above example, the sound sample read into the sound buffer is divided into three regions (regions S1 to S3) to obtain the frequency components in each region, but every predetermined cycle (for example, the basic cycle). A frequency component may be obtained by dividing the region, and a dummy sample may be inserted at a given position in a region where the amount of the frequency component included in the reproduced sound is greater than a predetermined amount. In addition, when playing back sound using multiple sound buffers, the frequency components are calculated for each sound buffer without dividing the area, and the sound buffer has a volume of frequency components included in the playback sound larger than the predetermined amount. A dummy sample may be inserted.

  Further, in the present embodiment, a given region in which the magnitude of the frequency component of the waveform in the frequency domain (Y-axis value) is equal to or greater than a predetermined value and the number of waveform peaks exceeds a predetermined number (for example, 12). A dummy sample may be inserted at the position. This is because if a dummy sample is inserted in such an area, the reproduced sound of the dummy sample is masked.

  As described above, it is possible to make the reproduced sound of the dummy sample inconspicuous by inserting the dummy sample at an appropriate position of the sound sample stored in the sound buffer.

  Further, when a plurality of positions where dummy samples are added are detected, the priority order of each position may be determined, and the dummy samples may be inserted in descending order of priority. A dummy sample may be inserted at each position. When inserting dummy samples at a plurality of positions, the total number of dummy samples to be inserted is limited to be equal to or less than the value of the delay counter.

  Further, when a dummy sample is inserted into a sound sample, a process of subtracting the number of inserted dummy samples from the value of the delay counter is performed. This is because the amount of deviation between the sound and the moving image is reduced by inserting the dummy sample. For example, when a dummy sample (50 samples) is inserted into the sound sample, a process of subtracting 50 samples from the value of the delay counter is performed.

  In the present embodiment, after the process of inserting dummy samples is performed, the process of subtracting the number of samples corresponding to the inserted dummy samples from the value of the delay counter is performed. For example, when the frame rate of the image processing is 60 fps, it is determined every 1/60 whether or not the moving image is delayed. If there is a delay, based on the value of the delay counter, A dummy sample is inserted into the sound sample, and the number of samples for the inserted dummy sample is subtracted from the value of the delay counter.

2.5 Method of setting playback frequency to correction frequency and inserting dummy sample In this embodiment, combining the method of setting playback frequency to correction frequency and the method of inserting dummy sample, You may make it eliminate gap.

  For example, by setting the playback frequency as the correction frequency, even if you try to eliminate the gap between the movie and the sound, if the amount of deviation is large, it will take time to synchronize, or even finally synchronize There are cases where it is not possible.

  Therefore, in this embodiment, when the value of the delay counter is larger than the predetermined value, that is, when the difference between the sound and the moving image is large, not only the reproduction frequency is set to the correction frequency but also a dummy sample is inserted. To eliminate the deviation. In this way, even if the amount of deviation between the video and the sound is large, not only the sound is reproduced with the correction frequency, but also a dummy sample is inserted, and as a result, the deviation between the sound and the video can be eliminated early. Because you can.

  For example, when the threshold value of the delay counter is set to 1600 samples and the delay counter value is lower than 1600 samples, the delay of the moving image is reduced only by the method of setting the reproduction frequency to the correction frequency. Try to eliminate. If the delay counter is 1600 samples or more, a method of inserting dummy samples is adopted. In this case, the value of the delay counter is subtracted based on the corrected reproduction frequency and reproduction time, and is subtracted according to the number of samples into which dummy data is inserted.

3. Processing of this Embodiment Next, a processing example for realizing the method of this embodiment will be described in detail with reference to FIG.

  In the game device according to the present embodiment, a process for generating a moving image (step S100) and a process for reproducing a sound (step S200) are started simultaneously, and the process is performed by multithreading.

  And in the process which produces | generates a moving image, the calculation process of the image which comprises a moving image is performed first (step S110). For example, object movement / motion calculation, vertex calculation, and the like are performed. Next, it is determined whether or not the calculation is completed (step S120). That is, it is determined whether the frame image to be monitored is generated within the scheduled time. When it is determined that the calculation has been completed (Yes in step S120), a process for displaying an image is performed (step S130). On the other hand, if it is determined that the calculation has not been completed (No in step S120), a delay amount (for example, the number of samples corresponding to the excess time when the image generation process exceeds the scheduled time) is added to the value of the delay counter (Step S140), and the process returns to step S110. The image calculation is carried over to the next frame timing.

  Then, it is determined whether or not the image display process for all the frames has been completed (step S150). If the image display for all the frames has not been completed (No in step S150), the process returns to step S110 to return to step S110. The processing from S110 to Step 150 is repeated. If the image display of all the frames has been completed (Yes in step S150), the process for generating a moving image is terminated.

  Next, a process for reproducing sound will be described. First, after the process is started (step S200), it is determined whether there is a delay (step S210). For example, it is determined whether or not the value of the delay counter is greater than zero. If it is determined that there is a delay (Yes in step S210), reproduction correction calculation is performed (step S220). For example, a process of obtaining a correction frequency and setting the obtained correction frequency as a reproduction frequency, a process of inserting a dummy sample into a sound sample of a sound buffer, and the like are performed. Then, a process of subtracting the number of canceled samples from the value of the delay counter is performed (step S230).

  On the other hand, when it is determined that there is no delay (No in step S210), a calculation process for reproducing a normal sound is performed (step S240).

  Then, a process for reproducing sound is performed (step S250), and it is determined whether or not the image display of all the frames has been completed (step S260). If the image display of all the frames has not been completed (step S260). In step S260, the process returns to step S210, and the processes in steps S210 to S260 are repeated. If the image display of all the frames has been completed (Yes in step S260), the process ends.

The example of the functional block diagram of the game device of this embodiment. FIGS. 2A and 2B are diagrams for explaining synchronization and deviation between audio and moving images. FIG. 3 is an explanatory diagram of the correction frequency. FIG. 4 is an explanatory diagram of the correction frequency. FIG. 5 is an explanatory diagram of the correction frequency. FIG. 6 is an explanatory diagram relating to a technique for inserting a dummy sample. FIG. 7 is an explanatory diagram relating to a technique for inserting a dummy sample. FIG. 8 is an explanatory diagram regarding a method of inserting a dummy sample. FIGS. 9A, 9B, and 9C are explanatory diagrams regarding a method of inserting a dummy sample. FIG. 10 is a flowchart of processing according to this embodiment.

Explanation of symbols

100 processing unit, 111 object space setting unit, 112 movement / motion processing unit,
113 virtual camera control unit, 114 image generation processing determination unit,
115 delay counter control unit, 120 drawing unit,
130 audio processing unit, 131 sound reproduction unit, 132 delay control unit,
133 reproduction frequency setting unit, 134 delay counter control unit, 135 sound data control unit,
160 operation unit, 170 storage unit, 172 main storage unit, 174 drawing buffer,
176 sound buffer, 178 delay counter, 180 information storage medium,
190 Display unit, 192 Sound output unit, 196 Communication unit

Claims (10)

A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
The computer is caused to function as an audio processing unit that sequentially reads the sound data to be reproduced according to the elapsed time from the start time into the storage unit, and sequentially reproduces the read sound data,
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is not delayed, the reference frequency is set as the reproduction frequency, and the sound data is reproduced according to the reference frequency.
When it is determined that the moving image is delayed, a process of setting a correction frequency lower than the reference frequency as a reproduction frequency based on the value of the delay counter and reproducing sound data using the correction frequency; Process to update the value of the delay counter,
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
A program for performing a process of subtracting a value corresponding to a reproduction time of sound data by the correction frequency from a value of a delay counter. In claim 1,
The audio processing unit is
When it is determined that the moving image is delayed, the difference between the reproduction frequency being reproduced and the correction frequency is lower than the reference frequency based on the value of the delay counter so that the difference is limited to a predetermined band. A program characterized by obtaining a correction frequency and setting the obtained correction frequency as a reproduction frequency. A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
The computer is caused to function as an audio processing unit that sequentially reads the sound data to be reproduced according to the elapsed time from the start time into the storage unit, and sequentially reproduces the read sound data,
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is delayed, a process of inserting dummy sound data as a reproduction target in the storage unit and a process of updating the value of the delay counter are performed.
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
A program for performing a process of subtracting a value corresponding to an insertion amount of the dummy sound data from a value of a delay counter. In claim 3,
The audio processing unit is
A program for obtaining a size of the dummy sound data according to a value of the delay counter. In claim 3 or 4,
The audio processing unit is
Detecting a reproduction position where the amplitude of the reproduced sound is lower than a predetermined value in a sound data string composed of a plurality of sound data read into the storage unit, and inserting the dummy sound data into the detected reproduction position; A featured program. In any one of Claims 3-5,
The audio processing unit is
A program characterized by detecting a reproduction position having a low amplitude change rate of reproduced sound in a sound data string composed of a plurality of sound data read into a storage unit and inserting the dummy sound data into the detected reproduction position. In any one of Claims 3-6,
The audio processing unit is
In a sound data string composed of a plurality of sound data read into the storage unit, a reproduction position where the amount of frequency components contained in the reproduction sound is larger than a predetermined amount is detected, and the dummy sound data is inserted into the detected reproduction position. A program characterized by that.   A computer-readable information storage medium, wherein the program according to any one of claims 1 to 7 is stored. A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
An audio processing unit that sequentially reads sound data to be played in accordance with an elapsed time from the start time into a storage unit, and performs a process of sequentially playing the read sound data;
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is not delayed, the reference frequency is set as the reproduction frequency, and the sound data is reproduced according to the reference frequency.
When it is determined that the moving image is delayed, a process of setting a correction frequency lower than the reference frequency as a reproduction frequency based on the value of the delay counter and reproducing sound data using the correction frequency; Process to update the value of the delay counter,
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
A game apparatus that performs a process of subtracting a value corresponding to a reproduction time of sound data by the correction frequency from a value of a delay counter. A moving image processing unit that sequentially generates images constituting a moving image from the start time, and sequentially outputs the generated images to the display unit;
An audio processing unit that sequentially reads sound data to be played in accordance with an elapsed time from the start time into a storage unit, and performs a process of sequentially playing the read sound data;
The moving image processing unit
Determine whether the images that make up the video were generated within the scheduled time, and update the value of the delay counter based on the determination result,
The audio processing unit is
Based on the value of the delay counter, determine whether the video is delayed with respect to the sound,
When it is determined that the moving image is delayed, a process of inserting dummy sound data as a reproduction target in the storage unit and a process of updating the value of the delay counter are performed.
The moving image processing unit
When it is determined that the image constituting the video is not generated within the scheduled time, a value corresponding to the excess time exceeding the scheduled time is added to the value of the delay counter,
The audio processing unit is
A game apparatus that performs a process of subtracting a value corresponding to an insertion amount of the dummy sound data from a value of a delay counter.

Images