KR20060103232A - Music data producing system and computer readable recording medium storing a music data producing program - Google Patents

Abstract

By freely generating original 3D music contents by a terminal user, this music data generation system is composed of a server SV connected to be communicatively connected to the Internet and a client terminal CL such as a mobile phone terminal. In the client terminal CL, the user sequentially designates and records virtual sound image positions using a 3D sequence input screen representing a 3D sound space while reproducing desired original music data transmitted from the server SV ( C6). The recorded series of sound image positions are transmitted to the server SV as a position sequence, and the server SV adds the position sequence to the original music data to generate three-dimensional (3D) acoustic characterization music data (3D music data). Then, this is distributed to the client terminal (S5). In the client terminal CL, it is possible to generate a music playing space in which the sound image moves in correspondence with the position sequence of the user's original in accordance with the distributed 3D sound data (C7).

3D sound, music data, sound image, 3D sound sequence

Description

Computer-readable recording medium storing music data generating system and music data generating program {MUSIC DATA PRODUCING SYSTEM AND COMPUTER READABLE RECORDING MEDIUM STORING A MUSIC DATA PRODUCING PROGRAM}

1 is a schematic configuration diagram of an entire music data generation system according to an embodiment of the present invention.

Fig. 2 is a diagram showing the overall processing operation in the music data generation system according to the embodiment of the present invention.

FIG. 3 is a diagram showing a structural example of three-dimensional (3D) acoustic characteristic-providing music data generated in a music data generating system according to an embodiment of the present invention. FIG.

4 is a flow chart showing a client terminal side music data generation process according to an embodiment of the present invention.

Fig. 5 is a flowchart showing sequence input processing during client terminal side music data generation processing according to an embodiment of the present invention.

6 illustrates an example of key arrangement of a client terminal.

7 is a diagram illustrating an example of a sequence input screen (3D screen) in a client terminal.

8 is a flowchart showing a server-side music data generating process according to an embodiment of the present invention.

<Explanation of symbols for main parts of drawing>

CL; CLa.CLb A client terminal (electronic music device) that can communicate with a server (SV) consisting of a client terminal or a PC.

Gr: coordinate grid

Po: listening position, i.e. origin coordinate position

Pg: Image indicating sound image position (also called sound source position) or sound image position

Dg: Current position notation indicating the latest coordinate value of sound position Pg

[Non-Patent Document 1] http://ascii24.com/news/i/tech/article/2004/12/09/653023-000. html

[Non-Patent Document 2] http://www.yamaha.co.jp/news/2005/05011101.htm1

The present invention relates to a music data generation system for generating a three-dimensional (3D) sound characteristic-provided music data desired by a user who uses a client terminal by communicatively connecting a server and a client terminal through a communication network such as the Internet. will be.

Background Art [0002] Conventionally, in a mobile phone terminal, there is a change in arrangement of music for incoming melody provided from a content delivery site via the Internet or the like. In recent years, as shown in Non-Patent Documents 1 and 2, a sound is moved to a mobile phone terminal employing a 3D positional audio technology including an effect such as a sound source is approaching, or such a mobile phone terminal. There is a service that distributes 3D incoming melodies that can sense stereo sound and sound spread.

However, most of the incoming melody delivery services use a so-called " template type " song conversion method in which a music (original file) or changeable parameter to be changed is selected from among those prepared in advance. That is, the user can only select the variation when converting the original file into the predetermined pattern, and cannot obtain the original music content of the user's original.

Moreover, as an apparatus which moves the position of a sound image, there exist something like patent document 1, 2, for example. In Patent Literature 1, as the movement state of the stereotactic position is instructed by a game program or an operator such as a trackball or a joystick, the sound position of the musical sound from the left and right two channels can be controlled to enrich the realism of the image movement. have. Patent Literature 2 also reliably moves a sound image on a predetermined sound image movement path close to the operation position based on the operation of an operator such as a track ball.

In view of such circumstances, it is an object of the present invention to provide a music data generation system capable of freely generating three-dimensional sound characteristic-providing music contents of a user's original person based on a positioning operation by a user of a client terminal.

According to a main feature of the present invention, a system comprising a server SV and a client terminal CL (CLa, CLb) communicatively connected via a communication network CN, wherein the server SV is configured to store a plurality of pieces of music data. Original music storing means DB for accumulating, original music reading means S4, S12 for reading music data from the original music storing means, and sequence receiving means S5 for receiving position sequence data from the client terminal CL; Sl1) and three-dimensional music generation means S5 for generating position sequence-provided music data (3D music data Md) reproducible as three-dimensional acoustic characteristics by adding position sequence data to the music data read by the music reading means. S14 to S18 and three-dimensional music transmission means S5 and S19 for transmitting the positional sequence assignment music data Md to the client terminal CL, and the client terminal CL is a user. Sequence input means (Cl9) C34 to C41 for sequentially inputting the positional coordinates Pg of the virtual sound image based on the operation, and sequence recording means (C19: C39) for recording the inputted positional coordinates as position sequence data. , C40, sequence transmitting means C20 for transmitting the recorded position sequence data to the server SV, and three-dimensional music receiving means C24 for receiving the position sequence providing music data Md from the server SV. And three-dimensional music storage means (2, 4; C25) for reproducibly storing the received positional sequence-assigned music data (Md) and the stored positional sequenced music data (Md) as three-dimensional acoustic characteristics. A music data generating system [claim 1] provided with a three-dimensional music reproducing means C25 '. In addition, writing in parentheses indicates reference symbols and the like of the corresponding embodiments, and the same also applies to the following.

In the music data generation system according to the present invention, the server SV further transmits the music data read by the music reading means S4 to the client terminal CL in response to a request from the client terminal CL. The client terminal CL further comprises a music transmitting means S4, and the client terminal CL further includes a request transmitting means C5 for transmitting a request for the music data to the server SV, and the server SV in accordance with the transmitted request. Original music receiving means (C18) for receiving music data transmitted from the memory; and original music storage means (2, 4; C18) for storing the received music data in a reproducible manner; The input of the positional coordinates Pg can be configured as described in Claim 2 performed in parallel with the reproduction of the music data stored in the original music storage means 2 (C33 to C41).

In the music data generation system according to the present invention, the sequence input means C19 instructs the first input means C38 for continuously inputting coordinate positions sequentially instructed by the user operation, and / or by the user operation. It can be comprised as [claim 3] provided with the 2nd input means C40 which generate | generates and inputs the coordinate position sequence which concerns on the movement mode to become.

According to another feature of the invention, a request for music data is sent to a server (SV) from a computer (client terminal: CL; CLa, CLb) communicatively connected to the server SV via a communication network CN. The request data transmission step C3 to transmit, the original music reception step Cl8 which receives the music data transmitted from the server SV according to the transmitted request, and the music data received by the original music reception step can be reproduced. By reproducing the original music storage step C18 and the music data stored in the original music storage step, and continuously inputting coordinate positions sequentially indicated by user operation (C38), and / or, By inputting the coordinate position according to the movement mode instructed by the user operation (C40), the sequence input steps C34 to C41 for sequentially inputting the position coordinates Pg of the virtual sound image, and the sequence entry. In the sequence recording steps C39 and C40 for recording the series of position coordinates input in the step as the position sequence data, the sequence transmission step C20 for transmitting the recorded position sequence data to the server SV, and the request transmission step. Three-dimensional music reception step of receiving position sequence-provided music data (3D music data Md) reproducible as a three-dimensional acoustic characteristic, generated by adding position sequence data to music data corresponding to the transmitted request, from the server SV. (C24), the three-dimensional music storage step C25 for reproducibly storing the position sequence-assigned music data Md received in the three-dimensional music reception step, and the position sequence-provided music data stored in the three-dimensional music storage step Music data generation program for executing the procedure consisting of three-dimensional music reproduction step C25 'for reproducing (Md) as a three-dimensional acoustic characteristic The Save the ram computer-readable medium of claim [4] is provided.

[Summary of system]

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the structural example of the whole three-dimensional (3D) acoustic characteristic provision music data generation system by one Example of this invention. This music data generating system includes a server SV serving as a 3D music providing site for providing three-dimensional (3D) acoustic characteristic-providing music data (hereinafter referred to as "3D music data"). A plurality of client terminals CL capable of communicating with the server SV via a wide area communication network CN such as the Internet; CLa and CLb (symbol "CL" generally indicates a client terminal). In this example, the client terminal (user terminal) CL has a communication function with the server SV, for example, and music data. An electronic musical apparatus such as a mobile phone terminal CLa having a processing function, a personal computer (PC) terminal CLb, etc. The client terminal CL is preferably a mobile phone terminal CLa. 3D music data generated in this system can be enjoyed in any use environment as an incoming melody or the like.

In the large-capacity external storage device of the server SV, a plurality of databases DB for providing 3D music data to the user of the client terminal CL are constructed. For example, various applications for client terminals are stored in the application database including a three-dimensional (3D) acoustic characterization music data generation program (hereinafter referred to as an "application") that can be executed in the client terminal CL. In the music database, a large number of existing music data (referred to as "original music data"), which is the basis for generating 3D music data, are stored, and each original music data is composed of music information for one to a plurality of music channel channels. do. In addition, a large number of 3D templates are stored in the template database for assigning a predetermined position sequence to generate 3D music data.

Each mobile phone terminal CL is connected to the server SV through the wireless base station BS and the communication network CN in the case of the mobile phone terminal CLa, and in the case of the PC terminal CLb. It is connected to the server SV via the communication network CN. FIG. 1 (2) is a block diagram for demonstrating the hardware configuration of each apparatus which comprises this music data generation system, and is shown based on the client terminal CL.

In the following description, the client terminal CL is mainly described as being used for the mobile telephone terminal CLa (however, the description of the ordinary telephone function is omitted). As described above, the communication function and You may use the general-purpose information processing apparatus CLb, such as a PC with a music sound data processing function. The client terminal CL includes a central processing unit (CPU) 1, a random access memory (RAM) unit 2, a read-only memory (ROM) unit 3, an auxiliary storage unit 4, and an input device 5. ), A sound source 6, a display 7, a communication interface (communication I / F) 8, and the like, and these devices 1 to 8 are connected to each other via a bus 9.

The CPU 1, the RAM unit 2, and the ROM unit 3 constitute a main body processing unit. The ROM unit 3 stores firmware, various programs, and the like for collective control of the entire apparatus, and the CPU 1 collectively controls the entire apparatus in accordance with the program stored in the ROM unit 3. The RAM unit 2 is used as a work area for the OS or various programs in various processes. For example, when creating position sequence data, the RAM unit 2 temporarily stores various pieces of data such as original music data, position events, or 3D templates. Various data storage (temporary memory) areas are secured for the temporary storage. Further, in the case of the mobile phone terminal CLa, a nonvolatile semiconductor memory such as a flash memory can be used for the RAM section 2 to the ROM section 3.

The auxiliary storage unit (external storage unit) 4 is used to store (store) various data including 3D music data, a net session control program, an application, and the like. In the case of the PC terminal CLb, the hard disk drive device HDD is used, but in the case of the mobile phone terminal CLa, no HDD is provided, and a nonvolatile semiconductor memory such as a flash memory is used. In this case, the non-volatile semiconductor memory such as the flash memory used for the RAM section 2 or the ROM section 3 has the function of the auxiliary storage section 4, and all the data types are stored in the semiconductor memory. can do.

The input device 5 consists of operators, such as various input keys, and the operation detection circuit which detects the operation content of these operators, and introduces into the main body process part. In the case of the cellular phone terminal CLa, the key operator includes, for example, a cursor key, a function key, an on-hook key, an off-hook key, a numeric and symbol key for a telephone, and a PC terminal CLb. The mouse or keyboard is used as the operator. The display 7 consists of a display which displays various screens, and a display circuit which controls a screen display according to the instruction from CPU1, and displays the operation state of the client terminal CL, or displays for user operation. Assistance

The sound source unit 6 and the sound system 10 connected to the sound source unit 6 constitute a music sound reproducing unit. That is, the sound source unit 6 includes a sound source circuit TG, a DSP (digital signal processor), and the like, and generates a sound signal corresponding to the original music data or the 3D music data, and in particular, the DSP is applied to the 3D music data. Therefore, it has a function of performing 3D sound field control. The sound system 10 includes a D / A converter, an amplifier, and a stereo twin speaker, and generates sound based on the sound signal from the sound source unit 6, and in particular, the sound image of the sound by reproducing 3D music data. By moving along this predetermined path, a stereophonic sound space with sound diffusion can be realized.

The communication I / F 8 is at least a communication interface such as a modem or ether, which can be directly or indirectly connected to the Internet, and may be an interface for performing wireless communication without being limited to a wire. The client terminal CL can communicate with the server SV by means of the communication I / F 8 via the network CN including the Internet, and the original music data and the 3D music data with respect to the server SV. Can be requested and received. In this case, the cellular phone terminal CLa communicates with the server SV via the base station BS in the wireless data communication from the communication I / F 8.

The hardware configuration of the server SV is substantially the same as that of the client terminal CL described in FIG. 1 (2), but the sound system 10 may not be provided. The server SV includes a large capacity external storage device (HDD, etc.). As described above, a plurality of databases (DB) in which applications, original music data, 3D templates, and the like are accumulated are constructed in the external storage device, and user management data files for managing each client terminal user are also stored. .

[Summary of movement]

Fig. 2 is a diagram showing the overall processing operation in the music data generation system according to the embodiment of the present invention together with the transition of the display screen of the client terminal. The processing operation of this system is divided into a first step of downloading an application, that is, a 3D acoustic characterization music data generation program, to the client terminal CL, and a second step of executing the downloaded application on the client terminal CL.

In the first step, the user first requests an application from the client terminal CL to the server SV (step Cl). At this time, the model of the client terminal CL, the user ID, and the like are notified together with the request of the application. If the server SV is a new user, the user is registered in the user management data file. Otherwise, the server SV checks the model, the user ID, the user address, and the like against the registration data of the corresponding terminal user in the user management data file, and then the client terminal is executed. It delivers to CL (step S1). In the client terminal CL, the received application is stored in the storage means (the auxiliary storage unit 4 or the ROM unit 3) (step C2).

The second step is started by activating the application stored in the storage means by the client terminal CL, and shows the screen transitions C3 to C7 as shown. First, when the application is started, a startup screen (application top screen) containing items necessary for selecting a piece of music such as "Recently Popular Songs", "Search by Song Name", and "Search by Artist" is displayed on the display 7 (hereinafter, referred to as "Top Songs"). Symbol "7" is also used for reference of the display) (step C3). When the user performs an operation of instructing a desired item using the startup screen, the client terminal CL requests the server SV for a music list corresponding to the indicated item. The server SV confirms the model, the user ID, and the like transmitted with the request of the music list, and then transmits the music list to the client terminal CL (step S2).

In response to the reception of the music list from the server SV, the client terminal CL displays on the display 7 a music selection screen on which the music list is loaded (step C4). When the user performs an operation of selecting a desired piece of music using the piece of music selection screen, the client terminal CL notifies the server SV that the piece of music has been selected, and the server SV performs a process capable of responding to the selected piece of music. The processing item data shown is returned to the client terminal CL (step S3).

The client terminal CL displays the process selection screen on which the processing item of the selected music piece is loaded on the display 7 accordingly (step C5). This processing item includes a first processing method for generating 3D music based on a sequence created by a terminal user's manual input, and a second processing method for generating 3D music based on the 3D template instructed by the user's selection operation. As the second processing method, names of a plurality of 3D templates prepared in advance in the server SV are enumerated by template information included in the processing item data, and a desired 3D template is selected by the user's template selection operation. Can be indicated.

On the other hand, when the user performs an operation of selecting a desired processing method, the client terminal CL notifies the server SV of the processing method, and the server SV performs processing corresponding to the processing method. For example, when the first processing method is selected, the original piece of music data corresponding to the piece of music selected on the piece of music selection screen is read from the piece of music database DB and transmitted to the client terminal CL together with the piece of music ID. In addition, when the second processing method is selected, in addition to the original music data and music ID, the 3D template selected in response to the template selection operation is transmitted.

When the first processing method is selected, when the client terminal CL receives the original music data from the server SV, the client terminal CL stores the original music data in the original music storage area of the RAM unit 2 and inputs the sequence. A screen (also called a "3D screen") is displayed on the display 7 (step C5). The terminal user can create position sequence data consisting of a series of sound image position event data strings by sequentially instructing the sound image positions of the music pieces by manual input in accordance with reproduction of the original music data using the sequence input screen. The client terminal CL transmits the position sequence data created corresponding to the original piece of music data to the server SV together with the piece of music ID of the original piece of music data.

If the second processing method is selected, the client terminal CL stores the original music data and the 3D template from the server SV in the original music and template storage area of the RAM unit 2, and the template confirmation screen. (Also called a "3D screen") is displayed on the display 7 (step C5). On the template confirmation screen, the sound image position based on the 3D template is displayed in accordance with the reproduction of the original music data. Therefore, when the terminal user confirms the displayed sound movement mode and performs an OK operation, the client terminal CL determines the 3D template. The request signal with respect to the music ID is transmitted to the server SV.

If the selected processing method or 3D template is performed by NG operation other than the preference of the terminal user, the processing returns to the process selection step C5 to reselect the processing method or 3D template. In addition, when the original music itself is not returned to the terminal user's preference, the "back" operation is performed, and the music composition selection step C4 can be returned, and the server SV is notified in any case.

When the first processing method is selected, the server SV adds the position sequence data to the original music data designated by the music ID, and converts the original music data from the original music data into 3D music data, that is, three-dimensional (3D) sound characteristic-provided music data. 3D transform is performed, and when the second processing method is selected, the 3D transform is performed according to the selected 3D template by receiving the required signal, and the 3D music data (MD: FIG. 3) obtained by the 3D transform is performed. It distributes to the client terminal CL (step S5). As the processing method in the case where the second processing method is selected, the client terminal (in step S4) is immediately executed in step S4 without passing the original music data, the transmission of the 3D template (S4) or the reception of the required signal (S5) as described above. It may be configured to determine that 3D conversion should be performed according to the 3D template selected on the CL) side.

When the client terminal CL receives the 3D music data from the server SV, the client terminal CL displays the download notification screen (3D screen) on the display 7, and the user confirms movement of the sound image, and the 3D music tune for the download notification screen. When the operation for determining the acquisition of data is performed, the 3D music data is stored in the storage means 2 and 4 (step C7). Thereafter, the client terminal CL can reproduce the 3D music data stored in the storage means, and enjoy a music playing space in which the sound image of the music moves in correspondence with the position sequence data created by the user or the 3D template selected by the user, In addition, the display 7 can display a 3D music playback screen (3D screen) which moves and displays the sound image position of the music in accordance with the music playback.

In addition, when 3D music data is desired for another music, it returns to the music selection screen C3 by operation of the "return" ("F3") of the input device 5, and the above-mentioned process operations C3-. C7 and S2 to S5 can be repeated, whereby the user of the client terminal CL can acquire new 3D music data based on the new original music data.

The music data generation system according to an embodiment of the present invention is characterized in that, in particular, position sequence data required for generation of 3D music data is created on the client terminal CL side using the sequence input screen C6. That is, the gist of the system is as follows: In the client terminal CL, the user reproduces the desired original music data transmitted from the server SV, while using a 3D sequence screen representing the 3D sound space to be virtual. The sound image positions are sequentially designated and recorded (C6). The recorded series of sound image positions is transmitted to the server SV as a position sequence, and the server SV generates three-dimensional (3D) acoustic characterizing music data (3D music data) by adding the position sequence to the original music data. Then, this is distributed to the client terminal (S5). In the client terminal CL, it is possible to generate a music playing space in which the sound image moves in correspondence with the position sequence of the user's original in accordance with the distributed 3D sound data (C7).

[3D Music Data Structure]

3 shows an example of the configuration of 3D music data obtained by 3D conversion on a server. 3D music data Md consists of header information Hd, music track information Mt, and position sequence track information St. As shown in FIG. The header information Hd indicates how many kinds of tracks are in the 3D music data Md, and what kind of channels each track has, the track information of each track and each channel, and the like. (Md) is described in the header section at the beginning. The music track information Mt is music information described in the music track corresponding to the original music data, and, like the original music data, consists of any number of channels corresponding to parts such as melody and accompaniment. The music information of each channel is composed of event sequence data representing musical notes in an MIDI format or an event format similar to the MIDI format, or audio signal data representing musical waveforms in an encoding format such as ADPCM.

In addition, the position sequence track information St described in the position sequence track corresponds to the music track information Mt of a predetermined channel, and has a movement time (Minimum) of the sound image position of each music sound based on the music track information Mt. Sound image position definition data that is sequentially defined by the position event for each? T). The position sequence track information St can represent one position event by setting the coordinate positions Pg (r, θ, φ) and the time information as a set. The time information includes the start of the performance of the music track information Mt. The absolute time (time data) t and the travel time Δt are used. For example, for one position event, the position event time (t) is 2 bytes, the movement time (Δt) is 2 bytes, and the coordinate position Pg (r, θ, φ) after the movement is 3 bytes (polar coordinate value). 7 bytes of data can be obtained.

In addition, although the content of the position event sequence input in the sequence input screen C6 is basically the same format as the above-mentioned position sequence track information St, a coordinate system is a drawing coordinate system (Euclide system, Cartesian coordinate system) or a pole. Any of the coordinate systems may be used. For example, for one position event, the position event time t = 2 bytes, the movement time Δt = 2 bytes, and the coordinate position Pg (x, y, z) = 3 bytes after the movement [Cartesian coordinate value] or Pg (r , θ, φ) = 3 bytes (polar coordinate value) can be 7 bytes of data.

[Process at Client Terminal]

4 is a flowchart showing 3D music data generation processing executed in a client terminal according to an embodiment of the present invention. In the client terminal CL, when the applications Sl and C2 (FIG. 2) downloaded from the server SV are started, the CPU 1 initializes and first displays the startup (top) screen on the display 7. The desired music list requested by this screen is then transmitted from the server SV (S2: Fig. 2), so that the music list is acquired (step C11). Accordingly, key functions and menus are assigned for music selection (step Cl2), the acquired music list is displayed on the music selection screen (step C13), and the user operation on the music selection screen is waited (steps Cl4 to NO). .

Here, when a desired piece of music is selected by user operation (C14? YES), the information of the selected piece of music is notified to the server SV, and processing items corresponding to the selected piece of music are sent from the server SV (S3: According to Fig. 2, key functions and menus are assigned for process selection (step C15), and these process items are displayed on the process selection screen (step C16), and the user operation on the process selection screen is awaited. (Step C17).

When the user himself selects to perform the input processing of the position sequence for the selected music by manual input by user operation on the process selection screen (Cl7 → YES), the notification is notified to the server SV. Since the original music data indicating the contents of the selected music is transmitted from the server SV (S4: FIG. 2), the original music data is received and stored in the original music storage area of the RAM unit 2 (step). C18). Next, the client terminal CL performs a sequence input process (FIG. 5), and creates position sequence data with respect to the received original music data (step C19). Then, the created position sequence data is transmitted to the server SV together with the music ID of the music data (step C20).

On the other hand, when a predetermined 3D template is selected from 3D templates prepared in advance in the server SV by user operation on the process selection screen, and the purpose of 3D converting the selected piece of music is selected according to this 3D template (C17). NO), the effect is notified to the server SV, and the selected 3D template is transmitted from the server SV together with the original music data indicating the content of the selected music (S4: Fig. 2). Received (step C21) and reproduced on the template confirmation screen of the sound reproduction part 6-10 and the display 7 (step C22). And when the user observes the movement display of the sound image position by this template confirmation screen, and the 3D template is satisfied, it notifies the server SV that it was OK by user operation (step C23). .

That is, in the server SV, the sound image position of the piece of music is rotated in a circular or elliptic shape, for example, with the listening position as the origin, gradually approaching and returning to the origin from a distance while being rotated. Sound image movement modes such as ..., etc. reciprocating in a pendulum shape, a curved shape or a straight line shape in a left-right, front-back or oblique manner are prepared as a 3D template, and the client terminal CL has these sound image movement modes in accordance with the template information. It is displayed as a template option on the process selection screen. Therefore, when one of the templates is selected by the user operation, the template confirmation screen is displayed on the display 7, which is selected by the user operation in accordance with the reproduction of the selected piece of music (the sound reproducing unit 6-10). According to the position sequence data of the template, the sound image position of the piece of music is displayed and moved into the template confirmation screen, so that it is possible to confirm whether or not the 3D template matched the user's desire.

In response to the notification in steps C20 and C23, the server SV creates the corresponding 3D music data Md and delivers it as a 3D-converted file (S5: FIG. 2). After acquiring the 3D music data (3D converted file) Md (step C24), the 3D music data is displayed on the download notification screen (3D screen) and the contents of the image movement are checked, and then the storage means (2, 4). Are stored so that they can be reproduced (step C25).

Further, with respect to the 3D music data Md stored in the storage means, when necessary, the 3D music data Md is read out by instructing selection and reproduction of the 3D music on the 3D music playback screen, whereby the sound reproduction unit 6 Through -10), the music can be reproduced so that the sound image position moves in accordance with the sequence input processing or the position sequence designated by the selected 3D template, and based on the position sequence track information St of the 3D music data Md, In accordance with the music playback, the sound image position of the music can be animatedly displayed on the 3D music playback screen (3D screen) of the display 7 (step C25 ').

After the 3D music data preservation process (C25), whether or not the 3D music data generation process is finished or not (step C26), when the user is instructed not to finish the process (C26-> NO), Returning to the original music selection step C12, the above-described operations of steps C12 to C25 are repeated for the newly selected original music. Then, when instructing to end (C26? YES), all data remaining in the various data storage areas of the RAM unit 2 is erased, and the 3D music data generation process is terminated.

[Example of Sequence Input Processing]

FIG. 5 is a flowchart showing a specific example of the sequence input process Cl9 in the sequence input process according to one embodiment of the present invention, that is, the 3D music data generation process on the client terminal CL side in FIG. In this sequence input processing, similarly to steps C12 and C15 in Fig. 4, a key sequence and a menu are assigned for sequence input processing, and a sequence input screen is displayed. The user operates a predetermined key using the sequence input screen. By doing this, the desired position sequence data can be generated by the magnetic force input.

Fig. 6 shows a typical arrangement example of a key used as an input device and an example of function assignment to numeric keys in a client terminal such as a mobile phone terminal. As shown in the key arrangement example of FIG. 6 (l), as the operator of the input device 5, the "Fl" to "F3" keys, the "decision" key, which are used as function keys for displaying menus, etc., Up and down left and right cursor keys, &quot; on &quot; and &quot; off &quot; hook keys, &quot; 1 &quot; to &quot; 9 &quot;, &quot; * &quot;, &quot; 0 &quot;, &quot;# &quot; For example, when the original music list is displayed on the music selection screen C4 or when the 3D music selection or playback is instructed on the 3D music playback screen C25 ', the menu is displayed using the F1 key, and the menu is moved with the cursor keys. A predetermined function is assigned to each key by selecting an item and performing a deciding operation with the "decision" key.

During the sequence input processing, retry is performed on the "Fl" key, mode changes are made between "Mode 1" and "Mode 3" on the "F2" key, and playback of the original music is reproduced on the "F3" key. The play / stop function which starts / stops sequentially at each time, and a predetermined function are assigned to each key by changing the viewpoint position to the "*" key. In the mode of sequence input processing, "mode 1" which can sequentially input (step) position events in the Euclidean system (orthogonal coordinates), and "mode 2" which can input position events sequentially in polar coordinates And "mode 3" (coordinate system is orthogonal or polar) which inputs all positional events by arbitrary automatic control processing (jumping) indicating a jump or a movement path, and the numeric keys (Fig. As shown in 2), a predetermined function is assigned to each mode.

7 shows an example of a sequence input screen displayed on the client terminal display. The sequence input screen displayed on the display 7 at the time of the sequence input processing in the client terminal CL, listens to the music according to the coordinate grid in which the listening position, that is, the headphone position Po of the sound system 10 as the origin. It simulates space in three dimensions. In addition, FIG. 7 is an exemplary representation of a listening space for the sake of understanding. For example, the mesh of the grid is simplified, and the coordinate values [unit: m (meter)] of the origin and vertices are also actually displayed. (But may be marked as necessary). In addition, the size of the listening space and the mesh size are set on an arbitrary scale.

By reproducing the original music, the user moves the figure (also referred to simply as the "sound") Pg, which represents the sound image of the musical sound, into the listening space of the sequence input screen by key operation, thereby specifying the sound position in real time and By performing control, it is possible to determine the position event of sound image sequentially and to record in real time. At this time, the trajectories (x, y, z) of the sound images Pg sequentially set on the sequence input screen are faded out for a predetermined time as shown in FIG. 7, for example, a schedule up to the latest set position. The image Pg (x, y, z) over time is displayed in a thicker form at the new image position and lighter at an old image position. The fade-out display is also performed on the download notification screen and the 3D music reproduction screens C7, C25, and C25 'described above.

Briefly explaining the relationship between the key operation and the position event. For example, in Mode 1, each time the "2" key is operated, the unit moves in the forward (+ y) direction by l unit distance and operates the "4" key. Each time, it moves by 1 unit in the left (-x) direction. In this way, according to the operation of the "2", "4", "6", and "8" keys, the sound image position can be specified in step-by-step, and this type of position input method is called "step" type. In addition, by operating the "1", "3", "5", "7", and "9" keys, it is possible to jump from the current position specified at that time to a predetermined position via a predetermined movement path. In addition, this path can be separately designated so as to have fluctuations in the movement trajectory such as "linear shape", "zigzag shape" and "random (irregular curved shape)". This type of position input method is called a "jump" type, and a predetermined position event is automatically created by an automatic control process according to an instruction by a key operation. In addition, the "0" key is provided to change the coordinate system of the key function, and the jump point or the movement direction by the "1" to "3", "7" to "9" key operation for each "0" key operation. Is switched back and forth (+ y.-y) and up and down (+ z, -z).

In Mode 2 after the "F2" key is operated once, as shown in the right side of FIG. 7, the position event is input in the polar coordinate format. For example, each time the "2" key is operated, the elevation angle φ is shifted by one unit angle in the + direction (+ z side), and each time the "4" key is operated, the azimuth angle θ is moved in the-direction (xy plane). Right)) and shift the distance by 1 unit and move the distance r from the origin by 1 unit in the-direction (origin side) every time the "5" key is operated. The position can be input in the "step" type by the "6" and "8" keys. In addition, when the &quot; l &quot;, &quot; 3 &quot;, &quot; 7 &quot; and &quot; 9 &quot; keys are operated, it is input in the &quot; jump &quot; In addition, when the "O" key is operated in the mode 1 and the front, rear, and up and down are switched, this switching state is also maintained for the operation of the "l", "3", "7", and "8" keys of the mode 2. For example, in Mode 1, operate the "O" key operation to switch from "before and after" to "up and down", and then enter Mode 2 and operate the "1" key to jump to the vertex on the left side of the up (+ z) direction. do.

In addition, operation of the "F2" key enters mode 3. This mode is also called an automatic control mode, and jumps in the same manner as in Mode 1 by the operation of the "1", "3", "7", and "9" keys. As described above, all position events are input in a &quot; jump &quot; type that reaches a predetermined position via a predetermined path. For example, when the "1" key is operated, it jumps to the coordinate position a of one of predetermined coordinate positions (vertex or random point) a-d previously determined through a movement path of a predetermined shape from the current position. When the "2" key is operated, the jump jumps to the position (x, y, -z) which is symmetrical with the current position (x, y, z) with respect to the xy plane. In addition, when the "4" key is operated, the plane parallel to the x-y plane is rotated by 1 about the z axis to return to the current position (x, y, z). , 0, 0) and any plane including the current position (x, y, z) are rotated once about a predetermined coordinate position, and then returned to the current position (x, y, z).

By the way, in the 1st step of the sequence input process of FIG. 5, CPU1 sets the initial position of sound image Pg according to a user's setting operation with respect to a sequence input screen (FIG. 7: 3D screen) (step C31). The sound image position Pg (x, y, z) set at this time is displayed as an image in the 3D listening space of a screen, and is displayed as a coordinate value in the current position notation Dg of an edge part, and a user can confirm. Further, in the initial position setting step C31, after operating the "*" key in the mode 1 or the mode 2, the viewpoint position is changed by the operation of the numeric keys (2, 4, 6, 8.0), and the determination is made with the "decision" key. You can change the position of the starting point of the coordinate grid.

Next, processing starts with the operation of the "F3" key, resets the moving time (Δt) of the sound image position event to "0", and sets the set sound image initial position Pg (x, y, z) and time information (absolute time). t = "0" and movement time [Delta] t = "0") are recorded in the position event storage area of the RAM unit 2 as the first position event (step C32) and from the original music storage area of the RAM unit 2; The original music data is read out and playback is started through the sound output unit 6-10 (step C33). In addition, playback of the original music can be paused by operating the "F3" key later, and playback of the original music can be resumed by further operating the "F3" key. On the other hand, while the reproduction of the original music is stopped, the counting operation is stopped and the counting operation is started again by resuming the reproduction.

Subsequently, it is determined whether or not a user input operation has been performed on the position event (step C34), and when there is no position input operation (C34 → NO), when there is no reproduction of the original music (step C35 → NO), an input operation determination is performed. Return to step C34. Then, when there is a position input operation (C34-> YES), the sound is elapsed from the time of the previous position input operation (the beginning of the sequence input processing is the start operation time by the "F3" key) to the time of the position input operation. After updating the movement time (DELTA) t of a position event (step C36), the type determination of the input operation regarding a position event is performed (step C37).

Here, when the type of the input operation is "step" (sequential) type position input (C37? YES), the current position Pg (x, y, z) of the sound image is updated according to the amount of movement or displacement based on the input operation (step C38). ), A position event consisting of the current image position Pg (x, y, z) after updating and its time information (absolute time t from the start of reproduction of the original music data and the moving time Δt) is generated, and this It records in the position event storage area of the section 2 (step C39).

On the other hand, when the type of the input operation is a "jump" (automatic control) type position input (C37? NO), the path in which the sound image moves is automatically controlled in accordance with the movement contents designated by the input operation, and the corresponding Generates a series of position events consisting of a plurality of representative coordinate positions and their time information (absolute time and relative movement time indicating the point of time passing each coordinate position calculated from the elapsed time Δt), and this is the position of the RAM section 2. The data is sequentially recorded in the event storage area (step C40). In addition, the moving speed at the time of moving a sound image position by a "jump" type (automatic control) type | mold position input may be a constant speed, an equivalent acceleration (acceleration direction, a deceleration direction), and acceleration and deceleration in the place of equivalent acceleration speed. Does not have to be the same. In addition, as described above, the movement trajectory may have a fluctuation such as a zigzag path or a random fluctuation.

When a new position event is recorded (C39, C40), as shown in the screen of Fig. 7, the sound image up to a certain time from the latest sound position Pg is faded out as a picture in the 3D listening space. At the same time, the coordinate value Pg (x, y, z) of the latest sound image position is displayed by the current position notation Dg at the end of the screen. If there is a recording process of the position event based on such a position input operation (C39, C40), the movement time Δt is immediately reset to a value of "0" (step C41), and the position input operation detection step C34 is returned. While the reproduction of the original music data is not finished (C35? NO), the above operations of steps C34 to C41 are repeated, and the position sequence data is recorded by recording the position event of one piece of the original music data in the position event storage area. Can be completed.

Then, when there is no position input operation (C34-> NO) and when reproduction of the original music data is finished (C35-> YES), it is determined whether or not an operation for instructing "retry" has been performed (step C42). . That is, at the end of playback of the original music data, the function assignments of the function keys F2 and F3 are changed, the replay function is assigned to the "F2" key, and the "(step C13)" is assigned to the "F3" key. "Retry" function is assigned, and it is determined in this step C42 whether a retry is instructed by the user operation of "F1". Therefore, when the retry instruction is issued (C42 to YES), the function assignments of the function keys F2 and F3 are restored to their original state, and then the process returns to Step C3l, and the operations of Steps C31 to C42 are repeated again. Create new position sequence data with respect to.

In addition, when a replay instruction is issued by the operation of the &quot; F2 &quot; key instead of the retry instruction (C42? NO), the sequence input processing is terminated, and the process returns to step C20 in FIG. The music ID and the position sequence data of the original music data are transmitted, and a request for conversion to 3D music data is requested. Although not shown, even when a "back" instruction is issued by the operation of the "F3" key, this sequence input processing is terminated and immediately returns to step C13 (equivalent to step C4 in FIG. 2) of FIG. Original music data can be selected.

Server-side processing

Fig. 8 is a flow chart showing music data generation processing on the server side according to an embodiment of the present invention, and this processing flow is based on the position sequence of the user's own reader, especially in the 3D conversion step S5 of Fig. 2. Shows a more detailed procedure of the 3D music conversion operation to be performed. The server SV stores a large number of original music data in the original music database DB, and in the user management data file, various information about the model of the client terminal CL, terminals such as a user ID and a user address, etc. I remember the information. As described above, these original piece music data are event sequence data representing the sound in MIDI format or an event format similar to the MIDI format, or audio signal data representing the sound waveform in a coding format such as ADPCM.

By the way, when the server SV receives the music ID and position sequence data of the original music data to be 3D converted from the client terminal CL and receives a request for data conversion to the 3D music, the received music ID is first received. And request contents such as the position sequence data, the model of the client terminal CL, and the user address of the client terminal CL (step S1). Next, the original music data specified by the music ID is read from the original music database DB (step S12), and the terminal information of the client terminal CL is read from the user management data file (step Sl3). ). Here, the terminal information to be read includes correction information such as a volume balance.

Next, the coordinate system of the received position sequence data is converted into an acoustic coordinate system (step Sl4). Then, the data portion of the original music data is regarded as the music track information Mt, and a position sequence track is added (step S15). Subsequently, the position sequence data obtained by transforming the coordinate system is recorded in the position sequence track to be the position sequence track information St, and the addition of the necessary information to the header portion of the original music data is added as the header information Hd. The music data is converted into stereoscopic image change effect-providing music data (step S16). In addition, when the music track information Mt is composed of a plurality of channels, the music channel corresponding to the position sequence track information St is assumed to be fixed (for example, a channel of the melody part). On the other hand, when the original music data is in the event sequence data format, the audio source circuit on the server SV side generates corresponding audio signal data based on the original music data, and converts the generated audio signal data into the soundtrack track information ( Mt).

Next, the balance correction which adjusts a volume balance etc. is performed with respect to the converted stereoscopic sound change effect provision music data (step S17). That is, depending on the specifications of the DSP mounted on the client terminal CL requesting 3D music conversion, the volume of the channel to which the 3D sound effect is applied may be lower than the volume of another channel to which the 3D sound effect is not applied. . Therefore, correction information such as volume balance is obtained from the model information of the user terminal, and volume correction of each channel of the soundtrack track information Mt is performed in accordance with the correction information to reproduce the volume balance optimized for the client terminal. .

Thereby, 3D music data Md which consists of sound track information Mt composed of audio signal data with balance correction and position sequence track information St composed of position sequence data whose coordinate system is transformed is generated. (Step Sl8). When the generated 3D music data Md is distributed to the client terminal CL (step S19), the music data generation process for the request is ended, and the process returns to the accepting step of the new request.

[Various embodiments]

As mentioned above, although one suitable Example of this invention was described, this is only an example and this invention can be variously changed in the range which does not deviate from the mind of invention, and can be implemented in various aspects. For example, in the embodiment, the server side provides 3D (stereoscopic sounding) music data converted into audio signal data such as ADPCM format, but the 3D conversion function is provided on the client terminal side to perform the conversion on the client terminal. It may be done.

In the embodiment, the original music data is transmitted from the server, but the music data stored in the client terminal can be used as the original music data. In this case, the 3D transformation may be requested by transmitting to the server along with the created position sequence, or 3D transformation may be performed on the client terminal as described above.

The music track information Mt of the 3D music data after the conversion is not limited to the ADPCM format. The digital audio signal data in other formats may be used, or may be in the form of event sequence data such as MIDI. In addition, in the case of 3D music data having music track information in the form of event sequence data such as MIDI, it is difficult to produce audio signal data in real time and give 3D sound effects in the client terminal. Prior to use of the music data, it is preferable to replace the music track information of the event sequence data format with the music track information of the corresponding audio signal data format in advance.

The 3D effect by the position sequence track information St can be added to the music track information Mt of a plurality of channels. For example, the same 3D effect may be added to another music channel of 3D music data in which the 3D sound effect is added to a specific music channel in advance. Moreover, you may input a position sequence multiple times with respect to one original music data, and may perform 3D conversion to each channel of original music data by each position sequence.

The position sequence track may be corrected while reproducing 3D music data to which a 3D effect has already been added. For example, if there is an input of a position sequence during reproduction of 3D music data given a 3D effect, the position sequence may be updated with relative displacement from the existing position sequence.

When converting into 3D music data, when the trajectory of three or more continuous positional displacements is monotonically changed on the plane passing through the origin, it may be replaced by movement between two points to compress the data amount. Alternatively, the data may be thinned at a reduced time resolution.

Although billing processing has not been described in the examples, billing can be performed at any point in time. For example, the charging process may be performed at the time when the converted 3D music data is downloaded and recorded on the client terminal so as to be reproduced (that is, registered in the data library of the client terminal). In this case, the charge for downloading the 3D music data and the charge for the conversion from the original music data to the 3D music data may be divided. For example, in the case of converting music data that has already been charged and stored in the client terminal, additional charging is only performed for the conversion process without recharging the music data itself.

In the music data generation system according to the present invention, the server SV is a mobile telephone terminal CLa or a personal having a communication function and a music reproduction function from the communication means 8 through a communication network CN such as the Internet. Original music accumulation that can transmit and receive between client terminals CL, such as computer CLb, and accumulate a plurality of pieces of music data (original music data) that are based on three-dimensional (3D) sound characteristic-provided music data (3D music data Md). A means DB, which transmits original music data in accordance with a request from the client terminal CL, or converts the original music data into three-dimensional acoustic characteristic-provided music data (3D music data Md) to the client terminal CL. You can betray

First, when the original music data is transmitted from the server SV to the client terminal CL (S4), the client terminal CL responds to the progress of the received original music data, and according to the user operation, the virtual sound image position ( The three-dimensional (3D) position coordinates to be Pg) are sequentially indicated, and the indicated series of coordinate positions are recorded in the storage means (C6, C19, C34 to C41). In this case, in the input method of the coordinate position, the first method C38 for continuously inputting the actual coordinate position sequentially and the second method C40 for inputting the coordinate position in accordance with the movement mode instructed by the user operation. There may be any one method or both methods in combination.

When the client terminal CL inputs a series of coordinate positions Pg corresponding to one piece of music data in this manner (C38, C40), the server sets the series of coordinate positions recorded in correspondence with the input as the position sequence data. (S20), the server (SV) adds the received position sequence data to the original music data to generate three-dimensional acoustic characterization music data (Md) (S14 to S18). The data is distributed to the client terminal CL (S19). The client terminal CL stores the three-dimensional acoustic characteristic-provided music data Md obtained from the server SV, and can reproduce the stereoscopic sound if necessary. For example, when the client terminal CL is the mobile telephone terminal CLa, it can be used as an incoming notification sound (incoming melody) that sounds as if a sound freely flows beyond the speaker position.

Therefore, according to the present invention, since the three-dimensional acoustic characterization music data is generated by recording the virtual sound source position (3D position information) on the basis of the positioning operation by the user in the client terminal, the user's own three-dimensional Sound characterization music content can be created. Furthermore, the virtual sound source position can be freely designated without arranging the music data. Further, since the user can designate an arbitrary virtual sound source position while reproducing the original music data, the 3D music data can be reproduced in three-dimensional sound matching the music. In addition, since two methods are prepared for the sequence input of the coordinate position, when the position designation is difficult, the position sequence can be generated in accordance with the automatic function by the second method prepared in advance.

Claims (4)

A system comprising a server and a client terminal communicatively connected via a communication network, the system comprising: The server, Original music storage means for accumulating a plurality of pieces of music data; Original music reading means for reading music data from the original music storing means; Sequence receiving means for receiving position sequence data from a client terminal; Three-dimensional music generation means for generating position sequence-provided music data reproducible as three-dimensional acoustic characteristics by adding the position sequence data to the music data read by the music reading means; Three-dimensional music transmission means for transmitting said positional sequence provision music data to a client terminal, The client terminal is Sequence input means for sequentially inputting position coordinates of a virtual sound image based on a user operation; Sequence recording means for recording the input position coordinates as position sequence data; Sequence transmitting means for transmitting the recorded position sequence data to the server; Three-dimensional music reception means for receiving position sequence-adjusted music data from a server; Three-dimensional music storage means for reproducingly storing the received positional sequence provision music data; Three-dimensional music reproducing means for reproducing the stored position sequence-provided music data as three-dimensional acoustic characteristics Music data generating system comprising a. The method of claim 1, The server further comprises original music transmission means for transmitting music data read by the music reading means to the client terminal in response to a request from the client terminal, The client terminal, Request transmitting means for transmitting a request for music data to a server; Original music receiving means for receiving music data transmitted from the server in accordance with the transmitted request; Original music storage means for reproducibly storing the received music data, The input of the position coordinates by the sequence input means is performed in parallel with the reproduction of the music data stored in the original music storage means. The method according to claim 1 or 2, The sequence input means includes: first input means for continuously inputting coordinate positions sequentially instructed by a user operation, and / or second second for generating and inputting a coordinate position sequence according to a movement mode instructed by a user operation. A music data generation system comprising input means. To a computer that is communicatively connected to a server via a communications network, A request sending step of sending a request for music data to a server, An original music reception step of receiving music data transmitted from the server in accordance with the transmitted request; An original music storage step for reproducingly storing music data received in the original music reception step, and By reproducing music data stored in the original music storage step, continuously inputting coordinate positions sequentially indicated by user operation, and / or inputting coordinate positions according to the movement mode indicated by user operation. A sequence input step of sequentially inputting position coordinates of a virtual sound image, A sequence recording step of recording a series of position coordinates input in the sequence input step as position sequence data; A sequence transmission step of transmitting the recorded position sequence data to the server; A three-dimensional music reception step of receiving, from the server, position sequence-provided music data that can be reproduced as a three-dimensional acoustic characteristic, generated by adding the position sequence data to the music data corresponding to the request transmitted in the request transmission step; A three-dimensional music storage step for reproducibly storing the positional sequence-provided music data received in the three-dimensional music reception step; A computer-readable recording medium storing a music data generating program for executing a procedure comprising a three-dimensional music reproducing step of reproducing a position sequence-provided music data stored in a three-dimensional music storage step as a three-dimensional acoustic characteristic.

Images