JP2024118633A - Image processing device, image processing program, and image processing method - Google Patents

Abstract

[Problem] To realize telop playback using a GPU (Graphics Processing Unit). [Solution] An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and plays telop data, the telop data including vertex data indicating the display position of each segment of the telop and texture data to be attached to each segment of the telop, causes the CPU to execute a first transmission process for transmitting a telop formation program to the storage unit, a second transmission process for transmitting the telop data to the storage unit, and a content playback process for playing content data, and causes the GPU to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data, based on the telop formation program. [Selected Figure] Figure 1

Description

The present invention relates to an image processing device, an image processing program, and an image processing method.

In recent years, development techniques that actively use GPUs (Graphics Processing Units) for image formation in various information processing devices have become common. GPUs are calculation devices, or processors, that specialize in real-time image processing, and excel at parallel calculation processing, achieving calculation speeds several times to over 100 times faster than CPUs (Central Processing Units).

Patent Document 1 describes how one screen is divided into a video display area (TV or other images), an information display area (waiting times and call numbers often used in hospitals), and a subtitle display area (advertisements). The subtitle display area is an area for displaying static information such as text advertisements. It describes how by displaying images converted into a raster format, processing by the GPU is easy, allowing the subtitles to be displayed smoothly without any jerks.

JP 2019-174734 A

The invention described in Patent Document 1 only allows captions to be displayed in the caption display area, and does not allow captions to be displayed in a free layout or with free expression.

In addition, in various information processing devices, the processing load on the CPU is increasing as functions are added and improved. In a processing device that handles subtitles, such as a karaoke device, if the subtitle processing is performed by the CPU, it is conceivable that the processing resources of the CPU will be reduced.

The present invention has been developed in consideration of these circumstances, and one of its objectives is to enable a variety of subtitle expressions using GPU processing by executing subtitle-related image processing on the GPU. Another objective is to efficiently perform subtitle-related image processing, thereby reducing the amount of processing on the CPU.

For this reason, the image processing device according to the present invention employs the following configuration.
An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that reproduces telop data,
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
Based on the telop formation program, the GPU is caused to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data.

Further, in the image processing device according to the present invention,
The subtitle formation program includes a vertex shader program and a pixel shader program.

Further, in the image processing device according to the present invention,
The first transmission process enables different telop formation programs to be transmitted.

Further, in the image processing device according to the present invention,
The first transmission process transmits a telop formation program corresponding to the content data to be reproduced in the content reproduction process.

Further, in the image processing device according to the present invention,
The first transmission process transmits a telop formation program selected by the user.

Further, in the image processing device according to the present invention,
The telop data is configured as a data dimensional array that is handled by the GPU, and includes at least reference data that defines the timing of each fragment.

Further, in the image processing device according to the present invention,
the CPU executes a third transmission process of transmitting a playback position of the content data to the GPU;
The GPU executes the telop playback process based on the playback position sent from the CPU and the timing of each fragment defined by the reference data.

Further, in the image processing device according to the present invention,
The timings defined in the reference data include the start and end times of each segment.

The image processing program according to the present invention further comprises:
An image processing program executed on an information processing device including a central processing unit (CPU), a graphics processing unit (GPU), and a storage unit accessed by the GPU,
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
Based on the telop formation program, the GPU is caused to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data.

The image processing method according to the present invention further comprises the steps of:
An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising:
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
Based on the telop formation program, the GPU is caused to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data.

The present invention may also adopt other forms that will be described below.
According to another embodiment, an image forming apparatus includes:
An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that reproduces telop data,
The telop data is
vertex data indicating the display position of each fragment of the telop;
The texture data to be attached to each subtitle fragment,
The data is composed of a data dimensional array that is handled by the GPU, and includes at least reference data that defines the timing of each fragment;
The CPU is
a transmission process for transmitting the telop data to a storage unit;
A content reproduction process for reproducing the content data is executed;
The GPU is
execute a telop reproduction process for reproducing a telop based on the telop data;
The telop playback process is performed in synchronization with the content playback process at a timing defined by the reference data.

In still another embodiment of the image processing device,
The CPU generates at least one of vertex data, texture data, and reference data in the texture data from the telop source data corresponding to the content data.

In still another embodiment of the image processing device,
The timing of each segment defined in the reference data includes the start and end times of each segment.

In still another embodiment of the image processing device,
The timing of each segment defined in the reference data further includes the end time of the previous segment and the start time of the next segment.

In still another embodiment of the image processing device,
The reference data defines the timing of a fragment collection that includes multiple fragments.

In still another embodiment of the image processing device,
The timing of a fragment collection defined in the reference data includes a start time and an end time of the fragment collection.

In still another embodiment of the image processing device,
The reference data includes position information for each fragment,
The telop playback process plays back the telop based on the position information defined by the reference data.

In still another embodiment of the image processing device,
The position information of each fragment defined by the reference data includes rectangular information indicating the planar extent of each fragment.

In still another embodiment of the image processing device,
The reference data defines position information of a fragment collection that includes a plurality of fragments.

In still another embodiment of the image processing device,
The position information of the fragment collection defined by the reference data includes rectangular information indicating the planar extent of the fragment collection.

Further, an image processing program according to another embodiment includes:
An image processing program executed on an information processing device including a central processing unit (CPU), a graphics processing unit (GPU), and a storage unit accessed by the GPU,
The telop data is
vertex data indicating the display position of each fragment of the telop;
The texture data to be attached to each subtitle fragment,
The data is composed of a data dimensional array that is handled by the GPU, and includes at least reference data that defines the timing of each fragment;
causing a CPU to execute a transmission process of transmitting telop data to a storage unit and a content reproduction process of reproducing content data;
causing the GPU to execute a telop reproduction process for reproducing a telop based on the telop data;
The telop playback process is performed in synchronization with the content playback process, based on the timing defined by the reference data.

In addition, an image processing method according to another embodiment includes the steps of:
An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising:
The telop data is
vertex data indicating the display position of each fragment of the telop;
The texture data to be attached to each subtitle fragment,
The data is composed of a data dimensional array that is handled by the GPU, and includes at least reference data that defines the timing of each fragment;
causing a CPU to execute a transmission process of transmitting telop data to a storage unit and a content reproduction process of reproducing content data;
causing the GPU to execute a telop reproduction process for reproducing a telop based on the telop data;
The telop playback process is performed in synchronization with the content playback process at a timing defined by the reference data.

According to the present invention, it is possible to easily express various types of subtitles using GPU processing. In addition, it is possible to have the GPU perform image processing related to subtitles, which makes it possible to reduce processing by the CPU.

FIG. 1 shows a configuration of a karaoke system according to an embodiment of the present invention. A flow diagram showing the regeneration process of this embodiment. FIG. 1 is a diagram for explaining the formation of texture data according to the present embodiment; FIG. 1 is a diagram for explaining vertex data according to an embodiment of the present invention; FIG. 1 is a diagram for explaining the concept of index data according to the present embodiment; FIG. 1 is a diagram showing a configuration of reference data according to an embodiment of the present invention; FIG. 13 is a diagram showing an example of caption generation on a screen according to the present embodiment;

Figure 1 shows the configuration of the karaoke system of this embodiment. The karaoke system of this embodiment includes a karaoke device 2 (sometimes called a commander) and a remote control device 1. In this embodiment, the karaoke device 2 and the remote control device 1 are communicatively connected to a LAN 100 by wire or wirelessly to form a network.

The karaoke device 2 installed in a store such as a karaoke booth is equipped with an audio control unit 25 that plays music (accompaniment). The audio control unit 25 of this embodiment applies effects such as echo to the audio signals input from each microphone 43a, 43b, adds a playback signal based on music information, and outputs the result to the speaker 42. Note that, for convenience, one speaker 42 is shown in FIG. 1, but it is possible to adopt various forms such as multiple speakers 42 and stereo or surround output.

The karaoke device 2 of this embodiment is configured by connecting various devices via interfaces 22a to 22d. Note that while four interfaces 22a to 22d are used in FIG. 1, the interfaces 22a to 22d can take various forms, such as combining them into one, providing one for each device, etc.

The karaoke device 2 includes an operation unit 21 that accepts various inputs from the user. The karaoke device 2 also includes a hard disk 32 that serves as a storage unit for storing various information. The karaoke device 2 also includes a LAN communication unit 24a that serves as a communication means for connecting to the LAN 100 and joining the network. The karaoke device 2 of this embodiment also includes a wireless LAN communication unit 24b, and instead of a wired network connection using the LAN communication unit 24a, it is also possible to establish a wireless network connection using the wireless LAN communication unit 24b.

The karaoke device 2 also includes a video playback unit that displays lyrics video, background video, and the like on the monitor 41. The video playback unit includes a GPU 29 (Graphics Processing Unit) that forms video based on various information, a RAM 28 (corresponding to the "storage unit accessed by the GPU" according to the present invention) that stores information required for video formation, and a video output unit 31 that outputs the played video to the monitor 41 and the touch panel monitor 33. These video playback units may be provided as a video card configured as a single device. Note that a cache that temporarily stores data may be provided between the GPU 33 and the RAM 28. This cache also corresponds to the "storage unit accessed by the GPU" according to the present invention.

Furthermore, in this karaoke device 2, in addition to the externally connected monitor 41, it is possible to display various information on the touch panel monitor 33. The touch panel monitor 33 is composed of a display unit 35 that displays the video input from the video output unit 31, and a touch panel 34 that outputs the touch input position to the interface 22b. The touch panel monitor 33 in this embodiment is disposed on the front of the housing of the karaoke device 2, and functions as an input unit that accepts various operations from the user. By selecting a song on the touch panel monitor 33, the user can perform various operations on the karaoke device 2, such as directly reserving a song on the karaoke device 2.

The karaoke device 2 further includes a control unit including a CPU 30 (Central Processing Unit) for controlling each component, and a memory 27 for temporarily storing information required to execute various programs.

With this configuration, the karaoke device 2 executes various processes, but its main functions include song reservation and playback processes. The song reservation process is a process for specifying and reserving a song based on a user's specification, and is executed in cooperation with the remote control device 1. Reservation information formed by the song selection process of the remote control device 1 is sent to the karaoke device 2. The karaoke device 2 registers the received reservation information in a reservation table in the memory 27. The playback process is a process for playing back a reserved song, and is a process in which performance processing, lyric display processing, and video playback processing are executed synchronously.

The performance process is a process that causes the audio control unit 25 to perform a performance based on the performance data included in the music data. The music played by the audio control unit 25 is mixed with audio signals input from the microphones 43a and 43b and is output from the speaker 42. The lyrics display process is a process that provides singing assistance by displaying a lyrics video (caption) based on the lyrics data included in the music data on the monitor 41. A background video display process may also be executed to superimpose a background video on the lyrics video displayed in this lyrics display process.

The remote control device 1, on the other hand, is capable of performing a song selection process that searches for songs based on instructions from the user and transmits reservation information for the song instructed to be played to the karaoke device 2. The remote control device 1 is also capable of receiving various information from the karaoke device 2 or a server device 5 connected to the Internet and performing various processes. In this embodiment, the remote control device 1 is equipped with an operation unit 17 and a touch panel monitor 11 as a user interface that receives various instructions from the user. The touch panel monitor 11 is configured with a display unit 11a and a touch panel 11b, and is capable of displaying various information on the display unit 11a and receiving touch input from the user.

The remote control device 1 further comprises a memory 14 as a storage unit for storing the database required for the music selection process, various programs, and various information generated by program execution, and a remote control control unit for controlling these components in an integrated manner. The remote control control unit includes a CPU 15, an image control unit 13 that forms images to be displayed on the touch panel monitor 11, a video RAM 12 that temporarily stores image data for the images to be displayed, and an operation processing unit 18 that interprets input from the touch panel monitor 11 or the operation unit 17 and transmits the input to the CPU 15.

The remote control device 1 is connected to the network formed by the LAN 100 by wirelessly connecting to the access point 130 via the wireless LAN communication unit 16. Each remote control device 1 is pre-associated with a specific karaoke device 2. Various commands output from the remote control device 1 are received by the associated karaoke device 2.

The remote control device 1 is configured in this way to accept various inputs from the user via the touch panel monitor 11 or the operation unit 17, and by providing various information through the display on the touch panel monitor 11, it is possible to perform various processes, such as song selection processing that transmits reservation information to be output to the karaoke device 2.

Figure 2 is a flow diagram showing the playback process according to this embodiment. The playback process is executed by the CPU 30. When a song is reserved using the remote control device 1 or the operation unit 21 of the karaoke device 2, the song ID indicating the reserved song is registered in a reservation table managed in the memory 27. In the playback process, this reservation table is checked (S101) to see whether there is a song to be played next.

If there is a song to be played next, song data corresponding to that song is read (S103). The song data may be pre-stored in the karaoke device 2, or may be obtained by being distributed from the server device 5 or the like. In this embodiment, the song data includes performance data and lyric data (corresponding to the "subtitle source data" of the present invention). The performance data is data that is played (performed) by the audio control unit 25. The lyric data is data that is displayed on the monitor 41 in synchronization with the performance to assist singing.

In this embodiment, before a performance based on the performance data is executed, caption data is created based on the lyrics data (S104). The caption data is data for displaying captions using the GPU 29. Note that in this embodiment, the caption data is created from the lyrics data, but instead of this form, the caption data may be included in the music data in advance.

The lyric data in this embodiment is composed of text data, and is composed of spaces, line breaks, and blank lines. In addition, timing data indicating the display start time, erasure time, and singing position are specified for each appropriate unit such as each character. Conventionally, the CPU 30 mounted on the karaoke device 2 has used such lyric data to display lyrics on the monitor 41.

However, with the karaoke device 2, the processing load on the CPU 30 is also increasing as the number of functions increases. In this situation, processing related to displaying lyric data is one cause of the processing load on the CPU 30. Furthermore, having the CPU 30 express new lyrics is likely to result in a significant processing load on the CPU 30.

Therefore, in this embodiment, the lyrics data that was previously used to display lyrics is converted into telop data that can be easily handled by the GPU 29, and the lyrics are displayed using the GPU 29. The CPU 30 creates the telop data based on the lyrics data.

The telop data in this embodiment is composed of (1) texture data, (2) vertex data, and (3) reference data. These data are explained below.

In the embodiment, the smallest unit is a character, but the smallest unit can take various forms, such as multiple characters or a portion of a character. In the present invention, a smallest unit such as a "character" used in the above embodiment is called a "fragment," and a collection of "fragments" such as a "phrase," "line," or "chapter" is called a "fragment collection."

(1) Texture Data Fig. 3 is a diagram for explaining the formation of texture data. This example takes the lyrics of the nursery rhyme "Elephant" shown in Fig. 4 as an example. First, overlapping characters are removed from the text data forming the lyrics information. Next, using font data or the like, the characters from which overlapping has been removed are rasterized (bitmapped) to form a single piece of image data.

(2) Vertex Data Vertex data is an array of data related to each vertex on a three-dimensional model. In other words, it is an array of data related to each vertex that forms a plurality of polygons. In this embodiment, one polygon is assigned to one character. Note that one polygon may be assigned to a portion of a character or to multiple characters. The data related to each vertex includes three-dimensional coordinates, texture coordinates, and index data.

(2-1) Three-dimensional coordinates Three-dimensional coordinates are data that define the position of polygons that are placed in three-dimensional space. FIG. 4 shows an example of the lyrics of the nursery rhyme "Elephant" placed in three-dimensional space. The three-dimensional coordinates indicate the position, i.e., the coordinates, of each vertex of each polygon that corresponds to each character that makes up the lyrics when the lyrics are placed in three-dimensional space. Note that the characters shown within the polygons in the figure are added for ease of understanding, and in reality, the characters are visualized by pasting image data cut out from texture data onto the polygons.

(2-2) Texture Coordinates Texture coordinates are data that define the position of the portion of the texture data to be applied to each polygon. In other words, texture coordinates indicate the positions, or coordinates, that correspond to the vertices of each polygon in the texture data. By referencing the texture coordinates, it is possible to extract the necessary character image data from the texture data. In other words, it is possible to extract the necessary character image data from the texture data and apply it to each polygon.

(2-3) Index Data Index data is information assigned to each vertex forming a polygon. Fig. 5(A) is a diagram for explaining index data. In this embodiment, lyrics are assigned information on "characters" consisting of one character, "phrases" consisting of one or more characters, "lines" consisting of one or more phrases, and "chapters" consisting of one or more lines.

As mentioned above, the lyrics data in this embodiment is composed of text data that includes spaces, line breaks, and blank lines. When creating index data from lyrics data, for example, a block of text up to a space is determined as a "phrase," a block of text up to a line break is determined as a "line," and a block of text up to a blank line is determined as a "chapter." In addition to this format, various other formats can be adopted, such as tagging lyrics data with "phrases," "lines," and "chapters."

Figure 5(B) is a diagram for explaining the index data assigned to each vertex of a polygon. Figure 5(B) explains the example of the first two phrases in Figure 5(A), "Zou-san Zou-san." In this embodiment, the initial value for characters, phrases, lines, and chapters is "0."

For the first character (0th character), "zo", index data (0,0,0,0) is assigned to the top left and bottom left vertices. This index data indicates that it is the 0th character, 0th phrase, 0th line, and 0th chapter. Additionally, index data (0.99,0,0,0,0) is assigned to the top right and bottom right vertices of the first character (0th character), "zo". The value "0.99" is assigned to the character because a smaller value is used in order to assign 1 to the next character (first character).

For the next character (first character), "u", index data (1,0,0,0) is assigned to the top left and bottom left vertices. Index data (1.99,0,0,0,0) is assigned to the top right and bottom right vertices of "u". Index data is assigned to the vertices of the second character, "sa", and the third character, "n", using the same rules.

In the next phrase (first phrase), for the fourth character "zo", index data (4,1,0,0) is assigned to the top left and bottom left vertices. This index data indicates that it is the fourth character, first phrase, 0th line, and 0th chapter. Furthermore, index data (4.99,1,0,0,0) is assigned to the top right and bottom right vertices of "u". Index data is assigned to the vertices of the fifth character "u", the sixth character "sa", and the seventh character "n" using the same rule.

By assigning index data to each vertex of a polygon in this way, it is possible to easily control subtitles by character, phrase, line, or chapter. Also, because the index data is a four-dimensional array (characters, phrases, lines, chapters) just like the data handled by the GPU, it can be processed efficiently in parallel on the GPU.

(3) Reference Data Reference data is data that specifies various information about the characters, phrases, lines, and chapters that form lyrics. Fig. 6 is a diagram for explaining the reference data of this embodiment. As shown in Fig. 6, the reference data has information for each character T, phrase P, line L, and chapter C.

For the 0th character (the character "zo" in the lyrics example in Figure 5(A)), index data, rectangular information indicating the position and size of the character (corresponding to "position information"), and the time range of the character are included. The index data is the information explained in Figure 5(B), and for this 0th character, it is (0,0,0,0).

The rectangular information indicating the position and size of the character is the same information as the vertex data described in Figure 4, and in the case of the 0th character, it is information indicating the position and size of the polygons that make up "zo." When processing each piece of vertex data, the GPU cannot refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the character in the reference data, it becomes possible to refer to the control position of the character to which the vertex data belongs or other characters by referring to the reference data when processing each piece of vertex data.

The time range of a character is information indicating timing. In this embodiment, (Sn-1_end, Sn_start, Sn_end, Sn+1_start) is assigned as the time range of a character. Sn-1_end indicates the end timing of the previous character Sn-1, Sn_start indicates the start timing of the character Sn, Sn_end indicates the end timing of the character Sn, and Sn+1_start indicates the start timing of the next character Sn+1.

When displaying the character Sn, two pieces of information, the start timing Sn_start of the character Sn and the end timing Sn_end of the character Sn, are sufficient. However, in this embodiment, like other data handled by the GPU, a four-dimensional data structure is used, and Sn-1_end and Sn+1_start are included. By including these, it is possible to perform processing using the timing of the previous character and the following character. As a comparative example, if Sn-1_end and Sn+1_start are not included, when performing processing using the timing of the previous character and the following character, it is necessary to refer to the reference data of the character Sn, the reference data of the character Sn-1, and the reference data of the character Sn+1, which results in a large number of references and a high processing load. On the other hand, in this embodiment, it is possible to reduce the number of references compared to the comparative example, thereby reducing the processing load.

Note that Sn-1_end and Sn+1_start may not be included in the time range. In that case, the time range will have a two-dimensional data structure, so by including two null data, it may become a four-dimensional data structure, making it easier to handle on the GPU. The same applies to the time ranges of phrases, lines, and chapters, which will be explained later.

Similar to the 0th character, the reference data specifies information from 1st to mXth (the last character).

For the 0th phrase (the character "zou-san" in the example lyrics of Figure 5(A)), index data for the first character of the phrase, rectangular information indicating the position and size of the phrase, and the time range of the phrase are included. The index data for the first character of the phrase is stored as the index data for the 0th character, (0,0,0,0).

In the case of the 0th phrase, the rectangular information indicating the position and size of the phrase is information indicating the position and size of the area surrounding the "elephant." When processing each piece of vertex data, the GPU cannot refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the phrase in the reference data, it is possible to refer to the control position of the phrase when referencing the reference data.

The time range of a phrase is information indicating timing. In this embodiment, (Tn-1_end, Tn_start, Tn_end, Tn+1_start) is assigned as the time range of a phrase. Tn-1_end indicates the end timing of the previous phrase Tn-1, Tn_start indicates the start timing of the phrase Tn, Tn_end indicates the end timing of the phrase Tn, and Tn+1_start indicates the start timing of the next phrase Tn+1.

As with text, when displaying the phrase Tn, two pieces of information are sufficient: the start timing Tn_start of the phrase Tn and the end timing Tn_end of the phrase Tn. However, in this embodiment, by including Sn-1_end and Sn+1_start, it is possible to perform processing using the timing of the previous and following phrases.

Similar to the 0th phrase, the reference data specifies information from 1st to mYth (the last phrase).

As with phrases, lines (or chapters) also contain index data for the first character of the line (or chapter), rectangular information indicating the position and size of the line (or chapter), and the time range of the line (or chapter).

In this way, the reference data of this embodiment specifies index data, rectangular information indicating the position and size, and time range (timing) for characters, phrases, lines, and chapters, making it possible to easily control the display timing of characters, phrases, lines, and chapters. Note that it is sufficient for the reference data to include information indicating the time range (timing) for characters, phrases, lines, and chapters.

In this embodiment, the GPU 29 uses the above-mentioned telop data (texture data, vertex data, reference data) to display telops on the monitor 41. At that time, by changing the shader program for forming the telops in the GPU 29, it is possible to perform a variety of telop expressions. In this embodiment, the CPU 30 transmits a shader program corresponding to a piece of music to the RAM 28 for each playback process (S106). For example, by using a shader program corresponding to a piece of music, it is possible to realize a telop expression corresponding to the music.

The shader program of this embodiment (corresponding to the "telop formation program" of the present invention) is a program that specifies how the caption is displayed on the screen, and is capable of controlling the position and color of characters, chapters, lines, and phrases. The shader program of this embodiment is composed of a vertex shader program and a pixel shader program.

The vertex shader program is a program that specifies the display position, and can control the display, for example, by character, phrase, line, or chapter. Display control includes layouts such as right-justified, left-justified, stepped, and line breaks, as well as animations such as scrolling and pop-ups. Display control is not limited to the layouts and animations shown as examples, but includes a variety of other layouts and animations. In addition, the vertex shader program can express subtitles as a stereoscopic image captured from a specified camera viewpoint in a three-dimensional space.

On the other hand, the pixel shader program can control the color information for each pixel that forms the caption, and can control various color expressions for the caption to be displayed, such as a single color or a gradation.

In this embodiment, a shader program (corresponding to the "telop formation program" of the present invention) is formed by combining a vertex shader program and a pixel shader program. By preparing multiple vertex shader programs and multiple pixel shader programs in advance and changing the combination, it is possible to form shader programs that express a variety of telops.

After sending the telop data and shader program to RAM 28, making it possible for GPU 29 to display telops, CPU 30 causes the audio control unit 25 to start playing the performance data (S107). At that time, the current performance position (playback position) is sent to GPU 29 in synchronization with the playback of the performance data. The current performance position is sent to GPU 29 periodically (for example, for each frame to be drawn). GPU 29 displays lyrics (telop display) on monitor 41 based on the telop data and shader program stored in RAM 28, and the current performance position received from CPU 30.

The vertex shader program executed by GPU 29 refers to the time range (timing) in the reference data, determines the characters (or phrases, lines, or chapters) to be displayed, and determines the display position based on the vertex data. At this time, depending on the vertex shader program, coordinate conversion is performed. The pixel shader program can also control the color of the character image to be displayed, and can display characters on polygons based on images extracted from texture data. The pixel shader program can also change the color of characters based on the current playing position, for example, or can display characters in a variety of colors.

Figure 7 shows an example of the display of subtitles on the screen in this embodiment. Figure 7 (A) shows a display form in which subtitles are displayed horizontally in the lower half of the screen, similar to conventional karaoke devices. The vertex program performs coordinate conversion on the vertex data explained in Figure 4, and arranges multiple polygons for a specified number of lines (two lines in the example of Figure 7) on the screen. Next, the pixel shader program displays characters on each polygon based on an image extracted from the texture data. The pixel shader program also determines the current singing position based on the current position sent from the CPU, and displays the subtitles by changing the color before and after the current singing position and outputting the display.

Figure 7 (B) shows a form in which subtitles are displayed over the entire screen. Figure 7 (B) shows a display form in which line breaks are performed for each phrase, and multiple phrases displayed on the screen are displayed in a staircase pattern. In this case, the vertex shader program performs coordinate conversion on the vertex data explained in Figure 4, and arranges multiple polygons on the screen so that a predetermined amount of phrases (four phrases in the example of Figure 7) are arranged in a staircase pattern. Next, the pixel shader program displays characters on each polygon based on an image extracted from the texture data. The pixel shader program also determines the current singing position based on the current position sent from the CPU, and displays the subtitles by changing the color before and after the current singing position and outputting the display.

Figures 7(A) and 7(B) show two-dimensional display, but by changing the shader program, it is possible to freely express layouts, such as displaying subtitles as seen from a certain viewpoint in three-dimensional space.

In this way, in this embodiment, the processing related to the playback of subtitles can be realized by the GPU 29, and the CPU 30 only needs to transmit the current playing position to the GPU 29, which makes it possible to reduce the processing load on the CPU 30. Also, by changing the shader program, it is possible to realize a variety of subtitle expressions.

The CPU 30 continues to transmit the current playing position (S108) until the playback of the performance data ends. When the playback of the performance data ends (S109: Yes), the reservation table is checked to play the next song (S101).

In the above, this embodiment has been described using a karaoke system as an example to display lyrics (captions), but in this embodiment, by using a shader program, caption data optimized for the GPU 29, and a shader program, it is possible to display captions using the GPU 29. Therefore, it is possible to display captions in a variety of ways, and it is possible to reduce the processing load on the CPU 30.

[First Modification]
Next, a modified example of the image processing device according to the present invention will be described. In the above embodiment, the display of telops has been described by taking the example of displaying lyrics using the karaoke device 2. The display of telops is not limited to displaying lyrics, and may be used for subtitles of movies and moving images. In this case, the information processing device that reproduces the telops corresponds to the image processing device.

[Second Modification]
In the above embodiment, a shader program (corresponding to the "telop formation program" of the present invention) is transmitted to the RAM 28 for each playback process. In addition to this form, various forms can be adopted for changing the shader program. For example, the shader program may be changed by a user operation. In this case, the shader program may be one prepared in advance, or may be changeable to a shader program created by the user himself.

Although the above-described embodiment does not mention in detail the correspondence between songs and shader programs, shader programs may be associated with at least one of each of the following: each song, each genre of song, each singer, each composer, and each lyricist. Alternatively, shader programs may be associated with timing information such as seasons, days of the week, and times of day. Alternatively, shader programs may be selected randomly.

Although various embodiments have been described above using the karaoke machine 2 as an example, the image processing device according to the present invention can be used not only as the karaoke machine 2, but also as various information processing devices such as personal computers, game devices, and smartphones. In addition, image processing programs and image processing methods executed in various information processing devices also fall within the scope of the invention.

1: Remote control device 24a: LAN communication unit 2: Karaoke device 24b: Wireless LAN communication unit 5: Server device 25: Sound control unit 11: Touch panel monitor 27: Memory 11a: Display unit 28: RAM
11b: touch panel 30: CPU
12: Video RAM 31: Video output section 13: Video control section 32: Hard disk 14: Memory 33: Touch panel monitor 15: CPU 34: Touch panel 16: Wireless LAN communication section 35: Display section 17: Operation section 41: Monitor 18: Operation processing section 42: Speaker 21: Operation section 43a, 43b: Microphones 22a to 22d: Interface 130: Access point

Claims (10)

An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that reproduces telop data,
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
The image processing device causes a GPU to execute a telop playback process for performing a display based on telop data in synchronization with playback of content data based on a telop formation program. The image processing device according to claim 1 , wherein the telop formation program includes a vertex shader program and a pixel shader program. The image processing device according to claim 1 , wherein the first transmission process is capable of transmitting different telop forming programs. The image processing apparatus according to claim 1 , wherein the first transmission process transmits a telop formation program corresponding to the content data to be reproduced in the content reproduction process. The image processing apparatus according to claim 1 , wherein the first transmission process transmits a telop forming program selected by a user. 2. The image processing device according to claim 1, wherein the telop data is configured as a data dimensional array handled by the GPU, and includes at least reference data defining the timing of each fragment. the CPU executes a third transmission process of transmitting a playback position of the content data to the GPU;
The image processing device according to claim 6 , wherein the GPU executes a telop playback process based on a playback position transmitted from the CPU and timing of each fragment defined by the reference data. The image processing apparatus of claim 7 , wherein the timing defined by the reference data includes a start time and an end time of each fragment. An image processing program executed on an information processing device including a central processing unit (CPU), a graphics processing unit (GPU), and a storage unit accessed by the GPU,
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
An image processing program that causes a GPU to execute a telop playback process that performs a display based on telop data in synchronization with playback of content data based on the telop generation program. An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising:
The telop data is
vertex data indicating the display position of each fragment of the telop;
texture data to be attached to each fragment of the telop;
causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data;
An image processing method, comprising: causing a GPU to execute a telop playback process for performing a display based on telop data in synchronization with playback of content data, based on a telop formation program.