JP6648003B2 - Information processing apparatus, control method of information processing apparatus, and control program - Google Patents

Description

The present invention relates to an information processing system that extracts and outputs frames from a movie.
The present invention particularly relates to an information processing system that extracts a frame and a sound from a movie and superimposes a text corresponding to the sound on the frame.

There is known a method of automatically extracting a still image of interest from a moving image with sound and creating a photo album (for example, see Patent Document 1).
Specifically, an attention portion that matches a designated waveform or a designated text is specified from waveform data or text data of audio data constituting the moving image data with sound, and the attention portion is determined from the moving image data constituting the moving image data with sound. A still image corresponding to the portion is extracted as a still image of interest, and a photo album in which audio text of the attention portion is added to the still image of interest is created (for example, [Claim 4] and [Claim 6] of Patent Document 1). .

JP 2006-333065 A

  When superimposing an object on a still image, it is convenient if the arrangement mode of the object can be set according to the display content of the still image. Patent Literature 1 does not disclose a specific arrangement of how text is added to a still image of interest.

  A problem to be solved by the present invention is to superimpose and arrange a text corresponding to audio extracted from a movie on a frame extracted from the movie in a highly appealing manner.

[A] In order to solve the above problem, an “information processing device” according to an aspect of the present invention includes a sound source of a designated sound included in a designated range of a movie in relation to a designated frame included in the designated range of the movie. A specifying unit configured to specify a position; and a setting unit configured to set an arrangement mode of a superimposed text to be superimposed on the specified frame in correspondence with the specified voice in consideration of a position of a sound source specified by the specifying unit. .
[B] In order to solve the above problem, an “information processing device control method” according to an aspect of the present invention includes a designated sound included in a designated range included in a designated range of a movie in relation to a designated frame included in the designated range. A specifying step of specifying the position of the sound source; and a setting step of setting the arrangement mode of the superimposed text superimposed on the specified frame corresponding to the specified voice in consideration of the position of the sound source specified in the specifying step. ,including.
[C] In order to solve the above-described problem, a “control program” according to an embodiment of the present invention includes a position of a sound source of a designated sound included in a designated range in relation to a designated frame included in the designated range of a movie. And a setting function of setting a layout mode of superimposed text superimposed on the specified frame corresponding to the specified voice in consideration of a position of a sound source specified by the specific function. It is realized by the computer of the device.
[D] In order to solve the above-mentioned problem, a control program recorded on a “computer-readable recording medium” according to an embodiment of the present invention includes: The specific function for specifying the position of the sound source of the designated voice included in the above, and the arrangement mode of the superimposed text that is superimposed and placed on the specified frame corresponding to the specified sound in consideration of the position of the sound source specified by the specific function. And a setting function for setting the information by the computer of the information processing apparatus.

The following technical limitation may be added to the “information processing device” of [A]. Further, the same technical limitation may be added to the “control method” of [B], the “control program” of [C], and the control program recorded by the “recording medium” of [D].
The setting means sets the arrangement mode according to a relationship between a position of the sound source specified by the specifying means and a display range of the specified frame.
The setting means sets the arrangement mode such that the superimposed text is arranged in association with the sound source when the position of the sound source is inside the display range, and the position of the sound source is set in the display range. The arrangement mode is set so that the superimposed text is arranged without being associated with the sound source when the text is outside.
The specifying means specifies the position of the sound source in accordance with a position specified by a user operation serving both as a position specification and an area specification, and the setting means performs the superimposition in accordance with an area specified by the user operation Set the text placement area.
The user operation is an operation of drawing a closed figure, the specifying means specifies the position of the sound source according to the position of an end point of the user operation, and the setting means forms a closed figure drawn by the user operation. The arrangement area of the superimposed text is set according to the position of the closed area to be placed.
A designating means for designating, as the designated range, a designated number range selected by a user from a candidate list including at least a plurality of candidate ranges, wherein the identifying means is provided for each designated range designated by the designated means; The position of a sound source of a designated sound included in the designated range is specified, and the setting unit is superimposed on a designated frame included in the designated range for each designated range designated by the designating unit. The arrangement mode of the superimposed text is set in consideration of the position of the sound source of the designated voice included in the designated range specified by the specifying unit.
The specifying means presents to the user the candidate list including the plurality of candidate ranges respectively associated with the plurality of candidate texts respectively corresponding to the group of sounds included in the movie, and specifies the specified number range; Let the user choose.

In this specification, the following terms are used.
"Movie" includes at least a plurality of frames and audio data.
"Specified range" refers to a temporal range specified in the timeline of "movie".
“Frame” refers to a still image that constitutes a “movie”.
The position of the sound source of “specified voice” exists inside or outside the range displayed by “specified frame”.
-The "superimposed text" is, for example, a converted text converted from the "specified voice", a supplementary text supplemented at the detection position of the "specified speech", an edited text obtained by editing the converted text or the supplementary text, and the like. Refers to text derived from “designated voice”.
The “arrangement mode” refers to a manner of arranging the “superimposed text”.
The “user operation” is performed using a position input device (for example, a touch panel, a mouse, a touch pad, or the like) that continuously inputs a designated position on the screen.

According to the present invention, the arrangement mode of the superimposed text superimposed on the specified frame is set according to the position of the sound source of the specified voice specified in relation to the specified frame.
Therefore, according to the present invention, it is possible to superimpose and arrange a text corresponding to a sound extracted from a movie on a frame extracted from the movie in a manner having a high appeal.

It is explanatory drawing of the arrangement | positioning mode of a superimposed text. (Example) It is explanatory drawing of the arrangement | positioning mode of a superimposition text. (Example) FIG. 3 is an explanatory diagram of a network configuration example. (Example) FIG. 3 is an explanatory diagram of a hardware configuration example of a user device. (Example) FIG. 3 is an explanatory diagram of a hardware configuration example of a server device. (Example) FIG. 2 is an explanatory diagram of a functional configuration example of an image generation system. (Example) FIG. 9 is an explanatory diagram of an image generation procedure. (Example) FIG. 9 is an explanatory diagram of an example of items of management data. (Example) FIG. 14 is an explanatory diagram of a display example of an editing screen A. (Example) FIG. 9 is an explanatory diagram of an example of items of edit data. (Example) FIG. 9 is an explanatory diagram of a display example of an edit screen B. (Example) FIG. 9 is an explanatory diagram of a display example of a thumbnail image. (Example) FIG. 9 is an explanatory diagram of a display example of a thumbnail image. (Example) It is explanatory drawing of the arrangement | positioning mode of a superimposed text. (Modification) It is explanatory drawing of the arrangement | positioning mode of a superimposed text. (Modification)

[1. Embodiment]
[1-1. Overview]
The present embodiment relates to an information processing system that extracts a frame and a sound from a movie, superimposes a text corresponding to the sound on the frame, and outputs the text.
In the present embodiment, the text corresponding to the sound extracted from the movie is superimposed on the frame extracted from the movie in a highly appealing manner. Is set in accordance with the position of.

[1-2. Information processing device]
An information processing apparatus that constitutes the information processing system according to the present embodiment includes a specification unit that specifies a position of a sound source of a specified sound included in the specified range in relation to a specified frame included in the specified range of the movie; Setting means for setting the arrangement mode of the superimposed text superimposed on the specified frame corresponding to the specified voice, taking into account the position of the sound source specified by the specifying means.

[2. Example]
[2-1. Overview]
The present embodiment relates to an image generation system that provides an image generation service that extracts a frame and a sound from a movie, lays out a plurality of images in which text corresponding to the sound is superimposed on the frame, and generates a synthesized thumbnail image. .
In the image generation service, frames and audio extracted from a movie are specified by a user using the image generation service.

In the image generation service, the text corresponding to the audio extracted from the movie is superimposed on the frame extracted from the movie in a highly appealing manner. This is set according to the relationship between the position of the sound source and the display range of the frame.
Further, in the image generation system, the position of the sound source is specified according to the position specified by a user operation that serves both as the position specification and the region specification, and the arrangement region of the superimposed text is set according to the region specified by the user operation. Is done.

With reference to FIG. 1, an example of an arrangement mode of the superimposed text will be described. In FIG. 1, it is assumed that the position of the sound source of the designated voice is inside the display range of the frame.
The subject 110 is included in the frame 110. When the user performs a user operation of drawing a closed graphic on the screen including the frame 110, a locus 130 is formed. At this time, the position of the sound source is specified according to the position of the starting point 131 of the trajectory 130, and the text arrangement area is set according to the position of the closed area 132 formed by the trajectory 130.
When the position of the sound source of the designated voice is inside the display range of the frame, the superimposed text is placed inside the balloon 140 arranged in the area corresponding to the closed area 132 in association with the position of the sound source corresponding to the position of the start point 131. Be placed.

With reference to FIG. 2, another example of an arrangement mode of the superimposed text will be described. In FIG. 2, it is assumed that the position of the sound source of the designated voice is outside the display range of the frame.
The frame 210 includes the subject 220. When the user performs a user operation of drawing a closed graphic on the screen including the frame 210, a locus 230 is formed. At this time, the position of the sound source is specified according to the position of the starting point 231 of the trajectory 230, and the placement area of the text is set according to the position of the closed area 232 formed by the trajectory 230.
When the position of the sound source of the designated voice is outside the display range of the frame, the superimposed text is placed in the region corresponding to the closed region 232 without being associated with the position of the sound source corresponding to the position of the starting point 231. Placed in

[2-2. Network configuration]
FIG. 3 illustrates a network configuration example of the system according to the embodiment.
The system according to the present embodiment includes a user terminal 10 used by a user and an image generation system 20 that provides an image generation service.
The image generation system 20 includes a user management server 21, a data processing server 22, a data management server 23, a file management server 24, and a storage 25.

The user terminal 10 and the user management server 21 can exchange data with each other through the communication network 30. The user management server 21 and the data processing server 22 can access data stored in the storage 25 via the data management server 23, respectively. Further, the user management server 21 and the data processing server 22 can access data stored in the storage 25 via the file management server 24, respectively.
The communication network 30 is at least one of existing networks (for example, the Internet (Internet), a mobile phone network, a wireless WAN (Wireless Wide Area Network), a wireless LAN (Wireless Local Area Network), an Ethernet (Ethernet) (registered trademark), and the like. Any).

[2-2-1. User terminal]
The user terminal 10 is a user device (computer) on which a predetermined Web browser program is installed.
In the system according to the present embodiment, as a user device, a general-purpose mobile device (for example, a mobile phone, a smartphone, a tablet terminal, a tablet PC (personal computer), a wearable device (wearable) that can install a Web browser program. device) or a general-purpose processing device (eg, a PC (personal computer)).

[2-2-2. Image generation system]
The image generation system 20 includes a user management server 21, a data processing server 22, a data management server 23, a file management server 24, and a storage 25.
The user management server 21 is a server device (computer) in which a Web server program (also referred to as an HTTP daemon (HyperText Transfer Protocol Daemon)) is installed.
The user management server 21 reads necessary data from the storage 25 via the data management server 23 in response to a request (request) from the user terminal 10 and provides the data to the user terminal 10 (response). Further, in response to a request from the user terminal 10, the user management server 21 writes data acquired from the user terminal 10 to the storage 25 via the data management server 23, and provides a processing result to the user terminal 10. (Response).
Note that a server system may be configured by linking a plurality of server devices, and the functions of the user management server 21 may be shared or the load on the user management server 21 may be distributed.

The data processing server 22 is a server device (computer) in which an application program is installed.
The data processing server 22 reads necessary data from the storage 25 via the data management server 23, performs calculation and processing on the data, and writes the calculated and processed data to the storage 25 via the data management server 23. Further, the data processing server 22 reads necessary data from the storage 25 via the file management server 24, performs calculation and processing on the data, and writes the calculated and processed data to the storage 25 via the file management server 24. .
Note that a server system may be configured by linking a plurality of server devices, and the functions of the data processing server 22 may be shared or the load on the data processing server 22 may be distributed.

The data management server 23 is a server device (computer) in which a DB (Database) server program is installed. The data management server 23 constitutes a DBMS (Database Management System) together with the built-in or externally connectable storage 25.
The data management server 23 has, for example, a function of storing data acquired from a request source in response to a data storage request in the storage 25 and a function of responding to the request source with data extracted from the storage 25 in response to a data extraction request. And
Note that a server system may be configured by linking a plurality of server devices, and the functions of the data management server 23 may be shared or the load on the data management server 23 may be distributed.

The file management server 24 is a server device (computer) in which a file server program is installed.
The file management server 24 has, for example, a function of storing data obtained from a request source in response to a data storage request in the storage 25 and a function of responding to the request source with data extracted from the storage 25 in response to a data extraction request. And
Note that a server system may be configured by linking a plurality of server devices, and the functions of the file management server 24 may be shared or the load on the file management server 24 may be distributed.

The storage 25 is a storage device that stores management data and file data.
Note that a plurality of storage devices may be prepared and stored separately for each type of data stored in the storage 25. It is also possible to distribute the data stored in the storage 25 to a plurality of storage devices.

[2-3. Hardware configuration]
[2-3-1. Hardware configuration of user device]
FIG. 4 illustrates an example of a hardware configuration of the user device.
A typical user device includes an MPU (Micro-Processing Unit) 411 configuring a control processing unit, a RAM (Random Access Memory) 421 configuring a main storage unit, and a ROM (Read Only Memory) configuring an auxiliary storage unit. 422, an EEPROM (Electrically Erasable Programmable Read-Only Memory) 423, a touch panel display 431 forming an input unit and a display unit, a speaker 432 forming an audio output unit, and a NIC (Network Interface Controller) forming a communication control unit. 441 and a wireless LAN (Local Area Network) chip 442.

The RAM 421, the ROM 422, the EEPROM 423, the touch panel display 431, the speaker 432, the NIC 441, and the wireless LAN chip 442 are connected to the MPU 411 via a bus line.
The MPU 411 reads (1) the program stored in the ROM 422 or the EEPROM 423 onto the RAM 421, and (2) acquires data from at least one of the touch panel display 431, the EEPROM 423, the NIC 441, and the wireless LAN chip 442 according to the instructions of the program. (3) After calculating and processing the acquired data according to the procedure specified in the program, (4) calculating and processing the processed and processed data by at least the EEPROM 423, the touch panel display 431, the speaker 432, the NIC 441, and the wireless LAN chip 442. Provide to either.

[2-3-2. Hardware configuration of server device]
FIG. 5 illustrates an example of a hardware configuration of the server device.
A typical server device includes a control processing device 510 including an MPU and a ROM, a main storage device 520 including a RAM, an auxiliary storage device 530 including an HDD (Hard Disc Drive), and an input device 540 including a mouse and a keyboard. , An output device 550 including a display and a speaker, and a communication control device 560 including a network card (Network Interface Card).

The main storage device 520, the auxiliary storage device 530, the input device 540, the output device 550, and the communication control device 560 are connected to the control processing device 510 via a bus line.
The control processing device 510 (1) reads the program stored in the auxiliary storage device 530 into the main storage device 520, and (2) executes at least one of the input device 540, the auxiliary storage device 530, and the communication control device 560 according to the instruction of the program. Data is obtained from any of them, and (3) the obtained data is calculated and processed according to the procedure specified in the program, and (4) the processed and processed data is communicated with the auxiliary storage device 530 and the output device 550. It is provided to at least one of the control device 560.

[2-4. Functional configuration]
FIG. 6 shows a functional configuration example of the image generation system.
As illustrated in FIG. 6, the user management server 21 includes a receiving unit 611, a creating unit 612, and a providing unit 613. Further, the data processing server 22 includes an extraction unit 621, a conversion unit 622, a designation unit 623, a specification unit 624, a setting unit 625, and a generation unit 626.

The function of the user management server 21 is realized by installing an OS (Operating System) for the server device and a Web server program running on the OS on the server device.
The function of the data processing server 22 is realized by installing an OS for a server device and an application program running on the OS on the server device.
The program to be installed in the server device is recorded on various recording media (eg, CD (Compact Disc), DVD (Digital Versatile Disk), MO disk (Magneto-Optical disk), flash memory (flash memory), etc.). It may be distributed in a state of being read and read into the server device from the recording medium, or may be supplied to the server device by being superimposed on a carrier wave via a communication network.

The receiving unit 611 receives a request from the user terminal 10.
The creating unit 612 creates a Web page according to the request (request) received by the receiving unit 611.
The providing unit 613 provides the Web page created by the creating unit 612 to the user terminal 10 (response).

The extraction unit 621 extracts frames and audio from the designated movie. The extracted frames and audio are stored in the file management server 24.
The conversion unit 622 performs a voice recognition process on the voice extracted by the extraction unit 621 to generate a converted text. If effective speech recognition is not possible, a supplementary text indicating that the speech is unrecognizable is generated. The generated text is stored in the data management server 23.

The specifying unit 623 presents to the user a candidate list including a plurality of candidate ranges respectively associated with a plurality of candidate texts (converted text or supplementary text) respectively corresponding to a group of sounds extracted from the specified movie by the extracting unit 621. Then, one or more ranges are selected, and the selected ranges are respectively designated as designated ranges.
Also, the sound included in the specified range is specified as the specified sound, and any of the frames included in the specified range is specified as the specified frame. The designation of the designated frame may be performed automatically or in accordance with the designation of the user.

The specifying unit 624 specifies the position of the sound source of the specified sound included in the specified range in relation to the specified frame included in the specified range, according to the position of the start point of the user operation that serves as both the position specification and the area specification. I do. In this embodiment, the position of the sound source is specified for each specified range.
The setting unit 625 sets an arrangement mode such that the superimposed text is arranged in association with the sound source when the position of the sound source specified by the specifying unit 624 is inside the display range of the designated frame. The arrangement mode is set so that the superimposed text is arranged without being associated with the sound source when the text is outside the display range. In any case, the arrangement region of the superimposed text is set in accordance with the position of the closed region formed by the closed figure drawn by the user operation which is both position designation and region designation.
The generation unit 626 generates a thumbnail image. The generated thumbnail image is stored in the storage 25 via the file management server 24, and is made public on the Web server.

[2-5. Image generation procedure]
FIG. 7 illustrates an image generation procedure. The image generation system 20 generates a thumbnail image according to the following procedure.
[S702]
The user management server 21 acquires the movie designation data from the user terminal 10. The movie designation data is a file path for specifying the designated movie file when the designated movie file is stored in the auxiliary storage unit of the user terminal 10, and the file path when the designated movie file is managed by another server device. It is a URL (Uniform Resource Locator) that specifies a specified movie file.

[S704]
The user management server 21 or the data processing server 22 acquires the specified movie file. The designated movie file is obtained from the user terminal 10 or another server device.
The acquired designated movie file is stored in the storage 25 via the file management server 24. Also, movie management information for managing the obtained designated movie file is stored in the storage 25 via the data management server 23.

FIG. 8A shows an example of items of the movie management information.
As illustrated in FIG. 8A, the movie management information includes a key item “movie ID”, a “file path” indicating a storage location of the designated movie file in the storage 25, and a thumbnail image based on the designated movie. The information includes a “user ID” that specifies the user who has requested the generation, a “reception time” that can specify a time at which a request to generate a thumbnail image based on the specified movie has been received, and a “reproduction time” of the specified movie.

[S706]
The data processing server 22 performs a voice recognition process on the voice data extracted from the designated movie to generate a converted text. If effective speech recognition is not possible, a supplementary text indicating that the speech is unrecognizable is generated. Text management information for managing the generated text is stored in the data management server 23.

FIG. 8B shows an example of items of the text management information.
As illustrated in FIG. 8B, the text management information includes “movie ID” as a key item, “head position” at which the start of the range can be specified, and “end position” at which the end of the range can be specified. , "Converted text or supplementary text" which is a speech recognition result for speech included in the range.

[S708]
The data processing server 22 extracts a temporary frame from the designated movie. Specifically, for each of a plurality of candidate ranges respectively associated with a plurality of candidate texts (converted text or supplementary text) respectively corresponding to a group of sounds extracted from the designated movie, one of the candidate ranges included in the candidate range Are extracted as temporary frames.

[S710]
The user management server 21 presents the editing screen A to the user terminal 10.
FIG. 9 is a display example of the editing screen A.
As illustrated in FIG. 9, the editing screen A900 has a plurality of candidate ranges 920 (920a to 920f) arranged in association with the timeline 910 of the designated movie. A text box 922 (922a to 922f), a play button 924 (924a to 924f), a temporary frame 926 (926a to 926f), and a check box 928 (928a to 928f) are displayed in association with the plurality of candidate ranges 920, respectively. You.
In the editing screen A900, the timeline 910 corresponds to the “reproduction time” of the movie management information (FIG. 8A).

The candidate range 920 corresponds to a range from the “head position” to the “end position” of the text management information (FIG. 8B). The text displayed in the text box 922 corresponds to “converted text or supplementary text” in the text management information (FIG. 8B).
When the play button 924 is tapped, the user terminal 10 requests the user management server 21 to reproduce the audio in the candidate range from the “start position” to the “end position” of the text management information (FIG. 8B).
The user edits the text in the text box 922 as necessary while reproducing the sound as necessary, and checks the check boxes 928 corresponding to one or more (four in the display example of FIG. 9) candidate ranges. When the user inserts and taps the button 930, the user terminal 10 transmits the edit data A to the user management server 21.

[S712]
The user management server 21 acquires the edited data A from the user terminal 10. The acquired edited data A is stored in the storage 25 via the data management server 23.
FIG. 10A shows an example of items of the edit data A.
As illustrated in FIG. 10A, the edit data A includes a “movie ID”, a “head position” that can specify the head of the candidate range, and a “tail position” that can specify the end of the candidate range. , An “edit text” corresponding to the candidate range, and a “selection flag” that is significant when the candidate range is selected.

[S714]
The user management server 21 presents the editing screen B to the user terminal 10.
FIG. 11 is a display example of the editing screen B.
As illustrated in FIG. 11, the editing screen B1100 includes an area 1110, an area 1120, an area 1130, and a button 1140.

In the area 1110, a temporary frame 1112 (1112a to 1112d), an edited text 1114 (1114a to 1114d), and an area 1116 (1116a to 1116d) are displayed for each selected range.
When the arrow displayed above the temporary frame 1112 is tapped, the frame is updated to another frame immediately before or before the temporary frame. Similarly, when an arrow displayed below the temporary frame 1112 is tapped, the frame is updated to another frame immediately after the temporary frame or later.
When a user operation for drawing a closed graphic is performed in the area 1116 including the temporary frame 1112 as described with reference to FIGS. 1 and 2, a trajectory is formed.

In the area 1120, the layout of the thumbnail image is selected. In the display example of FIG. 11, a selection is made between two rows and two columns or four rows and one column.
In the area 1130, the output mode of the thumbnail image is selected. In the display example of FIG. 11, the user is allowed to select whether to use JPEG (Joint Photographic Experts Group) format Web display data or PDF (Portable Document Format) format print data.
When the user selects a frame as needed, performs a user operation on the selected frame, selects a layout and an output mode, and taps a button 1140, the user terminal 10 edits the data B and selection information (layout selection). Information and output format selection information) to the user management server 21.

[S716]
The user management server 21 acquires the edited data B from the user terminal 10. The acquired edited data B is stored in the storage 25 via the data management server 23.
FIG. 10B shows an example of items of the edit data B.
As illustrated in FIG. 10B, the edit data B includes a “movie ID”, a “head position” that can specify the head of the candidate range, and a “tail position” that can specify the end of the candidate range. , The “selected frame” included in the candidate range, and “trajectory information” corresponding to the candidate range.

[S718]
The data processing server 22 generates a thumbnail image. The generated thumbnail image is stored in the storage 25 via the file management server 24, and is made public on the Web server.
Specifically, the data processing server 22 generates a thumbnail image in the following procedure.
In the editing data A (FIG. 10A), the range in which the "selection flag" is significant is specified as the specified range.
Specify the "edit text" of the edit data A (FIG. 10A) corresponding to the specified range as the superimposed text.
Specify the “selected frame” of the edit data B (FIG. 10B) corresponding to the specified range as the specified frame.

In accordance with the position of the starting point of the trajectory specified from the “trajectory information” of the editing data B (FIG. 10B) corresponding to the specified range, the designated voice included in the specified range in relation to the specified frame Identify the location of the sound source. When the position of the sound source is inside the display range of the specified frame, the arrangement mode is set so that the superimposed text is arranged in association with the sound source. When the position of the sound source is outside the display range, the superimposed text is The arrangement mode is set so as to be arranged without being associated with the sound source.
A superimposed text arrangement area is set in accordance with the position of the closed area formed by the trajectory specified from the "trajectory information" of the edit data B (FIG. 10B) corresponding to the specified range.
Generate an image in which the superimposed text is superimposed and arranged in the layout mode set in the set layout area in the designated frame.
The images for each of the designated ranges generated in the above procedure are arranged in the layout specified by the layout selection information, and are output in the format specified by the output form selection information.

[S720]
The user management server 21 provides the user terminal 10 with the thumbnail image.
FIG. 12 is a display example of a thumbnail image. The thumbnail image 1200 is data generated when the layout of two rows and two columns and the data for Web display are selected on the editing screen B1100.
The superimposed text is superimposed on each of the designated frames constituting the thumbnail image 1200. A hyperlink for playing a movie in a corresponding designated range may be set in each designated frame.

FIG. 13 is a display example of a thumbnail image. The thumbnail image 1300 is data generated when layout and print data of four rows and one column are selected on the editing screen B1100.
The superimposed text is superimposed on each of the designated frames constituting the thumbnail image 1300. A two-dimensional code obtained by converting a URL for reproducing a movie in a corresponding designated range may be arranged at a corner of each designated frame.

[2-6. Effect of the system of the embodiment]
An SNS (Social Networking Service) is often used as a means for informing other users of some information. The SNS has a high affinity with static information (for example, text, still images, etc.) due to the property used for spreading information.
Even when a new movie (moving image) can be viewed on the Internet, it is possible to use the SNS as a means for notifying the fact. However, since viewing a movie (moving image) takes a considerable amount of time, it is necessary to post the content in a static manner after viewing it to introduce it on the SNS.

The system according to the present embodiment provides a superimposed text to be superimposed on a specified frame included in a specified range of a specified movie, and a superimposed text to the sound source when the position of the sound source of the specified sound is inside the display range of the specified frame. When the position of the sound source of the designated voice is outside the display range of the designated frame, the superimposed text is placed without being associated with the sound source. The position of the sound source is specified according to the position of the end point (start point) of the user operation for drawing the closed figure. Further, the arrangement region of the superimposed text is set according to the position of the closed region formed by the closed figure.
These processes are performed for each designated range, and a thumbnail image in which a plurality of designated frames in which texts are respectively superimposed and laid out is finally output.
Therefore, according to the system of the embodiment, the text corresponding to the designated sound extracted from the movie can be superimposed on the designated frame extracted from the movie in a more appealing manner by a simpler operation. become.

[3. Modification]
[3-1. Modified example of data linkage]
In the above embodiment, a configuration is adopted in which the edit data A and the edit data B are obtained by synchronous communication. For transmitting data, for example, a POST method of the HTTP protocol is used.
On the other hand, a configuration in which the item of the edit data A and the item of the edit data B are sequentially acquired by asynchronous communication may be adopted. For data transmission, for example, an XMLHttpRequest object may be used.

[3-2. Modification of processing subject]
In the above embodiment, the configuration is adopted in which the image generation system 20 (particularly, the data processing server 22) executes the data processing related to the generation of the thumbnail image. The user terminal 10 has a role corresponding to an input / output device of the image generation system 20.
On the other hand, a configuration in which the user terminal 10 executes at least a part of the data processing regarding the generation of the thumbnail image may be adopted. For example, if the user terminal 10 such as a PC executes the voice recognition processing, the processing load on the image generation system 20 (data processing server 22) can be reduced.

[3-3. Modification of judgment subject]
The above embodiment employs a configuration in which the designation of the designated range, the designated frame, the position of the sound source, the designation of the arrangement area of the superimposed text, and the like are performed based on the operation of the user.
On the other hand, a configuration may be adopted in which the image generation system 20 (for example, the data processing server 22) performs at least part of these processes without being based on a user operation. For example, it may be specified and specified at random based on a pseudo random number, or may be specified and specified based on a predetermined condition.

[3-4. Modification of user operation]
In the above-described embodiment, the position of the sound source is specified according to the position specified by the user operation that serves both as the position specification and the region specification, and the arrangement region of the superimposed text is set according to the region specified by the user operation. . In the above embodiment, a configuration is adopted in which the position of the sound source is specified according to the position of the starting point of the trajectory of the user operation for drawing the closed figure on the screen, and the arrangement area of the text is set according to the position of the closed region formed by the trajectory. Have been.
On the other hand, a configuration is adopted in which the position of the sound source is specified according to the position of the start point of the trajectory of the user operation that draws a line on the screen, and the text arrangement area is set in a fixed area including the position of the end point of the trajectory. Is also good. It is preferable to set the text placement area so that the text does not overlap with another subject.

With reference to FIG. 14, an example of an arrangement mode of the superimposed text will be described. In FIG. 14, it is assumed that the position of the sound source of the designated sound is inside the display range of the frame.
The frame 1410 includes a subject 1420. When a user performs a user operation of drawing a line on the screen including the frame 1410, a trajectory 1430 is formed. At this time, the position of the sound source is specified according to the position of the start point 1431 of the trajectory 1430, and the text arrangement area is set according to the position of the end point 1432 of the trajectory 1430.
When the position of the sound source of the designated voice is inside the display range of the frame, the superimposed text is in the area corresponding to the position of the end point 1432 inside the speech balloon 1440 arranged in association with the position of the sound source corresponding to the position of the start point 1431. Placed in

With reference to FIG. 15, another example of the arrangement mode of the superimposed text will be described. In FIG. 15, it is assumed that the position of the sound source of the designated voice is outside the display range of the frame.
The frame 1510 includes a subject 1520. When a user operation for drawing a line on the screen including the frame 1510 is performed, a trajectory 1530 is formed. At this time, the position of the sound source is specified according to the position of the start point 1531 of the trajectory 1530, and the text placement area is set according to the position of the end point 1532 of the trajectory 1530.
When the position of the sound source of the designated sound is outside the display range of the frame, the superimposed text is placed in an area corresponding to the position of the end point 1532 without being associated with the position of the sound source corresponding to the position of the start point 1531. Located inside.

10 User terminal 20 Image generation system 21 User management server 22 Data processing server (an example of an information processing device)
23 data management server 24 file management server 25 storage 30 communication network

Claims (12)

Specifying means for specifying, based on a user operation, a position of a sound source of a sound included in the specified range in relation to a frame included in the specified range specified by the user in the movie;
Setting means for setting an arrangement mode of superimposed text superimposed on the frame corresponding to the voice based on a position of a sound source specified by the specifying means and a trajectory drawn by a user operation;
An information processing apparatus comprising: The user operation is an operation of drawing a closed figure,
The specifying means specifies a position of the sound source according to a position of an end point of the trajectory,
The setting means sets an arrangement area of the superimposed text according to a position of a closed area formed by the closed graphic,
The information processing device according to claim 1. Designation means for designating, as a designated range, a range selected by a user from a candidate list including at least a plurality of candidate ranges of the movie corresponding to sounds extracted from the movie,
Specifying means for specifying a position of a sound source of the sound included in the specified range in relation to a frame included in the specified range;
Setting means for setting an arrangement mode of superimposed text superimposed on the frame corresponding to the voice based on a position of a sound source specified by the specifying means;
An information processing apparatus comprising: The setting unit sets the arrangement mode according to the relationship between the position of the sound source specified by the specifying unit and the display range of the frame.
The information processing device according to claim 3. The setting means sets the arrangement mode such that the superimposed text is arranged in association with the sound source when the position of the sound source is inside the display range, and the position of the sound source is outside the display range. In the case of setting the arrangement mode such that the superimposed text is arranged without being associated with the sound source,
The information processing device according to claim 4. Wherein the specifying unit specifies the position of the sound source in accordance with the position specified by the user operations that serves both as a position designation and the area designation,
The setting unit sets the arrangement region of the superimposed text in accordance with the area designated by the user operation,
The information processing apparatus according to claim 4. The specifying means specifies, for each specified range specified by the specifying means, a position of a sound source of a sound included in the specified range,
The setting means includes, for each specified range specified by the specifying means, an arrangement mode of the superimposed text to be superimposed on a frame included in the specified range included in the specified range specified by the specifying means. Settings based on the position of the sound source of the audio
The information processing apparatus according to claim 3. The designating means presents to the user the candidate list including a plurality of candidate ranges respectively associated with a plurality of candidate texts respectively corresponding to a group of sounds included in the movie, and specifies the designated number range to the user. Let me choose
The information processing device according to claim 7. A specifying step of specifying, based on a user operation, a position of a sound source of a sound included in the specified range in relation to a frame included in the specified range specified by the user in the movie;
A setting step of setting an arrangement mode of superimposed text superimposed on the frame corresponding to the voice based on a position of a sound source specified in the specifying step and a trajectory drawn by the user operation;
An information processing method, including: A designation step of designating a range selected by a user from a candidate list including at least a plurality of candidate ranges of the movie corresponding to sounds extracted from the movie as a designated range,
A specifying step of specifying a position of a sound source of the sound included in the specified range in relation to a frame included in the specified range;
A setting step of setting the arrangement mode of the superimposed text that is superimposed on the frame corresponding to the voice, taking into account the position of the sound source specified in the specifying step;
An information processing method, including: A specifying function of specifying, based on a user operation, a position of a sound source of a sound included in the specified range in relation to a frame included in the specified range specified by the user in the movie;
A setting function of setting a layout mode of superimposed text superimposed on the frame corresponding to the voice based on a position of a sound source specified by the specific function and a trajectory drawn by the user operation;
Control program that causes a computer of an information processing apparatus to implement the above. A designation function of designating a range selected by a user from a candidate list including at least a plurality of candidate ranges of the movie corresponding to sounds extracted from the movie as a designated range,
A specifying function of specifying a position of a sound source of the sound included in the specified range in relation to a frame included in the specified range;
A setting function of setting the arrangement mode of the superimposed text superimposed on the frame corresponding to the voice, taking into account the position of the sound source specified by the specific function;
Control program that causes a computer of an information processing apparatus to implement the above.

Images