JP2012039391A - Reproducing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily and simply achieve processing of displaying even images reflecting sound contents included in sound data while maintaining the composition and configuration of original images in the middle of reproducing original image data.SOLUTION: A sound output unit 19 outputs sound expressed by sound data by reproducing the sound data. A sound content recognition unit 52 recognizes the sound contents included in the sound data by analyzing the sound data of the reproduction object of the sound output unit 19. A display unit 18 displays images expressed by the image data as original images by reproducing the image data. A sound content reflection unit 54 executes processing of displaying images reflecting the sound contents recognized by the sound content recognition unit 52 while maintaining the configuration and composition of the original images in the middle of reproducing the image data by the display unit 18, as sound content reflection processing.

Description

  The present invention relates to a playback apparatus and method, and a program, and particularly, while reproducing original image data, reflects the audio content included in the audio data while maintaining the composition and configuration of the original image. The present invention relates to a technique for enabling easy and easy processing for displaying an image.

In recent years, various types of image processing have been performed on image data for the purpose of enhancing the effect of viewing images.
For example, a reproducing apparatus represented by a digital photo frame can reproduce audio data together with image data.
Therefore, Patent Document 1 discloses a technique for converting audio data synchronized with image data into character data and associating the character data with the image data.
Patent Document 2 discloses a technique for operating a character displayed on a screen in accordance with the tempo of music output from a music playback device, that is, a new image including a character that changes in accordance with the tempo of music as a subject. A technique for creating and reproducing data is disclosed.

JP 2007-101945 A JP 2007-160065 A

However, the technique disclosed in Patent Document 1 requires complicated and time-consuming processing such as analyzing voice data to create character data and associating the created character data with image data.
Therefore, while executing the process in parallel during the reproduction of the image data, the process result is sequentially reflected in the reproduction content of the image data, that is, the image data reflecting the audio content included in the audio data. It is very difficult to realize the reproduction. In other words, the audio content included in the audio data is reflected unless the processing is executed in advance before the image data is reproduced and the text data is included in the metadata of the image data. It is very difficult to reproduce the image data.
Further, in the technique described in Patent Document 2, as described above, in order to create new image data including a character that changes in accordance with the tempo of music in a subject, a digital photo frame for the purpose of viewing an original image or the like It is not suitable to apply to.
Therefore, during the reproduction of the original image data, it is possible to easily and easily realize the process of displaying the image reflecting the audio content included in the audio data while maintaining the composition and configuration of the original image. It is a situation that is required.

  The present invention has been made in view of such a situation, and while reproducing the original image data, the audio content included in the audio data is maintained while maintaining the composition and configuration of the original image. It is an object to easily and easily realize a process of displaying a reflected image.

According to one aspect of the invention,
Sound reproducing means for outputting sound represented by the sound data by playing the sound data;
Audio content recognition means for recognizing audio content included in the audio data by analyzing the audio data to be reproduced by the audio reproduction means;
Image reproducing means for displaying an image represented by the image data as an original image by reproducing the image data;
While the image data is being reproduced by the image reproduction means, an image reflecting the audio content recognized by the audio content recognition means is displayed while maintaining the configuration and composition of the original image. Voice content reflecting means for executing processing as voice content reflecting processing;
A playback device is provided.

  According to another aspect of the present invention, there is provided each of a method and program corresponding to one aspect of the present invention described above.

  According to the present invention, during the reproduction of original image data, the process of displaying an image reflecting the audio content included in the audio data while maintaining the composition and configuration of the original image is easy and easy. Can be realized.

It is a block diagram which shows the hardware constitutions of the reproducing | regenerating apparatus which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional structure of the reproducing | regenerating apparatus for performing an audio | voice reproduction | regeneration process and an image reproduction process. It is a flowchart explaining the flow of the audio | voice reproduction | regeneration processing which CPU of FIG. 2 performs. 3 is a timing chart showing an example of audio data to be reproduced from which audio data for a processing target period is acquired by the reproduction audio acquisition unit of FIG. 2. An example of the result of performing FFT processing on audio data to be reproduced is shown. 3 is a flowchart illustrating a flow of image reproduction processing executed by a CPU in FIG. 2. An example of a decoration-added original image is shown. It is a flowchart explaining the flow of the image reproduction process which CPU which concerns on 2nd Embodiment of this invention performs. An example of an original image corresponding to audio content is shown.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a hardware configuration of a playback apparatus 1 according to an embodiment of the present invention. The playback device 1 can be constituted by a digital photo frame, for example.

  The playback apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an RTC (Real Time Clock) 14, a bus 15, and an input / output interface 16. , An operation unit 17, a display unit 18, an audio output unit 19, a storage unit 20, a communication unit 21, and a drive 22.

The CPU 11 executes various processes according to programs recorded in the ROM 12. Alternatively, the CPU 11 executes various processes according to a program loaded from the storage unit 20 to the RAM 13.
The RAM 13 also stores data necessary for the CPU 11 to execute various processes as appropriate.

  For example, in the present embodiment, programs for realizing the functions of the reproduction audio acquisition unit 51 to the audio content reflection unit 54 shown in FIG. 2 to be described later are stored in the ROM 12 and the storage unit 20. Therefore, when the CPU 11 executes processing according to these programs, each function of the reproduced sound acquisition unit 51 to the sound content reflection unit 54 of FIG. 2 to be described later can be realized.

  The RTC 14 counts time and outputs, for example, the current time to the CPU 11. That is, the CPU 11 acquires the current time from the RTC 14 every predetermined time, and controls the acquisition timing of the audio data and the image data based on the current time.

  The CPU 11, ROM 12, RAM 13, and RTC 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An operation unit 17, a display unit 18, an audio output unit 19, a storage unit 20, and a communication unit 21 are connected to the input / output interface 16.

The operation unit 17 includes various buttons and the like, and accepts user instruction operations.
The display unit 18 includes a display 62 shown in FIG. 2 described later, and displays an image represented by predetermined image data on the display 62 under the control of the CPU 11.
The audio output unit 19 includes a speaker 72 of FIG. 2 to be described later, and outputs sound expressed by predetermined audio data from the speaker 72 under the control of the CPU 11.

Note that displaying the image represented by the image data on the display 62 based on the predetermined image data is hereinafter referred to as “reproducing the image data”. Moreover, outputting the sound expressed by the sound data from the speaker 72 based on the predetermined sound data is hereinafter referred to as “reproducing the sound data”.
That is, under the control of the CPU 11, the display unit 18 reproduces image data, and the audio output unit 19 reproduces audio data.

  The storage unit 20 includes a DRAM (Dynamic Random Access Memory) or the like. The storage unit 20 also stores various data necessary for various audio processing and image processing, such as various flag values and threshold values. The storage unit 20 also includes an audio storage unit 31 and an image storage unit 32 as one area.

The sound storage unit 31 stores a plurality of reproducible sound data, for example, sound data such as music.
The format of the audio data stored in the audio storage unit 31 is not particularly limited. For example, in the present embodiment, audio data encoded according to a predetermined format and compressed as necessary is stored in the audio storage unit 31. As the predetermined format, for example, a WAVE format, an MP3 (Moving Picture Experts Group Audio Layer-3) format, an AAC (Advanced Audio Coding) format, or the like can be adopted.

The image storage unit 32 stores a plurality of reproducible image data, for example, various image data such as a photograph taken by a digital camera, an image read by a scanner, and an image processed or created by a personal computer. .
The format of the image data stored in the image storage unit 32 is not particularly limited. For example, in the present embodiment, image data of a still image that has been compression-encoded according to a predetermined format is stored in the image storage unit 32. For example, JPEG (Joint Photographic Experts Group) can be adopted as the predetermined format. Alternatively, GIF (Graphics Interchange Format), PNG (Portable Network Graphics), TIFF (Tagged Image File Format), or the like may be adopted as the predetermined format. The image data stored in the image storage unit 32 is hereinafter referred to as “original image data” in order to distinguish it from image data processed by the CPU 11 (more specifically, the audio content reflecting unit 54) described later. . An image displayed on the display 62 when the original image data is reproduced is hereinafter referred to as “original image”.

  The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  A drive 22 is connected to the input / output interface 16 as necessary, and a removable medium 41 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached. The program read from the removable medium 41 by the drive 22 is installed in the storage unit 20 as necessary. The removable media 41 can also store various data such as image data and audio data stored in the storage unit 20 in the same manner as the storage unit 20.

  The playback apparatus 1 having such a configuration can execute the following series of processes.

That is, the playback device 1 acquires audio data to be played out of a plurality of audio data stored in the audio storage unit 31.
The playback apparatus 1 recognizes the content of sound included in the sound data (hereinafter, simply referred to as “sound content”) by analyzing the sound data to be played back.
The playback device 1 plays back audio data to be played back.
Such a series of processing is hereinafter referred to as “audio reproduction processing”.

Further, when reproducing the image data to be reproduced among the plurality of image data stored in the image storage unit 32, the reproducing apparatus 1 uses the image data to be reproduced as original image data as follows. The image is displayed on the display 62. That is, the playback device 1 displays on the display 62 an image reflecting the audio content recognized by the above-described audio playback processing while maintaining the configuration and composition of the original image.
Such a series of processing is hereinafter referred to as “image reproduction processing”.

  In the present embodiment, as a part of the image reproduction process, the reproduction apparatus 1 performs image processing for adding a decoration image corresponding to the audio content recognized by the audio reproduction process described above to the original image. It is applied to image data. As a result, an image (decorative image in the present embodiment) reflecting the audio content recognized by the audio reproduction process described above is displayed on the display 62 while maintaining the configuration and composition of the original image.

FIG. 2 is a functional block diagram showing a functional configuration of the playback apparatus 1 for executing such audio playback processing and image playback processing.
2, only the CPU 11, the RTC 14, the operation unit 17, the display unit 18, the audio output unit 19, and the storage unit 20 are illustrated in the configuration of the playback device 1 in FIG.
The CPU 11 includes a reproduction audio acquisition unit 51, an audio content recognition unit 52, a reproduction image acquisition unit 53, and an audio content reflection unit 54.
The display unit 18 includes a display control unit 61 and a display 62.
The audio output unit 19 includes an audio output control unit 71 and a speaker 72.

The reproduction sound acquisition unit 51 acquires sound data to be reproduced from the sound storage unit 31 when the operation is started by a user instruction operation on the operation unit 17. Here, it is assumed that the audio data to be reproduced is determined in advance by a user instruction operation on the operation unit 17.
Although details will be described later with reference to the flowchart of FIG. 3, the audio data to be reproduced is audio data from the start to the end of one piece of music, and has a large capacity for handling in one process. Therefore, the audio data to be reproduced is divided into audio data for each predetermined period, and read out at predetermined time intervals. In other words, the CPU 11 including the reproduction audio acquisition unit 51 executes reproduction audio processing in units of audio data for the predetermined period. Hereinafter, one unit of audio data to be subjected to the reproduction audio processing is referred to as “audio data for the processing target period”.
In the present embodiment, the time interval at which the audio data in the processing target period is read out is set based on the time interval at which the current time is transmitted from the RTC 14 to the CPU 11, and is specifically a 23 ms interval. Shall.
That is, in the present embodiment, the reproduction audio acquisition unit 51 sequentially acquires audio data of the processing target period from the audio storage unit 31 at intervals of 23 ms in synchronization with the timing at which the current time is transmitted from the RTC 14. Sequentially supplied to the voice content recognition unit 52.
When the audio data in the processing target period is encoded and compressed as necessary, the reproduction audio acquisition unit 51 decompresses the audio data as necessary and decodes it. To do.

The audio content recognition unit 52 recognizes the audio content included in the audio data of the processing target period supplied from the reproduction audio acquisition unit 51.
The audio content recognized by the audio content recognition unit 52 is not particularly limited, and the method for recognizing the audio content is not particularly limited.
However, in the present embodiment, the audio content recognized by the audio content recognition unit 52 includes three types of audio content such as a male voice, a female voice, and a sound that does not include human (male and female) voices. It shall be one of them.
Although details will be described later, these three types of audio contents are all characterized by specific frequency bands. However, the audio data in the processing target period supplied from the reproduction audio acquisition unit 51 is audio data in the time domain. The time domain audio data refers to time-series sound data in which the horizontal axis is time, as shown in FIG. The sound data in the time domain is suitable for use in analyzing the time transition of sound pressure, but is unsuitable for use in analyzing frequency distribution. That is, it is difficult for the audio content recognition unit 42 to recognize the characteristics of a specific frequency band while maintaining the form of the audio data in the time domain.
Therefore, in the present embodiment, the audio content recognition unit 52 converts the audio data in the processing target period from the form of the time domain audio data to the form of the frequency domain. Here, the sound data in the frequency domain refers to sound data indicating a frequency distribution in which the horizontal axis is frequency as shown in FIG.
However, since “time domain” and “frequency domain” are modifiers related to “data”, the position of the dependency will be changed as appropriate.
A so-called Fourier transform process can be adopted as a technique for converting the form of the time domain audio data into the form of the frequency domain. More specifically, in this embodiment, FFT (Fast Fourier Transform) processing is adopted.
That is, the voice content recognition unit 52 performs FFT processing on the voice data in the processing target period, and uses the voice data in the frequency domain obtained as a result, as a voice content included in the voice data, Recognize female voices or sounds that do not contain human voices.
The recognition result of the voice content recognition unit 52 is supplied to the voice content reflection unit 54. On the other hand, the audio data in the processing target period is supplied to the audio output control unit 71 as audio data in the time domain.

The reproduction image acquisition unit 53 acquires the original image data to be reproduced from the image storage unit 32 when the operation is started by a user instruction operation on the operation unit 17. Here, it is assumed that a plurality of pieces of image data are determined in advance by a user instruction operation on the operation unit 17 as original image data to be reproduced. Furthermore, it is assumed that the reproduction order of the plurality of image data is also determined in advance by a user instruction operation on the operation unit 17.
Therefore, the reproduction image acquisition unit 53 acquires the image data in the next reproduction order from the image storage unit 32 at predetermined time intervals. Such image data in the next reproduction order is hereinafter referred to as “original image data to be reproduced next time”.
Further, in this embodiment, the time interval at which the original image data to be reproduced next time is read out is set based on the time interval at which the current time is transmitted from the RTC 14 to the CPU 11. Specifically, the time interval is 3 s. Suppose that
That is, in the present embodiment, the reproduction image acquisition unit 53 acquires the original image data to be reproduced next time from the image storage unit 32 every 3 s in synchronization with the timing at which the current time is transmitted from the RTC 14. To the audio content reflecting unit 54.
When the original image data to be reproduced next time is compressed and encoded, the reproduced image acquisition unit 53 decompresses and decodes the image data.

When reproducing the original image data to be reproduced next time, the audio content reflecting unit 54 reflects the audio content recognized by the audio content recognizing unit 52 while maintaining the configuration and composition of the original image. Is displayed on the display 62. Such processing of the audio content reflection unit 54 is hereinafter referred to as “audio content reflection processing”.
In the present embodiment, the audio content reflecting unit 54 performs, as the audio content reflecting process, the original image data to be reproduced next time supplied from the reproduction image acquiring unit 53 with respect to the original image. The image processing for adding the decoration image corresponding to the audio content recognized by the above is performed.
Thereby, image data of an image obtained by adding a decoration image to the original image (hereinafter referred to as “decoration addition original image”) is obtained and supplied from the audio content reflection unit 54 to the display control unit 61.

The display control unit 61 reproduces the image data supplied from the audio content reflection unit 54 under the control of the CPU 11. Thereby, in the present embodiment, the decoration-added original image is displayed on the display 62.
Here, the recognition result of the voice content recognition unit 52, that is, the voice content is supplied to the voice content reflection unit 54 at intervals of 23 ms as described above. On the other hand, the original image data to be reproduced next time is supplied to the audio content reflecting unit 54 at intervals of 3 s as described above.
Therefore, in the present embodiment, the audio content reflection unit 54 executes the audio content reflection process at intervals of 23 ms. That is, the data of the decoration-added original image is updated at intervals of 23 ms and supplied to the display control unit 61.
As a result, for the original image, the same image is displayed on the display 62 for 3 seconds (hence, the composition and the configuration are maintained for 3 seconds), and the decoration image corresponding to the audio content at intervals of 23 ms. Will be updated.

  Under the control of the CPU 11, the audio output control unit 71 reproduces audio data (time domain audio data) of the processing target period supplied from the audio content recognition unit 52. That is, in the present embodiment, as described above, the audio data in the processing target period is sequentially supplied from the audio content recognition unit 52 to the audio output control unit 71 at intervals of 23 ms. Therefore, the audio output control unit 71 outputs the audio corresponding to the audio data in the processing target period, that is, the audio for 23 ms of the audio content corresponding to the decoration image currently displayed on the display from the speaker 72.

Next, with reference to FIG. 3, an audio reproduction process among the processes of the reproduction apparatus 1 having the functional configuration of FIG. 2 will be described.
FIG. 3 is a flowchart for explaining the flow of the audio reproduction process.

For example, in the present embodiment, the audio reproduction processing is performed by the user operating the operation unit 17 to change the reproduction target audio data, a plurality of original image data to be reproduced, and the reproduction order of the image data. Start with the decision. In synchronization with the start of the audio reproduction process, an image reproduction process of FIG. 6 described later is also started.
Here, monaural audio data, which is 16-bit encoded audio data with a sampling frequency of 44.1 kHz, is included in a WAVE file without compression and stored in the reproduced audio acquisition unit 51. Shall.

In step S <b> 1, the reproduction audio acquisition unit 51 acquires audio data for a predetermined period from the read address among the audio data to be reproduced stored in the audio storage unit 31 as audio data for the processing target period.
Here, the read address is, in principle, the address next to the last address of the audio data in the previous processing target period (hereinafter referred to as the address of the audio storage unit 31 in which the audio data to be reproduced is stored). It is expressed as “the next address of the processing target period”). However, in the process of the first step S1 immediately after the start of the audio reproduction process and the process of step S1 after the process of step S4 described later is executed, the first part of the audio data to be reproduced is stored. Address (hereinafter simply referred to as “first”) is the read address.
More specifically, in this embodiment, 23 ms is adopted as the predetermined period. Audio data sampled at 44.1 kHz is employed. For this reason, the audio data in the processing target period is the number corresponding to 23 ms, that is, 1024 sample data. Accordingly, 1024 pieces of sample data are read out from the audio storage unit 31 in order from the read address, and are acquired by the reproduction audio acquisition unit 51 as audio data in the processing target period. That is, the size of the audio data in the processing target period is 2048 bytes corresponding to 1024 sample data. Accordingly, 2048 bytes of data from the read address is read from the voice storage unit 31 as voice data for the processing target period.

FIG. 4 is a timing chart showing an example of reproduction target audio data from which the reproduction audio acquisition unit 51 acquires audio data for the processing target period in the process of step S1.
In the timing chart of FIG. 4, the horizontal axis indicates time. The vertical axis indicates the sound pressure.
In FIG. 4, the scale value on the horizontal axis is given in accordance with the separation of the number of samples of audio data, and is not in units of seconds. The time (550 ms) corresponding to the collection of 24 audio data in the processing target period is about 1/8 of the scale interval. That is, data for a period of about (1/8) × (1/24) of the scale interval is acquired as audio data for the processing target period by one process of step S1.

  In step S2 of FIG. 3, the audio content recognition unit 52 determines whether or not the processing target period of the audio data acquired in the process of step S1 is the last period to be reproduced.

  If it is not the last period to be reproduced, it is determined as NO in step S2, and the process proceeds to step S3.

  In step S3, the voice content recognition unit 52 updates the read address to an address next to the processing target period.

On the other hand, if it is the last period to be reproduced, it is determined as YES in step S2, and the process proceeds to step S4.
In step S4, the voice content recognition unit 52 first updates the read address.
Thereby, in the next process of step S1, the audio data to be reproduced is read from the beginning. That is, the audio data to be reproduced is reproduced from the beginning. In this way, in this embodiment, repeated reproduction of audio data to be reproduced is realized.

  When the read address is updated by the process of step S3 or step S4, the process proceeds to step S5.

In step S5, the audio content recognition unit 52 performs FFT processing on the audio data in the processing target period.
Here, the result of the FFT processing in step S5, that is, the voice data in the frequency domain in the processing target period, is continuously several tens of times from the past, and is stored in a predetermined memory, for example, one area of the storage unit 20 (FIG. (Not shown).

In step S <b> 6, the audio content recognition unit 52 recognizes the audio content of the processing target period by comparing the FFT processing results (frequency domain audio data) of the processing target period and the past plural periods.
Specifically, based on the comparison in the frequency domain, the audio content of the audio data in the processing target period is a type of sound that does not include male voice, female voice, or human voice. Is determined.

  Further, the details of the processing of steps S5 and S6 by the audio content recognition unit 52 will be described below with reference to FIG.

FIG. 5A shows an example of the result of performing FFT processing on audio data corresponding to sound that does not include human voices among audio data to be reproduced. That is, FIG. 5A shows an example of frequency domain audio data corresponding to a sound that does not include a human voice.
FIG. 5B shows an example of the result of performing FFT processing on audio data corresponding to a sound including a male voice among the audio data to be reproduced. That is, FIG. 5B shows an example of audio data in the frequency domain corresponding to a sound including a male voice.

Comparing FIG. 5 (A) and FIG. 5 (B), in FIG. 5 (B), the intensity of the frequency component near 250 Hz is strong, whereas in FIG. 5 (A), the frequency around 250 Hz. It can be seen that the strength of the components is weak. This is consistent with the content that the male pitch (basic frequency) is said to be around 250 Hz.
Although not shown, it is also known that there is a difference between men and women, and it is said that the pitch of women is around 560 Hz, whereas the pitch of men is around 250 Hz mentioned above. ing.
Therefore, the speech content recognition unit 52 pays attention to the specific frequency band near 250 Hz and 560 Hz, compares the frequency domain data (FFT processing result) for the past several tens of times, and compares the intensity of the frequency component of the specific frequency band of interest. The voice content can be recognized based on the degree of change.
That is, the voice content recognition unit 52 can recognize that the voice content of the voice data in the processing target area is a male voice when the intensity change of the frequency component near 250 Hz is large.
When the intensity change of the frequency component near 560 Hz is large, the audio content recognition unit 52 can recognize that the audio content of the audio data in the processing target area is a female voice.
The voice content recognition unit 52 recognizes that the voice content of the voice data in the processing target area is a sound that does not include a human voice when the intensity change of any frequency component near 250 Hz or 560 Hz is small. can do.

Note that the speech content recognition method by the speech content recognition unit 52 is not particularly limited to the example in the previous paragraph, and may be arbitrary.
For example, the voice content recognition unit 52 first determines whether or not a human voice is included, and only when a human voice is included, determines whether the human voice is a male voice or a female voice. You may make it cut out. In this case, in determining whether or not the first human voice is included, the degree of intensity change of frequency components in a wide frequency band, for example, several tens to 2000 Hz can be used. That is, when the degree of intensity change is large, it is determined that a human voice is included, whereas when it is small, it is determined that a human voice is not included.
In addition, the elements used for the speech content recognition by the speech content recognition unit 52 are not only simple changes in the frequency component, but also, for example, between the holding time of the increased state and the time of the next increased state (in the dead state). Various elements such as (period) can be adopted.
Furthermore, in the case of recognizing the audio content of the audio data for one processing target period, the audio data used as a comparison target is not particularly limited. However, the number of audio data used for comparison is several hundreds of milliseconds, taking into account the interval of image display (3 s in the present embodiment), the time required for a series of processing from when image data is read to display, and the like. A number corresponding to the extent is suitable. Therefore, for example, when audio data for about 500 ms is adopted, when converted into audio data for the processing target period, the number of audio data is 24, which is consistent with the above example of several tens.

  The details of the processing in steps S5 and S6 by the voice content recognition unit 52 have been described above. When the processing of steps S5 and S6 is completed and the recognition result of the speech content recognition unit 52, that is, the speech content is supplied to the speech content reflection unit 54, the process proceeds to step S7.

In step S7, the audio output unit 19 reproduces the audio data in the processing target period.
That is, the audio output unit 19 outputs the audio corresponding to the audio data in the processing target period, that is, the audio of the audio content recognized in the process of step S6 from the speaker 72.

In step S8, the reproduction sound acquisition unit 51 determines whether or not there is an instruction to end the process.
The end instruction is not particularly limited, but in the present embodiment, it is assumed that an instruction to end the reproduction of images and sounds, which is performed by the user operating the operation unit 17, is adopted.

  In this case, when the instruction to end the reproduction is not given, it is determined as NO in Step S8, the process is returned to Step S1, and the subsequent processes are repeated. That is, the loop processing of steps S1 to S8 is repeatedly executed until an instruction to end reproduction is given. The process of recognizing the audio content and outputting the audio content of the audio content from the speaker 72 is repeatedly executed in units of audio data in the processing target period.

  Thereafter, when an instruction to end playback is given, it is determined YES in step S8, and the music playback process ends.

Heretofore, the audio reproduction process among the processes of the reproduction apparatus 1 of FIG. 2 has been described with reference to FIG.
Next, an image reproduction process among the processes of the reproduction apparatus 1 in FIG. 2 will be described with reference to FIG.
FIG. 6 is a flowchart for explaining the flow of the image reproduction process.

For example, in the present embodiment, in the present embodiment, when the user operates the operation unit 17 to instruct the reproduction of the audio data to be reproduced, a plurality of original image data to be reproduced, and the reproduction order of those image data. Start with the decision. That is, as described above, the image reproduction process of FIG. 6 is also started in synchronization with the start of the audio reproduction process of FIG.
Here, image data of a photograph taken with a digital camera or the like is so-called JPEG encoded data included in a JPEG format file (hereinafter referred to as “JPEG data”). It is assumed that it is determined as original image data. Then, it is assumed that the plurality of original image data are so-called slide-reproduced at intervals of 3 seconds.

  In step S21, the reproduction image acquisition unit 53 performs initial setting of the next reproduction target. That is, here, it is assumed that the original image data for which No. 1 is determined as the reproduction order is set as the next reproduction target image data.

In step S <b> 22, the reproduction image acquisition unit 53 acquires original image data to be reproduced next time among the plurality of original image data to be reproduced stored in the image storage unit 32.
Here, the original image data to be reproduced next time is JPEG data. Therefore, the reproduced image acquisition unit 53 performs decompression decoding processing on the JPEG data. Here, the form of the image data obtained as a result of the decompression decoding process is not particularly limited. For example, uncompressed RGB in accordance with the output size (for example, 1024 × 768, 1024 × 600, 800 × 600, 640 × 480) The image data of 24 bits per pixel, the image data of 16 bits per pixel of YUV422, the image data of 12 bits per pixel of YUV420, or the like can be employed.

In step S23, the audio content reflecting unit 54 acquires the audio content. Here, the audio content acquired by the audio content reflection unit 54 is the audio content of the audio data in the processing target period recognized by the audio content recognition unit 52 in step S6 of the audio reproduction process of FIG. .
In other words, the audio content reflection unit 54 acquires any type of a voice that does not include a male voice, a female voice, and a human voice as the audio content of the audio data in the processing target period.
More specifically, for example, in this embodiment, the audio content recognition unit 52 supplies the audio content reflection unit 54 with the recognition result of the audio content for the audio data in the processing target period as an identification code. That is, the voice content recognition unit 52 issues an identification code CB when recognizing a male voice, and issues an identification code CC when recognizing a female voice (possibly a child's voice). When a sound that does not contain human voice is recognized, an identification code CD is issued.
The audio content reflection unit 54 acquires any of these identification codes CB, CC, and CD as the audio content in the process of step S23.

  In step S24, the audio content reflection unit 54 determines whether or not the audio content acquired in the process of step S23 has changed from the previous time.

If the identification code acquired in the immediately previous step S23 is the same as the identification code acquired in the previous step S23, the audio content has not changed from the previous time, so step S24. In step S25, the process proceeds to step S25.
In step S <b> 25, the audio content reflecting unit 54 performs processing for adding the same decorative image as the previous image to the next reproduction target image as the audio content reflection processing on the next reproduction target image data.

On the other hand, when the identification code acquired in the previous step S23 is different from the identification code acquired in the previous step S23, the audio content has changed from the previous time. In step S24, it is determined as YES, and the process proceeds to step S26.
In step S <b> 26, the audio content reflection unit 54 performs a process for adding a decoration image corresponding to the audio content on the next reproduction target image as the audio content reflection process on the next reproduction target image data.

  In step S27, the display unit 18 reproduces the original image data to be reproduced next time to which the decoration image is added under the control of the CPU 11. Thereby, in this embodiment, the decoration addition original image as shown in FIG.

FIG. 7 shows an example of a decoration-added original image.
In the example of FIG. 7, an original image 81 including a cat as a subject is employed.
Further, the decoration image added in the process of step S25 or step S26 corresponds to a decoration image 91 corresponding to a male voice, a decoration image 92 corresponding to a female voice, and a sound not including a human voice. A decorative image 93 (decorative image 93 imitating a musical note) is employed.
For example, when the identification code CB is acquired as the voice content in the process of step S23, that is, when a male voice is recognized, the decoration image 91 is compared with the original image 81 in the next step S25 or step S26. The voice content reflection process to which is added is executed. As a result, in the process of the next step S27, the decoration-added original image 101 shown at the top on the right side of FIG.
Further, for example, when the identification code CC is acquired as the audio content in the process of step S23, that is, when a female voice is recognized, the decoration image is added to the original image 81 in the process of the next step S25 or step S26. A voice content reflection process to which 92 is added is executed. As a result, in the process of the next step S27, the decoration-added original image 102 shown at the center on the right side of FIG.
Further, for example, when the identification code CD is acquired as the audio content in the process of step S23, that is, when a sound that does not include a human voice is recognized, the original image 81 is displayed in the process of the next step S25 or step S26. On the other hand, the audio content reflecting process to which the decoration image 93 is added is executed. As a result, in the next step S27, the decoration-added original image 103 shown at the bottom right of FIG.
Note that the decoration-added original images 101 to 103 are merely examples. That is, the decoration image is not limited to the decoration images 91 to 93 in the example of FIG. In the present embodiment, it is sufficient that three types of audio contents can be presented to the user. Therefore, the types of decorative images need not be three, and may be two. Specifically, for example, in the case of a sound that does not include a human voice, even if the original image 81 without the decoration image 93 is displayed on the display 62 as it is, the user is informed that the sound does not include a human voice. Can be presented.

In step S28 of FIG. 6, the reproduction image acquisition unit 53 determines whether or not the display switching condition is satisfied.
Here, the display switching condition refers to a condition for switching image data to be reproduced next time. In the present embodiment, a condition that three seconds have passed since the display switching condition was satisfied last time is employed. . By adopting such a condition, it is possible to update the original image every 3 seconds.

Therefore, when 3 seconds have not yet passed since the last time the display switching condition was satisfied, that is, the original image (excluding the decorative image) currently displayed on the display 62 is continuously displayed for 3 seconds. If not, the display switching condition is not satisfied. In such a case, it is determined as NO in Step S28, and the process proceeds to Step S29.
In step S29, the reproduction image acquisition unit 53 maintains the next reproduction target as it is.

On the other hand, when 3 seconds have passed since the last time the display switching condition was satisfied, that is, the original image currently displayed on the display 62 (the portion excluding the decorative image) is continuously displayed for 3 seconds. If the operation is continued, the display switching condition is satisfied. In such a case, it is determined as YES in Step S28, and the process proceeds to Step S30.
In step S30, the reproduction image acquisition unit 53 updates the next reproduction target to image data of the next reproduction order.

  Thus, when the next reproduction target is determined in the process of step S29 or step S30, the process proceeds to step S31.

In step S31, the reproduction image acquisition unit 53 determines whether or not there is an instruction to end the process.
The end instruction is not particularly limited, but in this embodiment, the same instruction as the sound reproduction process of FIG. 3, that is, the instruction to end the reproduction of the image and sound, which is performed by the user operating the operation unit 17, is adopted. It shall be.

In this case, when the instruction to end the reproduction is not given, it is determined as NO in Step S31, the process is returned to Step S22, and the subsequent processes are repeated. That is, the loop process of steps S22 to S31 is repeatedly executed until an instruction to end reproduction is given.
Here, in this embodiment, the loop processing of steps S22 to S31 is executed every 23 ms together with the audio reproduction processing of FIG. That is, every time the audio content recognized by the audio reproduction process is updated, the updated audio content is acquired in the process of step S23. Thereby, the decoration image added by the process of step S25 or S26 is updated every 23 ms.
On the other hand, the original image data to be reproduced next time is updated every time the display switching condition is satisfied, that is, every 3 s in this embodiment, from the processing of steps S28 to S30. That is, the original image is so-called slide-played every 3 s.

  Thereafter, when an instruction to end reproduction is given, it is determined as YES in step S31, and the image reproduction processing is ended. At this time, the audio reproduction process of FIG. 3 is also almost completed.

As described above, the playback device 1 of this embodiment includes the display unit 18, the audio output unit 19, the audio content recognition unit 52, and the audio content reflection unit 54.
The audio output unit 19 outputs audio represented by the audio data by reproducing the audio data.
The audio content recognition unit 52 recognizes the audio content included in the audio data by analyzing the audio data to be reproduced by the audio output unit 19.
The display unit 18 displays the image represented by the image data as an original image by reproducing the image data.
The audio content reflecting unit 54 reflects the audio content recognized by the audio content recognizing unit 52 while maintaining the configuration and composition of the original image while the display unit 18 is reproducing the image data. Is displayed as a voice content reflection process.
As a result, while reproducing the original image data, it is possible to easily and easily realize a process of displaying an image reflecting the audio content included in the audio data while maintaining the composition and configuration of the original image. It becomes possible.
In addition, the audio content reflecting unit 54 performs image processing for adding a decoration image corresponding to the audio content recognized by the audio content recognizing unit 52 on the original image as audio content reflecting processing. To apply.
Therefore, by adding a decoration image corresponding to the audio content, it is possible to easily and easily realize a process of displaying an image reflecting the audio content included in the audio data while maintaining the composition and configuration of the original image. Is possible.
As a result, the audio content included in the audio data can be reflected on the image data to be reproduced without modifying the original image, and the user's method of expression can be diversified.

The playback device according to the first embodiment of the present invention has been described above.
Hereinafter, a reproducing apparatus according to the second embodiment of the present invention will be described.

[Second Embodiment]
As described above, the playback apparatus according to the present invention executes the following audio content reflection process when playing back image data to be played back as at least a part of the image playback process. That is, the playback device uses an image represented by the image data to be played back as an original image, and reflects an audio content recognized by the audio playback processing while maintaining the configuration and composition of the original image. The display process is executed as a voice content reflection process.

  In the first embodiment, image processing for adding a decoration image corresponding to the audio content recognized by the audio reproduction processing to the original image is adopted as the audio content reflection processing. This makes it possible to display an image (decorative image in this embodiment) that reflects the audio content recognized by the above-described audio reproduction process while maintaining the configuration and composition of the original image.

  On the other hand, in the second embodiment, the following audio content reflection process is adopted on the assumption that a reproduction target is selected from a plurality of reproduction target candidate image data. That is, a process of selecting image data corresponding to the audio content recognized by the audio reproduction process among the plurality of reproduction target candidate image data as the reproduction target image data is adopted as the audio content reflection process.

Such audio content reflection processing of the second embodiment can be realized by the same hardware configuration and functional configuration as those of the playback device 1 of the first embodiment.
Therefore, the playback apparatus according to the second embodiment also has the hardware configuration of FIG. 1 and the functional configuration of FIG. Therefore, the playback apparatus according to the second embodiment will also be described using the same reference numerals as those in the first embodiment, and the description of the configuration described in the first embodiment will be omitted.
Of the processes of the playback apparatus 1 according to the second embodiment, the music playback process can be executed according to the flowchart of FIG. 3 as in the first embodiment. Therefore, the description of the music playback process according to the second embodiment is omitted.

Therefore, with reference to FIG. 8, only the image reproduction process among the processes of the reproduction apparatus 1 according to the second embodiment will be described below.
FIG. 8 is a flowchart for explaining the flow of image reproduction processing according to the second embodiment.

  For example, the image reproduction processing according to the second embodiment is started when the user determines the sound data to be reproduced and a plurality of original image data to be reproduced by operating the operation unit 17. To do. That is, the image reproduction process of FIG. 8 is also started in synchronization with the start of the audio reproduction process of FIG.

However, in the second embodiment, the order of reproduction is not particularly determined for a plurality of original image data to be reproduced, and in some cases (depending on the audio content included in the reproduced audio data) There is a possibility that it is not. However, it is assumed that each of a plurality of original image data as candidates for reproduction is associated with at least one of audio contents that can be recognized by the audio reproduction process.
Specifically, for example, in the second embodiment as well, in the same way as in the first embodiment, the audio content that can be recognized by the audio reproduction process is 3 voices that do not include male voice, female voice, and human voice. It is assumed that the type is adopted. Accordingly, here, each of the plurality of original image data of the reproduction target candidates is associated with any one of three types of sounds including a male voice, a female voice, and a human voice. It shall be.
More specifically, for example, it is assumed that a male voice is associated with image data including a male as a subject. Further, it is assumed that a female voice is associated with image data including a female as a subject. It is assumed that sound data that does not include a human voice is associated with image data such as a landscape image that does not include a human being.

  In step S41, the audio content reflecting unit 54 acquires the audio content. Here, the audio content acquired by the audio content reflection unit 54 is the audio content of the audio data in the processing target period recognized by the audio content recognition unit 52 in step S6 of the audio reproduction process of FIG. . Specifically, for example, the audio content reflecting unit 54 acquires any of the identification codes CB, CC, and CD as in the first embodiment as the audio content in the process of step S23.

  In step S42, the audio content reflecting unit 54 determines whether or not the audio content acquired in the process of step S41 has changed from the previous time.

If the identification code acquired in the immediately previous step S41 is the same as the identification code acquired in the previous step S41, the audio content has not changed from the previous time, so step S42. In step S43, the process proceeds to step S43.
In step S43, the audio content reflection unit 54 executes an audio content reflection process such that the same image data as the previous time is acquired through the reproduction image acquisition unit 53 as the next reproduction target.
In other words, the audio content reflection unit 54 instructs the reproduction image acquisition unit 53 to acquire the same image data as the previous reproduction object as the next reproduction target. The reproduction image acquisition unit 53 acquires the instructed image data from the image storage unit 32 and supplies the acquired image data to the audio content reflection unit 54.
When the audio content reflecting unit 54 has a function of buffering image data, it is not particularly necessary for the reproduced image acquiring unit 53 to acquire the same image data.

On the other hand, if the identification code acquired in the previous step S41 is different from the identification code acquired in the previous step S41, the audio content has changed from the previous time. In step S42, it is determined as YES, and the process proceeds to step S44.
In step S44, the audio content reflection unit 54 executes an audio content reflection process such that image data corresponding to the audio content is acquired through the reproduction image acquisition unit 53 as a next reproduction target.
Note that the processing in step 44 is basically the same as the processing in step S43 except that the image data to be acquired is different, and therefore a description of a specific method for acquiring image data is omitted. To do.

  In step S45, under the control of the CPU 11, the display unit 18 reproduces the image data acquired as the next reproduction target in the process of step S43 or S44, that is, the original image data. Thereby, in the present embodiment, an original image as shown in FIG. 9 and an image corresponding to the audio content is displayed on the display 62.

FIG. 9 shows an example of an original image displayed in the process of step S45 and corresponding to the audio content.
In the example of FIG. 9, three original images 111 to 113 are adopted as reproduction target candidate images. The original image 111 is an image including a female 121 as a subject, and is associated with a female voice. The original image 112 is an image including a male 122 as a subject, and is associated with a male voice. The original image 113 is an image that does not include a human and includes a landscape 123 as a subject, and is associated with a sound that does not include a human voice.
For example, when the identification code CB is acquired as the audio content in the process of step S41, that is, when a male voice is recognized, the image data of the original image 112 is reproduced next time in the process of the next step S43 or step S44. A voice content reflection process such as acquisition as a target is executed. As a result, in the next step S45, the original image 112 including the male 122 as a subject is displayed on the display 62 as shown on the right side of FIG.
Further, for example, when the identification code CC is acquired as the audio content in the process of step S41, that is, when a female voice is recognized, the image data of the original image 111 is next time processed in the next step S43 or step S44. An audio content reflection process is performed such as acquisition as a reproduction target. As a result, in the process of the next step S45, the original image 111 including the woman 121 as a subject as shown in the upper left part of FIG.
Further, for example, when the identification code CD is acquired as the voice content in the process of step S41, that is, when a sound that does not include a human voice is recognized, the original image 113 is processed in the next step S43 or step S44. A sound content reflection process is executed such that image data is acquired as a next playback target. As a result, in the processing of the next step S45, an original image 113 that includes only the landscape 123 and does not include a person as a subject, as shown in the lower left of FIG.

  In this way, when the image corresponding to the audio content is displayed on the display 62 by the process of step S45, the process proceeds to step S46.

In step S46, the reproduced image acquisition unit 53 determines whether or not there is an instruction to end the process.
The end instruction is not particularly limited, but in the second embodiment as well, in the same manner as in the first embodiment, the same instruction as the sound reproduction processing in FIG. It is assumed that an instruction for ending audio playback is adopted.

In this case, if an instruction to end reproduction has not been given, it is determined as NO in step S46, the process returns to step S41, and the subsequent processes are repeated. That is, until the end of reproduction is instructed, the loop processing of steps S41 to S46 is repeatedly executed.
Here, assuming that the loop processing of steps S41 to S46 is executed every 23 ms in combination with the audio reproduction processing of FIG. 3, the display of the original image is updated in 23 ms, and the update timing is in the eyes of the user. Appears to be too early. Therefore, here, it is assumed that the loop processing of steps S41 to S46 is executed every 3 s together with the audio reproduction processing of FIG. Thus, every 3 s, the original image corresponding to the audio content is so-called slide reproduced.

  Thereafter, when an instruction to end reproduction is given, it is determined as YES in step S46, and the image reproduction processing is ended. At this time, the audio reproduction process of FIG. 3 is also almost completed.

As described above, the playback device 1 of this embodiment includes the display unit 18, the audio output unit 19, the audio content recognition unit 52, and the audio content reflection unit 54.
The audio output unit 19 outputs audio represented by the audio data by reproducing the audio data.
The audio content recognition unit 52 recognizes the audio content included in the audio data by analyzing the audio data to be reproduced by the audio output unit 19.
The display unit 18 displays the image represented by the image data as an original image by reproducing the reproduction target image data selected from the plurality of reproduction target candidate image data.
While the image data is being reproduced by the display unit 18, the audio content reflection unit 54 maintains the configuration and composition of the original image, and among the plurality of reproduction target candidate image data, the audio content recognition unit 52. The process of selecting the image data corresponding to the audio content recognized by the above as image data to be reproduced is executed as the audio content reflection process.
Accordingly, by selecting the image data corresponding to the audio content as the image data to be reproduced, an image reflecting the audio content included in the audio data can be displayed while maintaining the composition and configuration of the original image. .
Thereby, it is possible to diversify the expression method by the user using a plurality of original images without modifying the original image.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

For example, in the above-described embodiment, a decoration image corresponding to the audio content is added to the original image, but the decoration image is not limited to a still image without movement. For example, the decoration image may be a moving animation GIF, a moving image in the Adobe Flash (registered trademark) format, or the MP4 format.
In addition, as a decorative image, a decorative image 91 corresponding to a male voice, a decorative image 92 corresponding to a female voice, and a decorative image 93 corresponding to a sound not including a human voice (decorative image imitating a musical note) 93) is employed, but is not limited thereto. For example, a character on the decorative image may dance according to the tempo of the audio content, or a character such as a man, a woman, or an animal on the decorative image may puncture in response to the utterance of the audio content.

  In the above-described embodiment, the audio data is music, but is not limited thereto. For example, the voice data may be voice data composed of voice-only narration or dialogue.

  The audio data stored in the audio storage unit 31 is not limited to the audio data acquired from the removable medium 41 via the drive 22, and may be audio data acquired from the outside via the communication unit 21.

For example, in the above-described embodiment, the playback apparatus 1 to which the present invention is applied has been described as an example configured as a digital photo frame.
However, the present invention is not particularly limited to this, and can be applied to general electronic devices having a display function. For example, the present invention can be widely applied to personal computers, portable navigation devices, portable game machines, and the like. .

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is provided not only by the removable medium 41 distributed separately from the apparatus main body in order to provide the program to the user, but also provided to the user in a state of being preinstalled in the apparatus main body. Recording medium. The removable medium 41 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being incorporated in advance in the apparatus main body includes, for example, a ROM 12 in which a program is recorded, a hard disk included in the storage unit 20, and the like.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 1 ... Playback apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... RTC, 15 ... Bus, 16 ... Input / output interface, 17 ... Operation , 18 ... display unit, 19 ... audio output unit, 20 ... storage unit, 21 ... communication unit, 22 ... drive, 31 ... audio storage unit, 32 ... image Storage unit 41... Removable media 51. Reproduction audio acquisition unit 52. Audio content recognition unit 53. Reproduction image acquisition unit 54. Audio content reflection unit 61. Display control unit, 62... Display, 71... Audio output control unit, 72.

Claims (5)

Sound reproducing means for outputting sound represented by the sound data by playing the sound data;
Audio content recognition means for recognizing audio content included in the audio data by analyzing the audio data to be reproduced by the audio reproduction means;
Image reproducing means for displaying an image represented by the image data as an original image by reproducing the image data;
While the image data is being reproduced by the image reproduction means, an image reflecting the audio content recognized by the audio content recognition means is displayed while maintaining the configuration and composition of the original image. Voice content reflecting means for executing processing as voice content reflecting processing;
A playback device comprising: The voice content reflecting means is
Image processing for adding a decoration image corresponding to the audio content recognized by the audio content recognition means to the original image is applied to the reproduction target image data as the audio content reflection processing.
The playback apparatus according to claim 1. The image reproduction means reproduces the reproduction target image data selected from a plurality of reproduction target candidate image data,
The voice content reflecting means is
The process of selecting the image data corresponding to the audio content recognized by the audio content recognition means as the image data to be played back as the audio content reflection process from among the plurality of reproduction target candidate image data To
The playback apparatus according to claim 1. Sound reproducing means for outputting sound represented by the sound data by playing the sound data;
Image reproducing means for displaying an image represented by the image data as an original image by reproducing the image data;
A playback method for a playback device comprising:
An audio content recognition step for recognizing audio content included in the audio data by analyzing the audio data to be reproduced by the audio reproduction means;
While the image data is being reproduced by the image reproduction means, an image reflecting the audio content recognized by the audio content recognition step is processed while maintaining the configuration and composition of the original image. A voice content reflection step for executing the display process as a voice content reflection process;
A playback method including: Sound reproducing means for outputting sound represented by the sound data by playing the sound data;
Image reproducing means for displaying an image represented by the image data as an original image by reproducing the image data;
A computer for controlling a playback device comprising:
An audio content recognition function for recognizing audio content included in the audio data by analyzing the audio data to be reproduced by the audio reproduction unit;
While the image data is being reproduced by the image reproduction means, an image reflecting the audio content recognized by the audio content recognition function is maintained while maintaining the configuration and composition of the original image. A voice content reflection function for executing the display process as a voice content reflection process;
A program that realizes

Images