JP4352409B2 - Multimedia coded data separation and transmission device - Google Patents

Description

  The present invention relates to multimedia encoding for performing processing that enables synchronous reproduction of video data and audio data included in one multimedia encoded data in a physically separated video playback apparatus and audio playback apparatus. The present invention relates to a data separation and transmission apparatus.

  In this specification, data including video information (video information) such as general luminance signals and color difference signals is referred to as video data, and linear PCM (Pulse Code Modulation) or analog audio signal Data including such audio information (voice information) will be referred to as audio data. Also, encoded data in which video data is compressed is referred to as video encoded data, and encoded data in which audio data is compressed is referred to as audio encoded data. Therefore, the video encoded data and the audio encoded data are included in the video data and the audio data. In this specification, data including both video encoded data and audio encoded data is referred to as multimedia encoded data.

  Conventionally, several techniques for synchronously reproducing video data and audio data on respective physically separated playback apparatuses (video playback apparatus and audio playback apparatus) have been proposed. Products suitable for the diverse needs of consumers are also on the market. For example, for data transmission using wireless transmission, a data transmission device that transmits audio data as an optical signal in the infrared region, and the optical signal is received by a light receiving device and converted from an optical signal to an electrical signal for audio reproduction. There exists an infrared transmission type audio reproduction system composed of headphones.

  Recently, IEEE802.11a using 5 GHz (gigahertz) band and IEEE802.11b and IEEE802.11g using 2.4 GHz band are known as transmission systems using radio waves. For example, a short-range wireless transmission method such as Bluetooth (registered trademark) is also used for transferring audio data. Data transmission using these radio waves can reduce the size of the transmission / reception unit. In particular, the cost of Bluetooth (registered trademark) is low, and the transmission range is wide compared to wireless communication using light. Because it is very convenient.

  Also, when constructing a multi-channel playback environment by arranging rear speakers, there is a problem that the speaker cord that extends to the rear speakers gets in the way. In order to solve this problem, an amplifier is mounted on the rear speakers. In addition, there is an audio reproduction system that is provided with a wireless signal receiving device and transmits audio data to the rear speaker as a radio signal (wireless signal), thereby supplying the audio data to the rear speaker.

  The synchronized playback of video data and audio data given as an example above is usually supplied to a speaker through a speaker cord or a headphone cord, and an electrical signal of audio data returned to vibration energy in the speaker is converted into an optical signal or a radio signal. By converting the signal into a wireless signal and transmitting it, audio output can be performed in an audio output device that is located physically apart. Therefore, the above-described synchronized playback is not performed while measuring the timing of video data and audio data, and the synchronized playback depends on the output timing of video data and audio data in the video playback device and audio playback device. ing.

  On the other hand, the following Patent Document 1 discloses a technique for realizing synchronized playback of video data and audio data included in multimedia encoded data in a video playback device and an audio playback device that are physically separated from each other. It is disclosed. In the technique disclosed in Patent Document 1, for example, a video playback device and an audio playback device (headphones) are connected by a wireless medium. The video playback device plays back video data and also encodes audio encoded data. Is supplied to the audio reproducing apparatus. The audio reproduction device performs a decoding process on the audio encoded data received from the video reproduction device and outputs the audio. In addition, the audio playback device performs audio playback synchronized with video playback in consideration of input / output buffer capacity, communication delay, decoding delay, and the like at the start of playback. It is possible to eliminate the deviation by measuring the amount of deviation between the video output and the audio output using the clock synchronization and changing the audio reproduction speed.

  Next, an internal configuration of a multimedia encoded data reproducing apparatus that reproduces multimedia encoded data according to a conventional technique will be described.

  FIG. 5 is a block diagram showing an example of a multimedia encoded data reproducing apparatus according to the prior art. Note that the multimedia encoded data playback device shown in FIG. 5 performs synchronized playback of video data and audio data included in the multimedia encoded data, and corresponds to, for example, a portable DVD player.

  The multimedia encoded data reproduction apparatus 1000 shown in FIG. 5 includes a multimedia stream buffer 1001, a system stream separation unit 1002, an audio decoding unit 1003, a video decoding unit 1004, a control unit (CPU: Central Processing Unit) 1005, An audio data buffer 1006, a video data buffer 1007, a speaker 1008, and a monitor 1009 are provided.

  In FIG. 5, a multimedia stream buffer 1001 is, for example, a DVD disk that stores multimedia encoded data. For example, a system stream read from the multimedia stream buffer 1001 using an optical pickup is separated into a video stream and an audio stream by a system stream separation unit 1002, and the audio stream is decoded by an audio decoding unit 1003 and the video stream is video decoded. The unit 1004 performs a decoding process on each compressed data. Note that encoded data, a stream, and a bit stream are synonyms, and the encoded data is generally called a stream.

  The audio data decoded by the audio decoding unit 1003 and the video data decoded by the video decoding unit 1004 are temporarily stored in an output buffer (audio data buffer 1006 or video data buffer 1007). The control unit 1005 that performs playback control based on time information described in the system stream controls audio data output from the audio data buffer 1006 and video data output from the video data buffer 1007. The audio data output from the audio data buffer 1006 is converted into audio through the speaker 1008, and the video data output from the video data buffer 1007 is converted into an image through the monitor 1009.

  As described above, the synchronous reproduction of the video data and the audio data is managed by the control unit 1005 in the multimedia encoded data reproduction apparatus 1000. For example, the audio decoding unit 1003, the video decoding unit 1004, or an output buffer (audio data) This is realized by controlling the buffer 1006 and the video data buffer 1007).

  FIG. 6 is a block diagram illustrating an example of a separated multimedia encoded data reproducing apparatus according to a conventional technique. FIG. 7 illustrates an example of a separated multimedia encoded data reproducing apparatus according to a conventional technique. FIG. The separated multimedia encoded data reproducing apparatus shown in FIGS. 6 and 7 wirelessly transmits audio data by light or radio waves to, for example, physically separated headphones and speakers.

  The separation type multimedia encoded data reproduction apparatus shown in FIG. 6 includes a multimedia encoded data reproduction transmission apparatus 2001 and an audio reproduction apparatus 2002. The separation type multimedia encoded data reproduction apparatus shown in FIG. 6 functions as a multimedia encoded data reproduction transmission apparatus 2001, for example, as shown in FIG. 7, and has a monitor for performing video output, operation buttons, and the like. This corresponds to a set of a main device 3001 and a headphone 3002 that functions as the audio playback device 2002. That is, the separated multimedia encoded data reproduction device shown in FIG. 6 separates the audio reproduction device from the multimedia encoded data reproduction device shown in FIG. 5, and transmits audio data wirelessly. It is configured as follows.

  The multimedia encoded data reproduction / transmission apparatus 2001 shown in FIG. 6 includes a multimedia stream buffer 2101, a system stream separation unit 2102, an audio decoding unit 2103, a video decoding unit 2104, a control unit 2105, an audio data buffer 2106, and a video. A data buffer 2107, a monitor 2109, and an audio data transmission unit 2110 are included. However, as will be described later, the audio decoding unit 2103 is not necessarily implemented.

  When compared with the multimedia encoded data reproduction apparatus 1000 shown in FIG. 5, the multimedia encoded data reproduction apparatus 1000 of FIG. 5 outputs audio from the speaker 1008, while shown in FIG. The multimedia encoded data reproduction / transmission device 2001 is different in that it is configured to wirelessly transmit audio data to the audio reproduction device 2002 through the audio data transmission unit 2110. The function) is basically the same.

  On the other hand, the audio playback device 2002 illustrated in FIG. 6 includes an audio data receiving unit 2201, an audio decoding unit 2202, an audio data buffer 2203, and a speaker 2204. The audio data transmitted from the multimedia encoded data reproduction / transmission apparatus 2001 is received by the audio data receiving unit 2201, and after being decoded by the audio decoding unit 2202 and the buffer by the audio data buffer 2203, the audio data is received from the speaker 2204. Sound is output.

  Note that when the audio decoding unit 2103 is implemented in the multimedia encoded data reproduction / transmission device 2001, the audio decoding unit 2202 does not need to be implemented in the audio reproduction device 2002, and conversely, in the audio reproduction device 2002. When the audio decoding unit 2202 is installed, the audio decoding unit 2103 need not be installed in the multimedia encoded data reproduction / transmission apparatus 2001.

  For example, when the audio decoding unit 2103 is mounted in the multimedia encoded data reproduction / transmission device 2001, the multimedia is taken into account in consideration of delays associated with wireless transmission to the audio reproduction device 2002 and buffer delay. The control unit 2105 of the encoded data reproduction / transmission apparatus 2001 controls the output timing of video data and audio data, so that the video data and the audio data are the same as the multimedia encoded data reproduction apparatus 1000 shown in FIG. Can be realized.

  On the other hand, when the audio decoding unit 2202 is mounted on the audio reproduction device 2002, the multimedia encoded data reproduction / transmission device 2001 and the audio reproduction device, as in the technique disclosed in Patent Document 1 described above. The difference between the output timing of the video data and the output timing of the audio data is adjusted by examining the shift amount of the reference time each referred to by 2002, and the audio data buffer storage amount per unit time is adjusted based on the change degree of the audio data buffer. It is necessary to control the output timing of audio data in the playback apparatus 2002. Note that the difference between the output timing of the video data and the output timing of the audio data is based on the reference operating frequency of the multimedia encoded data playback / transmission apparatus 2001 on the main body side and the reference of the audio playback apparatus 2002 on the headphone side. This is a phenomenon that occurs because the operating frequency is different.

  FIG. 8 shows a data structure of a system stream of MPEG-2 (Motion Picture Experts Group-2), which is a representative multimedia system stream in the prior art. FIG. 8 shows, as an example, a format used when recording on a storage medium called PS (Program Stream). As another typical format of the data structure of the MPEG-2 system stream, TS (Transport Stream) is defined. This TS is mainly used for communication and broadcasting applications. In the following, the PS format will be mainly described assuming that the storage medium is reproduced.

  PS is a general multimedia system stream, and has a structure in which auxiliary information including video data, audio data, time information, etc. is subdivided into a predetermined unit amount and then multiplexed. The PS is configured by a PS pack in which a plurality of variable length PES (Packetized Elementary Stream) packets are further continued. Here, Elementary Stream (elementary stream) is video element data and audio element data called ES.

The PES packet in the PS pack is a collection of audio ES and video ES to be reproduced at the same time. Further, the header in the PES packet includes a start code, a stream type (video, audio, data) data length, and a part (PES_header_option_data) describing PES option information. The option information includes information for managing time such as PTS (video or audio display or output time) and DTS (decoding start time), and is used as an index for realizing synchronized playback of video and audio. It is possible.
JP 2004-7140 A (FIGS. 1, 4, and 7)

  As described above, when playing back multimedia encoded data, a video playback device that plays back video data (output to a monitor) is an audio playback device that is physically separated from the audio playback device. By wirelessly transmitting audio data in real time, simultaneous playback of video playback on the video playback device and audio playback on the audio playback device is performed.

  By the way, recently, manners when using radio waves in public places are regarded as social problems including physical effects. In particular, the use of mobile phones in public institutions such as trains and hospitals is concerned with noise to the people around them and danger to the human body, and there are practical restrictions on their use. The same applies to the transmission of audio data using radio waves. Therefore, when synchronized playback with video data, wireless transmission of audio data in real time using radio waves is very effective if it is a closed space in each home or a space that does not affect the surroundings. Although it is a technology, it cannot be said to be a very suitable technology in situations where the use of radio waves is not preferred.

  Therefore, when synchronized playback of video data and audio data, wireless transmission of audio data in real time using radio waves to an audio playback device that is located physically apart is particularly important for trains. Not suitable for use in medium or highly public facilities. That is, for example, wireless transmission of audio data in real time using the above-described Bluetooth (registered trademark), a transmission method using a 5 GHz band or a 2.4 GHz band, or the like must be basically avoided. is there.

  On the other hand, wireless connection in which a cord is not connected to headphones or speakers can provide a comfortable environment with high convenience to the user. For example, as described above, a wireless transmission method (infrared wireless transmission method) using light such as infrared rays is effective as a technique for wirelessly transmitting audio data without using radio waves while maintaining a wireless state. .

  When synchronized playback of video data and audio data, if a wireless transmission method using light is used to transmit audio data in real time to an audio playback device that is physically located, audio Light is transmitted to both the data transmission side (corresponding to the multimedia encoded data reproduction / transmission apparatus 2001 shown in FIG. 6) and the audio data reception side (corresponding to the audio reproduction apparatus 2002 shown in FIG. 6). It is necessary to provide an optical transmission / reception unit for performing transmission / reception of audio data, and to transmit audio data in real time via this optical transmission / reception unit. However, this optical transmission / reception unit is more expensive than a transmission / reception unit using radio waves, and it is difficult to reduce the size of the optical transmission / reception unit. The problem of being disturbed arises.

  As described above, when radio waves are used for wireless transmission of audio data, it is possible to reduce the cost and size of the apparatus. However, for example, the effects of radio waves on the human body, etc. As a result, there arises a problem that it is necessary to sufficiently consider the influence on the surrounding environment. On the other hand, when light is used for wireless transmission of audio data, it is not necessary to consider the impact on the surrounding environment that occurs when radio waves are used, but this increases the costs associated with the installation of optical transceiver units and increases the size of the device. There is a problem that convenience is reduced due to the increase in weight and the like. In view of these problems, it is desirable to maintain a wireless connection such as headphones and speakers and to realize a comfortable reproduction environment for multimedia encoded data.

  In order to solve the above problems, the present invention maintains the convenience of a state in which the video playback apparatus and the audio playback apparatus are physically separated and connected wirelessly (wireless connection state). Multimedia encoded data separation that enables synchronized playback of video and audio data with a simple configuration, even in environments where radio wave usage is restricted, such as in highly public places An object is to provide a transmission apparatus.

To achieve the above object, according to this onset bright, and encoded video data, and audio encoded data, said auxiliary information including time information for managing the playback time of each encoded data is multiplexed Multimedia encoded data reading means for reading the multimedia encoded data from a recording medium on which the encoded multimedia encoded data is recorded;
Multimedia encoded data separating means for separating and extracting the audio encoded data and the auxiliary information from the multimedia encoded data read by the multimedia encoded data reading means;
Audio reconfiguration for reconstructing the audio encoded data and the auxiliary information separated and extracted by the multimedia encoded data separation means so that audio reproduction related to the audio encoded data is performed in a predetermined audio reproduction device Means,
Said audio encoded data and the auxiliary information reconstructed by the audio reconstruction means, and a data transmission means for transmitting to said predetermined audio reproduction apparatus,
An instruction from the outside regarding the reproduction operation of the multimedia encoded data after the audio encoded data and the auxiliary information reconstructed by the audio reconstructing means are transmitted to the predetermined audio reproducing device is input. Operation input means for
The auxiliary information including time information indicating the reproduction position related to the reproduction operation input by the operation input unit and the operation information for instructing the start of the reproduction operation are wirelessly transmitted to the predetermined audio reproduction device. Wireless signal transmission means for transmitting via
Video decoding means for performing decoding processing after the reproduction position related to the reproduction operation input by the operation input means with respect to the video encoded data included in the multimedia encoded data;
Video display means for visually displaying video based on video data generated by the decoding process of the video encoded data in the video decoding means;
An apparatus for separating and transmitting multimedia encoded data is provided.

  The present invention has the above-described configuration, and the video playback apparatus and the audio playback apparatus exist in physically separated locations and maintain a wireless connection state (wireless connection state), for example, Even in environments where radio wave use is restricted, such as in highly public places, it is possible to achieve synchronized playback of video data and audio data with a simple configuration, maintain the convenience of wireless connection, and reduce the cost of equipment And having the effect of enabling downsizing and weight reduction of the apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The synchronized playback method of video data and audio data according to the embodiment of the present invention is roughly divided into two stages: a preparation stage before playback (pre-process stage) and a playback stage (post-process stage) in which the user actually watches. It is done.

  First, the pre-process stage, which is a pre-reproduction stage, will be described with reference to FIG. FIG. 1 is a block diagram showing an example of the configuration of a multimedia encoded data separating and transmitting apparatus and an audio reproducing apparatus that perform processing related to the previous process stage in the embodiment of the present invention.

  FIG. 1 shows a multimedia encoded data separation / transmission apparatus 100 and an audio reproduction apparatus 200. A multimedia encoded data separation / transmission apparatus 100 shown in FIG. 1 includes a multimedia stream buffer 101, a system stream separation unit 102, an audio stream buffer 103, an auxiliary information buffer 104, an audio stream reconstruction unit 105, and an audio stream transmission. Part 106 is provided. 1 includes an audio stream receiving unit 201, an audio stream buffer 202, and a post-process processing unit 203 that performs a process related to a post-process stage.

  The multimedia stream buffer 101 is a storage medium capable of storing multimedia encoded data. For example, if the main body of the multimedia encoded data separating / transmitting apparatus 100 is a DVD player, it corresponds to a DVD disk. An information terminal having a multimedia playback function such as a mobile phone or PDA (Personal Digital Assistance) corresponds to an HDD (Hard Disc Drive), a semiconductor memory, or the like. Here, it is assumed that the multimedia system stream related to the content that the user desires to view is stored in advance in the multimedia stream buffer 101.

  In general, multimedia encoded data is composed of video data and audio data, as well as information obtained by encoding content, content identification information, and other auxiliary information (including time information and attribute information in particular). After the information is compressed according to the respective encoding methods (the information may not be compressed), each is multiplexed to generate a system stream (multimedia system stream). Therefore, when decoding video encoded data or audio encoded data, it is necessary to extract and extract each element (elementary stream) from the system stream in advance.

  The system stream read from the multimedia stream buffer 101 is separated into an audio stream and auxiliary information (particularly including time information and attribute information) associated with the system stream by the system stream separation unit 102. After separation by the system stream separation unit 102, the audio stream is stored in the audio stream buffer 103, and the auxiliary information is stored in the auxiliary information buffer 104.

  The audio playback device 200 that is the transmission destination of the audio stream cannot perform playback even if only the audio stream separated by the system stream separation unit 102 is provided. That is, when an audio stream is played back by the audio playback device 200, content information accompanying the audio stream and auxiliary information for managing playback time are required. Therefore, the audio stream reconstructing unit 105 of the multimedia encoded data separation and transmission apparatus 100 multiplexes the auxiliary information stored in the auxiliary information buffer 104 into the audio stream stored in the audio stream buffer 103, and creates a new one. The correct audio stream. The reconstructed audio stream is generated so that it can be played back by the audio playback device 200.

  The audio stream reconstructed by the audio stream reconstructing unit 105 is transmitted through the audio stream transmitting unit 106 to the audio reproducing device 200 physically separated from the multimedia encoded data separation / transmission device 100. Note that the transmission method of the audio stream from the multimedia encoded data separation transmission device 100 to the audio reproduction device 200 is not particularly limited, and for example, wired transmission via USB (Universal Serial Bus), infrared light, or the like. Or wireless transmission using radio waves can be employed. Further, for example, the audio stream reconstructed by the audio stream reconstructing unit 105 is stored in a portable storage medium, and the audio reproduction apparatus 200 is separated from the multimedia encoded data separation / transmission apparatus 100 by attaching / detaching the portable storage medium. Alternatively, the reconstructed audio stream may be moved.

  On the other hand, for example, the audio stream transmitted from the audio stream transmission unit 106 is received by the audio stream reception unit 201 of the audio reproduction device 200, stored in the audio stream buffer 202 provided in the audio reproduction device 200, Preparation before reproduction (pre-process stage) is completed. Note that the reconstructed audio stream stored in the audio stream buffer 202 is reproduced by the post-process processing unit 203 at the post-process stage. The post-process processing unit 203 will be described in detail later.

  As described above, in the pre-process stage, after the audio stream and the auxiliary information are once separated from the multimedia encoded data in the multimedia encoded data separating and transmitting apparatus 100, the auxiliary information is multiplexed into the audio stream, The audio stream is reconstructed without the video stream. The reconstructed audio stream is stored in the audio playback device 200 in advance. In the pre-process stage, the reconstructed audio stream is stored in the audio playback apparatus 200 in advance, so that the multimedia encoded data separation / transmission apparatus 100 can perform the actual playback process (the post-process stage described later). Audio data need not be transmitted to the audio playback device 200 in real time.

  Next, a post-process stage, which is a playback stage where the user actually watches, will be described. FIG. 2 is a block diagram showing an example of a configuration of a multimedia encoded data reproducing apparatus that performs processing related to a post-process stage in the embodiment of the present invention.

  The separation type multimedia encoded data reproduction apparatus shown in FIG. 2 includes a multimedia encoded data reproduction apparatus 300 and an audio reproduction apparatus 200. The multimedia encoded data playback device 300 shown in FIG. 2 includes a multimedia stream buffer 301, a system stream separation unit 302, an auxiliary information extraction unit 303, a video decoding unit 304, a control unit 305, and auxiliary information / operation information transmission. Section 306, video data buffer 307, and monitor 308. Note that the multimedia encoded data playback device 300 is a video playback device that can output video from the monitor 308.

  On the other hand, the audio playback device 200 illustrated in FIG. 2 is the same as the audio playback device 200 illustrated in FIG. 1, and a reconfigured audio stream is stored in the audio stream buffer 202. FIG. 2 also shows details of the components of the audio playback device 200 (the post-processing unit 203 shown in FIG. 1) for performing playback processing of the reconstructed audio stream. That is, the audio playback device 200 illustrated in FIG. 2 includes an audio stream receiving unit 201, an audio stream buffer 202, an auxiliary information / operation information receiving unit 204, a control unit 205, an audio decoding unit 206, an audio data buffer 207, a speaker. 208.

  The multimedia encoded data separation / transmission apparatus 100 shown in FIG. 1 and the multimedia encoded data reproduction apparatus 300 shown in FIG. 2 are preferably implemented by the same apparatus, but are different. It may be realized by a device.

  That is, for example, the multimedia encoded data reproduction device 300 shown in FIG. 2 is realized by the main body device 3001 shown in FIG. 7, and the headphone 3002 shown in FIG. 1 is provided, the main body apparatus 3001 is provided with the function of the multimedia encoded data separation / transmission apparatus 100 shown in FIG. 1, and the multimedia encoding shown in FIG. It is preferable that the functions of both the data separation and transmission apparatus 100 and the multimedia encoded data reproduction apparatus 300 shown in FIG. 2 are realized by one apparatus. However, for example, the multimedia encoded data separation / transmission apparatus 100 shown in FIG. 1 is realized by a fixed DVD player, a personal computer (PC), or the like. It is also possible to provide only the function of the multimedia encoded data reproduction device 300 shown in FIG. In this case, the user reconstructs audio data using a fixed DVD player or PC, and the reconstructed audio data is stored in advance in the audio playback device 200 and is shown in FIG. Playback control of the audio playback device 200 is performed using the main body device 3001.

  In the multimedia encoded data reproduction device 300 shown in FIG. 2, for example, when a content reproduction instruction is given by a user operation, first, the multimedia encoded data related to the content to be reproduced is Read from the multimedia stream buffer 301. Note that, as described above, the content to be played is reconstructed by the multimedia encoded data separation and transmission apparatus 100 shown in FIG. 1, and the reconstructed audio data related to the content is Suppose that it is already stored in the audio playback device 200.

  The system stream separation unit 302 reads the system stream from the multimedia stream buffer 301 and performs separation processing, and sends data including auxiliary information for video / audio synchronized playback described in the system stream to the auxiliary information extraction unit 303. On the other hand, the video stream is output to the video decoding unit 304.

  The video decoding unit 304 performs a video decoding process such as converting the video stream separated by the system stream separation unit 302 into a luminance signal and a color difference signal representing each pixel data for one frame of video, and performs post-processing. Are output to a video data buffer 307 such as a frame buffer. The video data is once stored in the video data buffer 307, then transmitted to the monitor 308 in accordance with an output command from the control unit 305, and video is output through the monitor 308.

  On the other hand, the auxiliary information extraction unit 303 is necessary for synchronized playback of content identification information for identifying content and time information indicating a position at the time of playback from the data including the auxiliary information separated by the system stream separation unit 302. This information is extracted as auxiliary information, and this auxiliary information is supplied to the auxiliary information / operation information transmitting unit 306. In accordance with an instruction from the control unit 305, the auxiliary information / operation information transmission unit 306 receives operation information including content identification information, auxiliary information including time information indicating the position of data to be reproduced, and operation instructions by the user. The data is transmitted to the audio playback device 200. The auxiliary information includes, for example, time information in the system stream and content identification information for identifying content as described above. An example of the operation information is instruction information such as an instruction to start / stop playback or an instruction to start / stop special playback. In the present specification, operations related to playback of various other contents including start / stop of playback and start / stop of special playback are referred to as playback operations. Further, the auxiliary information / operation information transmission unit 306 may combine the auxiliary information and the operation information into one information and transmit the information to the audio playback device 200.

  For example, when a user inputs an operation using an operation panel (not shown in FIG. 2) provided in the multimedia encoded data reproduction apparatus 300, operation information indicating the operation content is transmitted through the control unit 305. While being supplied to the auxiliary information / operation information transmission unit 306, auxiliary information related to the operation information is read from the auxiliary information extraction unit 303 and transmitted to the audio reproduction device 200 through the auxiliary information / operation information transmission unit 306. The

  The auxiliary information / operation information transmitted from the auxiliary information / operation information transmission unit 306 of the multimedia encoded data reproduction device 300 is received by the auxiliary information / operation information reception unit 204 of the audio reproduction device 200 and transmitted to the control unit 205. It is done. For example, when the auxiliary information includes content identification information for identifying the content to be played back, time information indicating the playback start position of the content to be played back, and information for instructing the start of playback as operation information The audio playback device 200 reads out an audio elementary stream at a position corresponding to this time information from the audio stream buffer 202, and the audio decoding unit 206 performs a decoding process (normal playback process here) according to the operation information. As a result, the audio data is output as audio from the speaker 208 via the audio data buffer 207.

  Note that it is necessary to synchronize the reproduction of video (video display by the monitor 308) in the multimedia encoded data reproduction device 300 and the reproduction of audio (audio output from the speaker 208) in the audio reproduction device 200. This synchronization processing is based on the video system delay amount caused by the video data decoding delay and the frame buffer accumulation delay, and the audio system delay caused by the auxiliary information transmission delay of the time information, the audio encoded data decoding delay, the audio data accumulation delay, etc. According to the relationship with the amount, the transmission information of the operation information (audio reproduction instruction timing) from the multimedia encoded data reproduction apparatus 300 to the audio reproduction apparatus 200 and the multimedia encoded data reproduction apparatus 300 in order to determine the video reproduction timing This is realized by setting the video reproduction start timing transmitted in the above to an appropriate timing in advance. In other words, in addition to the timing of the playback instruction by transmitting auxiliary information and operation information, by setting the output timing of video and audio taking into account the delay in each process, synchronization between video playback and audio playback is eliminated. It becomes possible to make it.

  Subsequently, referring to FIGS. 3 and 4, the audio stream reconstructed in the multimedia encoded data separating and transmitting apparatus 100 shown in FIG. 1 (that is, the audio data stored in advance in the audio reproducing apparatus 200). ) Will be described. FIG. 3 is a diagram illustrating a first example of a structure of a reconstructed audio stream according to the embodiment of the present invention. FIG. 4 is a diagram illustrating a structure of a reconstructed audio stream according to the embodiment of the present invention. It is a figure which shows the 2nd example of a structure.

  First, the reconstructed audio data needs to include information corresponding to content identification information (for example, a file name or a program name) for indicating the correspondence between the video data of the content and the audio data. Furthermore, the reconstructed audio data needs to include time information indicating the position of the frame unit or the data length unit of the audio stream that is framed and encoded in a predetermined unit time. These pieces of time information correspond to the conventional PS PTS shown in FIG. As shown in FIG. 3, the reconstructed audio data includes content identification information that enables identification of content, audio data (audio ES) in a predetermined data length unit, and audio data of this audio data. It is possible to have a structure in which time information including reproduction output time is integrated.

  The reconstructed audio stream shown in FIG. 3 has a structure in which time information is given in frame units or data length units as described above. Note that in the audio playback device 200, it is necessary to perform processing for extracting only the audio ES before or simultaneously with the decoding processing, but the amount of processing is extremely small, and the processing load applied to the audio playback device 200 is light. Is something.

  The audio data shown in FIG. 3 has an integrated structure in which time information is added to a predetermined amount of audio ES, and audio playback time management is performed in the audio decoding unit 206. Auxiliary information including time information (eg, time information and data length) is separated from the audio ES, and the auxiliary information portion and the audio portion are separated from each other as shown in FIG. Alternatively, the separated auxiliary information portion and audio ES portion can be combined as they are, and the first half auxiliary information portion can be used as a header portion for the audio stream. In this case, for example, the control unit 205 of the audio playback device 200 can read the auxiliary information and perform time management.

  Also, a stream in which audio PS stored in the multimedia stream buffer 101 is aggregated is transmitted from the multimedia encoded data separating and transmitting apparatus 100 to the audio reproducing apparatus 200, and the audio reproducing apparatus 200 uses this audio. It is also possible to store a stream in which the PSs are aggregated. In this case, although the processing load related to the synchronous playback processing of the audio playback device 200 increases, the processing load when the audio stream is reconstructed in the multimedia encoded data playback device 300 is reduced. Can be expected to speed up data transmission and shorten the time required for pre-processing.

  Next, the flow of operations including post-process processing (synchronized playback processing of video data and audio data) according to the present invention will be described. Here, the above-described pre-process has already been completed, and audio data (audio ES) is stored in advance in the audio stream buffer 202 of the audio playback device 200 together with auxiliary information including time information and content identification information. It is assumed that

  For example, when a content playback command is supplied by a user operation, the multimedia encoded data playback device 300 decodes the video stream and outputs the decoded video stream to the monitor 308, while corresponding to the same time as the video output on the monitor 308. Content identification information (information indicating which content is to be played back) and time information extracted from the stream being played back (from which audio ES of the content) so that the current audio is output from the speaker 208 of the audio playback device 200 Information indicating whether to start reproduction), and operation information indicating an operation requested by the user (for example, information indicating an operation such as normal reproduction or special reproduction) is transmitted to the audio reproduction apparatus 200. Note that transmission of auxiliary information including content identification information, time information, operation information, and the like from the multimedia encoded data playback device 300 to the audio playback device 200 is a processing delay of audio data, a processing delay of video data and audio data, and wireless It is performed at a predetermined timing in consideration of transmission delay and the like.

  For example, when a playback command for a certain content is transmitted to the control unit 305, the control unit 305 issues a decoding start command to the video decoding unit 304, and also transmits content identification information and time information to the external audio playback device 200. Send a playback start command. Similarly, a playback stop command is transmitted from the multimedia encoded data playback device 300 to the audio playback device 200 together with time information. As a result, the audio playback of the audio playback device 200 is always synchronized with the video playback of the multimedia encoded data playback device 300.

  As described above, the multimedia encoded data playback device 300 and the audio playback device 200 are physically separated devices and have different reference operating frequencies, so that there may be a time lag between the operations. There is sex. In general, this deviation depends on the clock accuracy of the apparatus, but since this deviation is minute, it may take time for the user to perceive the deviation. Therefore, after synchronizing the playback start timing by sending a control signal only when starting playback for the first time, the video and audio are basically kept in sync as they are. It is also possible to make the playback device self-run.

  On the other hand, especially when the playback time of the content is long, there is a possibility that a gap between the video and the audio will occur to the extent that the user can perceive. In order to deal with such a deviation, the time information is periodically transmitted from the multimedia encoded data reproduction device 300 to the audio reproduction device 200, and the audio reproduction device 200 detects the deviation of the synchronous reproduction, and the multimedia reproduction is performed. It is also possible to perform audio playback time management according to the time information periodically sent from the encoded data playback device 300. When the deviation is detected, the audio playback device 200 appropriately selects the audio data while taking care not to give the user a sense of incongruity as in the conventional technique (the technique disclosed in Patent Document 1), for example. By inserting or deleting as many samples as possible, it is possible to keep video and audio synchronized at all times by periodically correcting the deviation.

  Further, when the user perceives the shift, the user himself / herself issues a shift correction instruction, whereby a reproduction start command is transmitted again from the multimedia encoded data playback device 300 to the audio playback device 200, and video and audio are transmitted. It is also possible to re-set the synchronized playback. The resetting of the synchronization by the user operation is to correct the deviation as necessary. Therefore, transmission of time information from the multimedia encoded data playback device 300 to the audio playback device 200 is performed only when resetting of synchronous playback is necessary, and the synchronous playback is maintained by the minimum necessary wireless transmission. Has the advantage of being possible.

  In addition, for transmitting auxiliary information and operation information from the multimedia encoded data reproduction device 300 to the audio reproduction device 200, a radio transmission method using radio waves instead of optical wireless communication may be adopted. The present invention does not perform audio data transmission in real time as in the prior art, but between the multimedia encoded data reproduction device 300 and the audio reproduction device 200, a control signal and a response signal with a very small amount of transmission. It can be said that the possibility of affecting the surroundings is extremely small even in an environment where only exchange is performed and the use of radio waves is not preferable.

  As described above, according to the present invention, in a state where the multimedia encoded data playback device (video playback device) and the audio playback device are physically separated and connected wirelessly, When the audio data is reconstructed from the multimedia encoded data in advance by the multimedia encoded data separating and transmitting apparatus, the reconstructed audio data is stored in the audio reproducing apparatus in advance, and the content is reproduced. The reproduction start command (or other operation command) is transmitted from the multimedia encoded data reproduction device to the audio reproduction device at a timing taking delay into consideration in real time. Video data and audio data without wireless transmission of audio data It is possible to realize synchronized playback.

  Note that the configurations and operations described in the above embodiment are merely examples, and the present invention is not limited to the description of the above embodiment. In addition, each component of the block diagram illustrated for explaining the present invention represents a function, and each of these components can be realized by hardware, software, or a combination of hardware and software. It is.

  Moreover, according to this invention, the following aspect is also provided.

According to the third aspect of the present invention, the video encoded data, the audio encoded data, and the auxiliary information including time information for managing the reproduction time of each encoded data are multiplexed. Multimedia encoded data reading means for reading the multimedia encoded data from a recording medium on which the encoded media data is recorded;
Multimedia encoded data separating means for separating and extracting the audio encoded data and the auxiliary information from the multimedia encoded data read by the multimedia encoded data reading means;
Audio reconstructing means for reconstructing the audio encoded data and the auxiliary information separated and extracted by the multimedia encoded data separating means so that audio reproduction related to the audio encoded data is performed;
Data storage means for storing the audio encoded data and the auxiliary information reconstructed by the audio reconstruction means in a recording medium readable by a predetermined audio reproduction device;
A multimedia encoded data separation device is provided.

  According to the fourth aspect of the present invention, in the first to third aspects of the present invention, the audio reconstruction means integrates each of the audio encoded data and the auxiliary information. The audio encoded data and the auxiliary information that have been combined are set as separate data, respectively, or the data that has the structure in which the integrated auxiliary information is added as a header to the integrated audio encoded data. A multimedia encoded data separation / transmission apparatus or a multimedia encoded data separation apparatus configured to reconstruct the audio encoded data and the auxiliary information by generating is provided.

  According to a fifth aspect of the present invention, in the first to third aspects of the present invention, the audio reconstructing means converts the audio encoded data and the auxiliary information into an MPEG-2 program stream format. Thus, a multimedia encoded data separation / transmission apparatus or multimedia encoded data separation apparatus configured to reconstruct the audio encoded data and the auxiliary information is provided.

According to the sixth aspect of the present invention, video encoded data, audio encoded data, and auxiliary information including time information for managing the reproduction time of each encoded data are multiplexed. The audio encoded data and the auxiliary information are reconfigured so that audio reproduction of the audio encoded data is performed after the audio encoded data and the auxiliary information are once separated from the multimedia encoded data. Data storage means,
Radio signal receiving means for receiving the auxiliary information including time information indicating a reproduction position related to a reproduction operation and operation information for instructing to start the reproduction operation from a predetermined video reproduction device via radio;
Audio for performing decoding processing after the reproduction position related to the reproduction operation on the audio encoded data stored in the data storage unit based on the auxiliary information and the operation information received by the wireless signal reception unit Decryption means;
Audio output means for outputting the audio encoded data decoded by the audio decoding means;
An audio playback device is provided.

  According to a seventh aspect of the present invention, in the sixth aspect of the present invention, the wireless signal receiving means is configured to transmit a wireless signal by optical wireless transmission using infrared rays. An audio playback device is provided.

According to the eighth aspect of the present invention, video encoded data, audio encoded data, and auxiliary information including time information for managing the reproduction time of each encoded data are multiplexed. Multimedia encoded data reading means for reading the multimedia encoded data from the recording medium on which the multimedia encoded data is recorded;
Operation input means for inputting an external instruction relating to the reproduction operation of the multimedia encoded data;
The multimedia encoded data includes the auxiliary information including time information indicating the playback position related to the playback operation input by the operation input means and operation information for instructing to start the playback operation. Wireless signal transmission means for wirelessly transmitting to a predetermined audio reproduction device in which the audio encoded data is stored in advance;
Video decoding means for performing decoding processing after the reproduction position related to the reproduction operation input by the operation input means with respect to the video encoded data included in the multimedia encoded data;
Video display means for visually displaying video based on video data generated by the decoding process of the video encoded data in the video decoding means;
Video reproduction control for controlling to start video output after the reproduction position related to the reproduction operation input by the operation input means at a timing based on transmission of the auxiliary information and the operation information by the wireless signal transmission means Means,
A video playback device is provided.

  According to a ninth aspect of the present invention, in the eighth aspect of the present invention, the wireless signal transmitting means is configured to transmit a wireless signal by optical wireless transmission using infrared rays. A video playback device is provided.

  The present invention limits the use of radio waves, such as in highly public places, while the video playback apparatus and the audio playback apparatus exist in physically separated locations and remain connected wirelessly. Even in an environment where video data and audio data are synchronized with a simple configuration, the convenience of wireless connection can be maintained, the cost of the device can be reduced, and the size and weight of the device can be reduced. Technology for enabling synchronized playback of video data and audio data contained in one multimedia encoded data in a video playback device and an audio playback device that are physically separated from each other It is applicable to.

In the embodiment of the present invention, it is a block diagram showing an example of the configuration of a multimedia encoded data separation and transmission apparatus and an audio playback apparatus that perform processing according to the previous process stage. In the embodiment of the present invention, it is a block diagram showing an example of a configuration of a multimedia encoded data reproduction device that performs processing related to a post-process stage. In the embodiment of the present invention, it is a diagram showing a first example of the structure of a reconstructed audio stream. In the embodiment of the present invention, it is a figure which shows the 2nd example of the structure of the reconfigure | reconstructed audio stream. It is a block diagram which shows an example of the multimedia coding data reproducing | regenerating apparatus based on a prior art. It is a block diagram which shows an example of the separation type | mold multimedia coding data reproduction apparatus based on a prior art. It is an external view which shows an example of the separation type | mold multimedia coding data reproduction apparatus based on a prior art. It is a figure which shows an example of the data structure of the system stream of MPEG-2 which is a representative of the multimedia system stream in a prior art.

Explanation of symbols

100 Multimedia Encoded Data Separation / Transmission Device 101, 301, 1001, 2101 Multimedia Stream Buffer 102, 302, 1002, 2102 System Stream Separation Unit 103, 202 Audio Stream Buffer 104 Auxiliary Information Buffer 105 Audio Stream Reconstruction Unit 106 Audio Stream Transmitter 200, 2002 Audio playback device 201 Audio stream receiver 203 Post-process processor 204 Auxiliary information / operation information receiver 205, 305, 1005, 2105 Controller 206, 1003, 2103, 2202 Audio decoder 207, 1006, 2106 2203 Audio data buffer 208, 1008, 2204 Speaker 300, 1000 Multimedia encoded data reproduction device 303 Auxiliary information extraction unit 304, 1004, 2104 Video decoding unit 306 Auxiliary information / operation information transmission unit 307, 1007, 2107 Video data buffer 308, 1009, 2109 Monitor 2001 Multimedia encoded data reproduction / transmission device 2110 Audio data transmission unit 2201 Audio data receiver 3001 Main unit 3002 Headphone

Claims (1)

Recording medium on which multimedia encoded data in which video encoded data, audio encoded data, and auxiliary information including time information for managing the reproduction time of each encoded data are multiplexed is recorded From the multimedia encoded data reading means for reading the multimedia encoded data,
Multimedia encoded data separating means for separating and extracting the audio encoded data and the auxiliary information from the multimedia encoded data read by the multimedia encoded data reading means;
Audio reconfiguration for reconstructing the audio encoded data and the auxiliary information separated and extracted by the multimedia encoded data separation means so that audio reproduction related to the audio encoded data is performed in a predetermined audio reproduction device Means,
Said audio encoded data and the auxiliary information reconstructed by the audio reconstruction means, and a data transmission means for transmitting to said predetermined audio reproduction apparatus,
An instruction from the outside regarding the reproduction operation of the multimedia encoded data after the audio encoded data and the auxiliary information reconstructed by the audio reconstructing means are transmitted to the predetermined audio reproducing device is input. Operation input means for
The auxiliary information including time information indicating the reproduction position related to the reproduction operation input by the operation input unit and the operation information for instructing the start of the reproduction operation are wirelessly transmitted to the predetermined audio reproduction device. Wireless signal transmission means for transmitting via
Video decoding means for performing decoding processing after the reproduction position related to the reproduction operation input by the operation input means with respect to the video encoded data included in the multimedia encoded data;
Video display means for visually displaying video based on video data generated by the decoding process of the video encoded data in the video decoding means;
A multimedia encoded data separation / transmission apparatus.