WO2024034389A1

WO2024034389A1 - Signal processing device, signal processing method, and program

Info

Publication number: WO2024034389A1
Application number: PCT/JP2023/027190
Authority: WO
Inventors: 千智劔持; 祐樹松村; 修一郎錦織
Original assignee: ソニーグループ株式会社
Priority date: 2022-08-09
Filing date: 2023-07-25
Publication date: 2024-02-15

Abstract

This technology pertains to a signal processing device, a signal processing method, and a program that make it possible to reduce delay time in processing for adjustment of the bit rate of encoded data. The signal processing device cuts out encoded data in one frame including header information and main information from the head of the header information with a cutout size that is an integer multiple of a cutout unit of N bytes, and adds an end identifier to the end of cutout data which has been cut out. This technology can be applied to a real-time communication reproduction system.

Description

Signal processing device, signal processing method, and program

The present technology relates to a signal processing device, a signal processing method, and a program, and particularly relates to a signal processing device, a signal processing method, and a program that can shorten processing delay time for bit rate adjustment of encoded data.

When transmitting information such as video, audio, tactile sensation, and vibration, it is widely practiced to adjust the compression ratio, that is, the bit rate, depending on the conditions of the communication channel.

For example, Patent Document 1 and Patent Document 2 disclose a method of suppressing sound interruptions by adjusting the bit rate in stages according to the status of the communication channel when transmitting audio data.

Additionally, WebRTC (Web Real-Time Communication) Simulcast is being used as a method for communicating between multiple devices connected via the Internet at the optimal bit rate for each device.

With Simulcast, the distribution side uploads multiple encoded data encoded at multiple bit rates to a relay server (SFU: Selective Forwarding Unit), and the relay server selects the optimal bit rate according to the communication status with each terminal. This is realized by a mechanism that selects and transfers only encoded data. That is, the feature of SFU is that received encoded data is selected and transferred without being processed.

There is also a method using an MCU (Multipoint Control Unit) in which a relay server combines and distributes encoded data received from each terminal. A feature of the MCU is that it once decodes multiple pieces of received encoded data, combines it, encodes it again, and transmits it to each terminal. That is, in the MCU, the bit rate is adjusted by re-encoding the decoded data.

Furthermore, there is a method of using SVC (Scalable Video Coding), which cuts out at the boundaries of the hierarchy by providing a hierarchy to the algorithm. A feature of SVC is that it can extract and process data for each layer. That is, in SVC, bit rate adjustment is performed in stages on a layer-by-layer basis.

JP2009-260931A Japanese Patent Application Publication No. 2016-208215

These bit rate adjustments are mainly realized by encoding. In other words, in order to adjust the bit rate of encoded data, the encoded data must be decoded once and encoded again to achieve the target bit rate, resulting in processing delay time and increased processing load. .

For example, when compressing and recording content, in order to transmit the recorded content over a limited communication band, it is necessary to decode it once and then encode it again. Therefore, if the content is compressed and recorded, a processing delay time will occur and the processing load will be increased compared to the case where the content is recorded uncompressed. Conversely, it is possible to record the content uncompressed, but when recording the content uncompressed, it is necessary to secure a larger recording capacity than when recording the content compressed.

When transmitting data received from the Internet using wireless communication with a narrow communication band such as Bluetooth (registered trademark), even if the same compression technology is used, the data must be decoded and re-encoded to achieve the target bit rate. need to be encoded.

The present technology was developed in view of this situation, and is intended to shorten the processing delay time for adjusting the bit rate of encoded data.

A signal processing device according to one aspect of the present technology includes a cutting unit that cuts out one frame of encoded data including header information and main information in a cutting size that is an integral multiple of a cutting unit of N bytes from the beginning of the header information; and a termination processing section that adds a termination identifier to the end of the cutout data cut out by the cutout section.

In a signal processing device according to another aspect of the present technology, one frame of encoded data including header information and main information is cut out from the beginning of the header information in a cutout size that is an integral multiple of a cutout unit of N bytes, and then transmitted. Based on the leading bit of the extracted data, the extracted data is decoded while detecting a termination identifier added to the end of the extracted data, and information that cannot be decoded at the time when the termination identifier is detected is decoded as zero. It includes a decoding section.

In one aspect of the present technology, one frame of encoded data including header information and main information is cut out from the beginning of the header information in a cutout size that is an integral multiple of a cutout unit of N bytes, and the cutout data is A termination identifier is added to the termination.

In another aspect of the present technology, one frame of encoded data including header information and main information is cut out and transmitted in a cutout size that is an integral multiple of a cutout unit of N bytes from the beginning of the header information. Based on the leading bit of the data, the cut-out data is decoded while detecting a termination identifier added to the end of the cut-out data, and information that cannot be decoded at the time of detecting the termination identifier is decoded as zero.

FIG. 1 is a diagram illustrating a configuration example of a real-time communication playback system according to a first embodiment of the present technology. FIG. 2 is a diagram illustrating an example of a format of encoded data according to the present technology. FIG. 3 is a diagram illustrating an example of initial information regarding cutting out encoded data. FIG. 2 is a block diagram showing a configuration example of a transmitter and a receiver. It is a flow chart explaining processing of a transmitter. 6 is a flowchart illustrating the cutout setting process in step S11 of FIG. 5. FIG. It is a flow chart explaining processing of a receiver. 8 is a flowchart illustrating the decoding process in step S56 of FIG. 7. FIG. FIG. 2 is a block diagram illustrating a configuration example of a real-time communication playback system according to a second embodiment of the present technology. 3 is a flowchart illustrating main processing of a transmitter. 3 is a flowchart illustrating sub-processing of a transmitter. It is a flow chart explaining processing of a receiver. 13 is a flowchart illustrating the process of writing encoded data in step S158 of FIG. 12. FIG. 13 is a flowchart illustrating the process of writing out remaining encoded data in step S162 of FIG. 12. FIG. FIG. 3 is a block diagram showing a configuration example of a real-time recording communication playback system according to a third embodiment of the present technology. 3 is a flowchart illustrating processing of an encoder. FIG. 12 is a block diagram illustrating a configuration example of a real-time communication system according to a fourth embodiment of the present technology. 1 is a block diagram showing an example of the configuration of a computer. FIG.

Hereinafter, a mode for implementing the present technology will be described. The explanation will be given in the following order.
1. First embodiment 2. Second embodiment 3. Third embodiment 4. Fourth embodiment 5. others

<1. First embodiment>
<Configuration of real-time communication playback system>
FIG. 1 is a diagram illustrating a configuration example of a real-time communication playback system according to a first embodiment of the present technology.

The real-time communication playback system 1 in FIG. 1 is configured to include a transmitter 11 and a receiver 12.

The transmitter 11 transmits at least a portion of audio encoded data (hereinafter simply referred to as encoded data) recorded in a storage area (not shown) to the receiver 12.

The receiver 12 receives at least a portion of the encoded data and reproduces it in real time.

Note that the communication in the first embodiment may be long-distance communication via the Internet, short-distance communication such as Bluetooth, or intra-device communication. Although the data to be transmitted may be information such as video, audio, tactile sensation, or vibration, the first embodiment will be described using audio data such as music as an example.

Furthermore, in the first embodiment, it is assumed that instructions to start and stop reproduction are exchanged between the transmitter 11 and the receiver 12. These "notes" in the first embodiment are the same in the subsequent second to fourth embodiments.

Furthermore, it is assumed that the transmitter 11 and the receiver 12 are ready to transmit encoded data in the format shown in FIG. 2, which will be described later, and to reproduce it in real time. For example, in the case of Bluetooth A2DP (Advanced Audio Distribution Profile), preparation for real-time playback includes determining the type of audio codec used for communication, the sampling frequency, etc. Note that the format of the encoded data may be determined as one of the specifications of the audio codec in preparation for real-time playback.

<Format of encoded data>
FIG. 2 is a diagram illustrating an example of the format of encoded data according to the present technology.

The encoded data in FIG. 2 is a data string obtained by encoding audio data in units of frames, which are divided by a time length such as 10 milliseconds.

The encoded data of each frame is composed of header information, sub information, and main information.

The header information includes, for example, information indicating the data size of the frame (hereinafter referred to as frame size) and sampling frequency.

The sub-information includes, for example, outline information of the signal to be restored. The general shape information of the signal to be restored is information that allows the general shape of the audio data to be restored or estimated. For example, information for restoring the time waveform envelope or frequency spectrum envelope of the audio data, or information for restoring the time waveform envelope or frequency spectrum envelope of the audio data, or This is information such as the scale factor of the frequency band.

The main information includes information for restoring the fine structure of the audio data, which is the signal to be restored, such as an encoded time waveform signal and a signal for each frequency bin.

Additionally, information is arranged in the main information in descending order of importance. For example, when the main information is audio data, audio information is arranged from low-frequency audio information with high importance to high-frequency audio information. For example, if the main information is hierarchically encoded information such as SVC, the hierarchically encoded information is arranged in order from the first layer information with the highest degree of importance. Note that the information is not limited to the above-mentioned example, but the information may be arranged in descending order of importance, or the information may be arranged in a manner that is not particularly linked to importance. .

In the following, to simplify the explanation, it is assumed that the sub information and the main information each consist of only information about the frequency spectrum. It may be composed of both information regarding the waveform.

Furthermore, encoded data has a feature in which bits with a value of "0" appear at regular intervals. The interval at which bits with a value of "0" appear (hereinafter referred to as the first bit appearance interval) may be predetermined by, for example, a standard. Alternatively, a recording area may be provided for header information, etc., and information regarding the leading bit appearance interval may be included in the recording area and transmitted from the transmitting side to the receiving side.

In FIG. 2, the first bit appearance interval is set to "N" bytes. Note that the unit is not limited to bytes. The first bit appearance interval N bytes is, for example, 20 bytes or 32 bytes. In the following examples, unless otherwise specified, the first bit appearance interval N is described as 32 bytes, but it may take a value other than this.

Regarding the end of the encoded data, data may be appropriately packed so that the data size is an integral multiple of the leading bit appearance interval N bytes, but in Figure 2, the encoded data when no data is packed at the end An example is shown.

As described above, the encoded data shown in FIG. 2 is characterized in that it is cut out and used in a size that is an integral multiple of the leading bit appearance interval N bytes.

Note that in FIG. 2, the first bit appearance interval N bytes is always constant, but it may be possible to change it midway. For example, the leading bit appearance interval N bytes may be set to 32 bytes for the first 1 to 128 bytes, and 64 bytes for the next 129 to 256 bytes. Note that if the leading bit appearance interval N bytes can be changed midway, for example, the leading bit appearance interval N bytes is predefined in a standard, or the leading bit appearance interval N bytes is recorded in the header information. It may also be transmitted together with encoded data.

As a result, for example, if data with high importance is placed at the front in the main information, it is possible to transmit the data with higher priority and avoid sending data with low importance.

<Initial information regarding extraction of encoded data>
FIG. 3 is a diagram illustrating an example of initial information regarding cutting out encoded data.

In FIG. 3, as an example of initial information regarding the extraction of encoded data, the extraction unit and the maximum size and minimum size at the time of extraction are shown.

The extraction unit is set to N bytes (=leading bit appearance interval N (N bytes), and the extraction size is set to an integral multiple of the extraction unit (N bytes).

The minimum extraction size is an integral multiple of the extraction unit, and is the minimum size that includes all header information and sub information.

The maximum extraction size is an integral multiple of the extraction unit, and is the minimum size that includes all encoded data of one frame. In the case of FIG. 3, the minimum size is 2N bytes and the maximum size is 6N bytes.

<Transmitter and receiver configuration>
FIG. 4 is a block diagram showing a configuration example of the transmitter 11 and receiver 12 in FIG. 1.

Upon receiving the playback start instruction, the transmitter 11 generates cutout setting information, which will be described later, and cuts out encoded data (cutout data) to be transmitted to the receiver 12 from one frame of encoded data according to the cutout setting information. , performs termination processing on the cut out encoded data. The transmitter 11 collects the terminated encoded data into a data chunk to be transmitted at once (that is, encoded data in the format shown in FIG. 2), and transmits the data to the receiver 12.

The transmitter 11 in FIG. 4 is configured to include a cutout setting determination section 21, an encoded data cutout section 22, a termination processing section 23, a transmission data configuration section 24, and a transmission processing section 25.

The cutout setting determination unit 21 obtains information necessary for determining cutout setting information in advance from a receiving system (not shown), and determines the range of setting values that can be taken in the current playback.

When playback is started, the cutout setting determining unit 21 adjusts the cutout setting information based on information such as the communication status that is intermittently received from the reception system, and sends the adjusted cutout setting information to the encoded data cutout unit 22. and is output to the transmission data configuration section 24.

The cutout setting information is configured to include, for example, the cutout size of encoded data Cut _SZ and the number F _NUM of frames (frames collected into one packet) to be transmitted in one communication. Hereinafter, the pair of frame number and cutout size will be expressed as <F _NUM , Cut _SZ >.

The information that the cutout setting determination unit 21 receives in advance includes, for example, the maximum data size to be transmitted in one communication, information on the type and size of packets used for communication, the number of frames that can be combined into one packet, and the permissible number of frames. There is information such as a delay time, default cutout setting information, and cutout specifications regarding encoded data to be reproduced (for example, initial information regarding cutout in FIG. 3).

The range of setting values that the cutout setting information can take is determined based on information received in advance. For example, if the packet size is 200 bytes and the number of frames that can be combined into one packet is 3, if N is 32 bytes, then <1, 2N to 6N (2N, 3N, 4N, 5N, 6N )>, <2, 2N and 3N>, <3, 2N>. When the packet size is 150 bytes and the number of frames that can be combined into one packet is 3, the calculations are <1, 2N to 4N> and <2, 2N>.

Note that if the allowable delay time is shorter than the time for two frames, only the above-mentioned pattern in which F _NUM is 1 is possible.

Furthermore, for the sake of simplicity, the following explanation does not specifically mention the data size added in the termination process, but in reality, processing is performed taking this data size into consideration.

The cutout setting information is determined by analyzing trends in information such as the communication status of the network, the processing load of the transmitter 11, and resource availability information of the receiver 12, and is adaptively determined based on the analysis results. Alternatively, the cutout setting information is determined according to settings selected by the user.

When determining the cutout setting information adaptively, for example, default cutout setting information is used for the first cutout setting information, and subsequent cutout setting information is determined based on the estimated communication situation.

For estimating the communication status, reference is made to, for example, the amount of data waiting to be sent in the transmission processing unit 25, the moving average value of this amount, the retransmission rate of data in the last few seconds, the error rate, etc. For example, if all of them show high values, it is assumed that the communication situation is poor. If only the retransmission rate shows a high value, it is presumed that the communication situation is deteriorating. If only the data quantity shows a high value, it is presumed that the communication situation is improving. If all of them show low values, it is assumed that the communication status is good.

The estimating time from observing the communication status to estimating it may always be constant. For example, the estimation time for estimating a worsening trend or bad is set short, and the estimating time for estimating an improving trend or good is set long. good.

Further, the processing load of the transmitter 11 is estimated by referring to, for example, the CPU (Central Processing Unit) usage rate, the remaining battery level, etc. Therefore, the cutout setting information is adaptively determined based on the CPU usage rate, remaining battery level, and the like.

If the remaining battery level is below 20%, for example, in order to reduce transmission power, the extraction setting information may be set to increase the number of frames transmitted at one time, or the F _NUM of the selectable extraction setting information may be set. For example, restrictions may be placed on the cutout setting information, such as limiting the number to 3 or more. Note that cutout setting information for reducing power consumption may be prepared separately.

For example, when the communication situation is poor, the communication situation is deteriorating, or the processing load is high, the cutout setting information is determined to lower the bit rate and increase the number of retransmissions. On the other hand, when the communication situation is good, or when the communication situation is improving, and when the processing load is low, the cutout setting information is determined to increase the bit rate.

For example, if the default cutout setting information is <1, 4N> and it is determined that the communication situation is in a deteriorating trend, the cutout setting information is updated to <2, 3N>. Conversely, if it is determined that the communication situation is good, the cutout setting information is updated to <1, 5N>.

Furthermore, when determining the cutout setting information according to the settings selected by the user, options such as "Sound quality priority" which gives priority to the maximum bit rate and "Connection priority" which gives priority to connection stability are provided at the time of design. , the relationship between the settings of the cutout setting information and the options presented to the user is determined in advance. Then, the cutout setting information is set according to the option selected by the user before or during communication.

The encoded data cutout section 22 reads only the encoded data corresponding to Cut _SZ of the cutout setting information supplied from the cutout setting determination section 21 out of the encoded data of one frame from the storage area, and sends it to the termination processing section 23. Output. That is, in the first embodiment, encoded data other than the encoded data corresponding to Cut _SZ of the cutout setting information is not read out or transmitted to the receiver 12.

The termination processing unit 23 adds a termination identifier, which is, for example, 1-byte data in which all bits are 1, to the encoded data (cutout data) supplied from the encoded data extraction unit 22, and adds the termination identifier to the encoded data (cutout data) supplied from the encoded data extraction unit 22. Output to 24.

The transmission data configuration unit 24 temporarily holds the encoded data supplied from the termination processing unit 23 until the number of encoded data equal to F _NUM of the extraction setting information supplied from the extraction setting determination unit 21, and When the number reaches , the data are collectively output to the transmission processing section 25.

However, as described above, since F _NUM and Cut _SZ may be updated midway through, the timing of supplying them to the transmission processing unit 25 is changed depending on the updated contents. For example, when F _NUM is 1, the transmission data configuration section 24 immediately outputs the encoded data to the transmission processing section 25. When F _NUM is 2, the transmission data configuration section 24 receives two frames of encoded data, concatenates the two frames of encoded data into one, and outputs it to the transmission processing section 25 .

Further, when F _NUM is updated from 3 to 2, and the encoded data prepared with F _NUM 3 is held, the transmission data configuration unit 24 updates the stored encoded data. The data is immediately output to the transmission processing unit 25, and preparation of encoded data with F _NUM of 2 is newly started.

Conversely, F _NUM of 3 is updated to 4, and the coded data that was prepared with F _NUM of 3 is retained, and the coded data to be newly concatenated is changed to the prepared coded data. When concatenated with data, the packet size may exceed the size. In this case, the transmission data configuration unit 24 immediately outputs the encoded data it holds to the transmission processing unit 25, and newly starts preparing encoded data with F _NUM of 4.

If the packet size is not exceeded, the transmission data configuration unit 24 continues preparation until F _NUM reaches 4, and outputs the encoded data to the transmission processing unit 25 at the timing when F NUM reaches 4. The encoded data chunk supplied to the transmission processing section 25 is transmitted to the receiver 12.

When the receiver 12 receives a chunk of encoded data from the transmitter 11, it stores it in a temporary storage area (hereinafter referred to as a reception buffer) not shown. When a certain number of stored encoded data or a certain data size are accumulated, the receiver 12 reads one frame of encoded data from the reception buffer, decodes it, dequantizes it, synthesizes audio data, It is reproduced by a speaker or the like using an output system (not shown).

The receiver 12 in FIG. 4 is configured to include a bitstream reading section 31, a decoding section 32, a dequantizing section 33, and a signal combining section 34.

The bitstream reading unit 31 reads one frame worth of encoded data from the reception buffer, and combines the read encoded data with the data size (=data size after cutting out) not including the termination identifier added by the termination processing unit 23. ) is output to the decoding unit 32.

In the header information of encoded data in this embodiment, the frame size of one original frame (data size before cutting out) is recorded. Note that the data size after extraction may be included in the header information. Hereinafter, a reading method when the data size after extraction is not included in the header information will be explained.

As described above, encoded data is characterized in that a bit with a value of "0" appears every N bytes in the cutout unit (first bit appearance interval). Furthermore, due to the termination process after the extraction, a termination identifier with a value of "1" is added to the first bit (hereinafter also referred to as the beginning bit) of the encoded data for each extraction unit.

That is, the bitstream reading unit 31 determines whether the read data is data within one frame by checking that the first bit is "0" every time N bytes are read. Therefore, when the first bit becomes "1", the bit stream reading unit 31 determines that it is the end identifier, and can read forward to the beginning of the next frame. In this embodiment, since the end identifier is 1 byte, the next byte becomes the beginning of the next frame.

By reading as described above, it is possible to read out one frame's worth of extracted encoded data without including the data size after extraction in the header information.

Note that it is only necessary that the first bit of the extraction unit and the first bit of the terminal identifier can be distinguished, and the value of the first bit of the extraction unit may be set to "1" and the value of the first bit of the terminal identifier may be set to "0". Further, both the first bit of the extraction unit and the first bit of the terminal identifier may take values or symbols other than "0" or "1".

The decoding unit 32 decodes the encoded data supplied from the bitstream reading unit 31 while skipping bits with a value of “0” at every leading bit appearance interval N, and generates a decoded signal and decoded range information ( Thereafter, the decoding range information) is output to the inverse quantization unit 33. For example, the decoding unit 32 processes information that can be correctly decoded within the data size range supplied from the bitstream reading unit 31, and outputs, for example, zero for the remaining information that cannot be decoded.

For example, if the supplied data size is 100 bytes, and the 99th byte and 4th bit from the beginning are decoded, and 7 bits are required for the next decoding, the decoding unit 32 detects that the number of bits necessary for decoding is insufficient. It is determined that there is no such value, and zero is output from then on. For example, when the 100th frequency spectrum is decoded, 100 is output as the decoding range information.

The dequantization unit 33 dequantizes the decoded signal supplied from the decoding unit 32, and generates a frequency spectrum obtained by dequantization and sub-information such as supplied decoding range information and Scale Factor of each frequency band. and is output to the signal synthesis section 34. Inverse quantization is, for example, a process of returning the value of the supplied frequency spectrum using Scale Factor information of each frequency band. For example, if the Scale Factor is 128 and the decoded signal is 0.75, the value after dequantization, that is, the value of the original frequency spectrum, is calculated as 96, which is the sum of these values.

The signal synthesis unit 34 synthesizes the audio signal using sub-information such as decoding range information and frequency band Scale Factor information supplied from the dequantization unit 33, and the frequency spectrum, and synthesizes the audio signal to a subsequent processing block (not shown). Output to. The subsequent processing block performs audio signal processing such as equalization on the supplied audio signal, and reproduces it through a speaker or the like.

The audio signal is synthesized by interpolating the frequency spectrum that is not included in the decoding range information and then performing frequency-time conversion. Interpolation in the frequency domain can be performed using, for example, the Scale Factor information of a frequency band that is not included in the decoding range information, the decoded frequency spectrum, and the Scale Factor information of the frequency band that includes this frequency spectrum. This is done by multiplying the frequency spectrum by a ratio. Alternatively, for example, a frequency spectrum that is not included in the decoding range information may be estimated from previously decoded information using AI (Artificial Intelligence) learned in advance. As described above, the currently decrypted information and previously decrypted information may be used together.

Note that in this embodiment, as described above, in order to simplify the explanation, the information to be sent is related to the frequency spectrum, so an example of interpolation in the frequency domain is shown, but information related to the time waveform may also be sent. Can be done. For example, when transmitting envelope waveform information after band-dividing a time waveform, interpolation may be additionally performed after frequency-time conversion.

<Transmitter processing>
FIG. 5 is a flowchart illustrating the processing of the transmitter 11.

In the transmitter 11, when playback is started, the process shown in FIG. 5 is activated intermittently, and the number of frames according to the activation time interval is processed. For example, playback of encoded data in which one frame is 10 ms is started, and the processing shown in FIG. 5 is started at approximately 50 ms intervals such as 0 ms, 48 ms, 105 ms, and 140 ms. In this case, five frames are processed at each startup. Furthermore, for the sake of simplicity, the prepared encoded data in the figure is data used by the encoded data cutout section 22, and is cleared to 0 at the start of reproduction.

In step S11 of FIG. 5, the cutout setting determination unit 21 performs cutout setting determination processing. The cutout setting determination process will be described later with reference to FIG. Through the process in step S11, <F _NUM , Cut _SZ > is calculated.

In step S12, the encoded data cutting unit 22 determines whether or not the prepared encoded data plus the encoded data calculated for Cut _SZ fits within the maximum transmission size for one time. Note that "+1" in the figure indicates the data size of "0xFF" added in the termination process.

If it is determined in step S12 that the prepared encoded data plus the calculated Cut _SZ worth of encoded data fits within the maximum transmission size for one time, the process proceeds to step S13.

In step S13, the encoded data cutting unit 22 reads the encoded data for the calculated Cut _SZ from the beginning of the frame and writes it to the end of the prepared encoded data.

In step S14, the termination processing unit 23 writes "0xFF" to the end of the prepared encoded data.

In step S15, the transmission data configuration unit 24 determines whether the number of prepared frames, which is the number of frames written in the prepared encoded data, is less than the calculated F _NUM .

If it is determined in step S15 that the number of prepared frames is not less than the calculated F _NUM , that is, if it is determined that the number of prepared frames has reached the calculated F _NUM , the process proceeds to step S16. Proceed to.

Further, if it is determined in step S12 that the coded data calculated for Cut _SZ added to the prepared coded data does not fit within the maximum transmission size for one time, the process continues in step S16. Proceed to.

In step S16, the transmission data configuration unit 24 outputs the prepared encoded data to the transmission processing unit 25.

In step S17, the transmission data configuration unit 24 clears the information on the prepared encoded data. After that, the process proceeds to step S18.

On the other hand, if it is determined in step S15 that the number of prepared frames is less than the calculated F _NUM , the process also proceeds to step S18.

In step S18, the transmission data configuration unit 24 determines whether or not the number of frames to be processed has been completed at the current startup timing. If it is determined in step S18 that the amount to be processed has not been completed at the current startup timing, the process returns to step S12, and the subsequent processes are repeated.

In step S18, if it is determined that the processing to be processed at the current startup timing has been completed, the processing of the transmitter 11 in FIG. 5 ends.

<Cutout setting determination process>
FIG. 6 is a flowchart illustrating the extraction setting determination process in step S11 of FIG.

In step S31 of FIG. 6, the cutout setting determining unit 21 estimates (estimates) the communication status.

In step S32, the cutout setting determining unit 21 estimates (estimates) the processing load.

In step S33, the cutout setting determining unit 21 performs a trend analysis.

In step S34, the cutout setting determining unit 21 determines whether the communication situation is bad or is on a downward trend. If it is determined in step S34 that the communication situation is not bad and that the communication situation is not in a deteriorating trend, the process proceeds to step S35.

In step S35, the cutout setting determining unit 21 determines whether the processing load is high. If it is determined in step S35 that the processing load is high, the process proceeds to step S36.

Furthermore, if it is determined in step S34 that the communication situation is poor or that the communication situation is on the decline, the process also proceeds to step S36.

In step S36, the cutout setting determining unit 21 performs processing to improve the success rate of communication. In FIG. 6, for example, the number of frames sent in one transmission is increased. This makes it possible to increase the number of retransmissions in the event of failure. That is, when the number of frames sent in one transmission is increased, the transmission interval becomes wider, and by widening the transmission interval, time for retransmission can be acquired. Specifically, if one frame takes 1 second and one transmission takes 0.5 seconds, if the number of frames to be sent at one time is 5, it will be sent at 5 second intervals, which will reduce the retransmission time. is 4.5 seconds, which is obtained by subtracting 0.5 seconds of the first transmission time from 5 seconds, which is 9 times when converted to the number of times. If this is increased by one frame, it will be possible to retransmit for 5.5 seconds, which will be 11 times when converted to the number of retransmissions.

Specifically, in step S36, the cutout setting determining unit 21 increases F _NUM by one. However, since the maximum value of F _NUM must not exceed F _{NUM_MAX} calculated at the time of initialization, the cutout setting determining unit 21 determines the final value using the MIN function that takes the minimum value of the two inputs (F _NUM = MIN(F _NUM +1, F _{NUM_MAX} )).

In step S37, the cutting setting determining unit 21 sets Cut _SZ based on the calculated F _NUM . For example, Cut _{SZ_MIN} , which is the minimum value that Cut _SZ can take at the calculated F _NUM , is set to Cut _SZ (Cut _SZ = Cut _{SZ_MIN} @F _NUM ).

For example, if <F _NUM , Cut _SZ > calculated during the initialization described above is <1, 2N to 6N>, <2, 2N or 3N>, <3, 2N>, F _{NUM_MIN} is 1, F _{NUM_MAX} becomes 3. For example, when F _NUM is 2, Cut _{SZ_MIN} is 2N and Cut _{SZ_MAX} is 3N.

After step S37, the cutout setting determination process in FIG. 6 ends.

On the other hand, if it is determined in step S35 that the processing load is low, the process proceeds to step S38.

In step S38, the cutout setting determining unit 21 determines whether the favorable situation has continued for a specified period of time. If it is determined in step S38 that the favorable situation has continued for the specified time, the process proceeds to step S39.

In step S39, the cutting setting determining unit 21 calculates the new Cut _SZ to be the current Cut _SZ +N (Cut _SZ =Cut _SZ +N).

In step S40, the cutout setting determining unit 21 determines whether the calculated Cut _SZ exceeds the maximum Cut _SZ that can be obtained by the current F _NUM , that is, Cut _{SZ_MAX} (Cut _SZ > Cut _{SZ_MAX} @F _NUM ). .

If it is determined in step S40 that the calculated Cut _SZ exceeds the maximum Cut _SZ that can be taken by the current F _NUM , that is, Cut _{SZ_MAX} , the process proceeds to step S41.

In step S41, the cutout setting determining unit 21 updates F _NUM in the direction of decreasing it by one (F _NUM = MAX (F _NUM -1, F _{NUM_MIN} )).

In step S42, the cutout setting determining unit 21 compares Cut _SZ with the minimum Cut _SZ that can be obtained with the calculated F _NUM , and sets the larger value to Cut _SZ (Cut _SZ = MAX (Cut _SZ , Cut _{SZ_MIN} @F _NUM )). After step S42, the cutout setting determination process in FIG. 6 ends.

On the other hand, if it is determined in step S38 that the favorable situation has not continued for the specified time, or in step S40, the calculated Cut _SZ exceeds the maximum Cut _SZ possible with the current F _NUM , that is, Cut _{SZ_MAX} . Even if it is determined that there is no cutout setting determination process in FIG. 6, the cutout setting determination process in FIG. 6 ends.

<Receiver processing>
FIG. 7 is a flowchart illustrating the processing of the receiver 12.

In the receiver 12, when playback is started, the process shown in FIG. 7 is intermittently called, and one frame of audio data is generated.

In step S51 of FIG. 7, the bitstream reading unit 31 sets the data size after extraction (Data _SZ ) to zero (Data _SZ = 0), and obtains one frame's worth of received data from the reception buffer.

In step S52, the bitstream reading unit 31 obtains one byte from the received data.

In step S53, the bitstream reading unit 31 determines whether the first bit is 0 or not. If it is determined in step S53 that the first bit is 0, the process proceeds to step S54.

In step S54, the bitstream reading unit 31 obtains N-1 bytes from the received data.

In step S55, the bitstream reading unit 31 updates Data _SZ by the acquired data size (Data _SZ +=N).

After step S55, the process returns to step S52, and the subsequent processes are repeated.

On the other hand, if it is determined in step S53 that the first bit is not 0, the process proceeds to step S56.

In step S56, the decoding unit 32 performs decoding processing. Details of the decoding process will be described later with reference to FIG.

In step S57, the dequantization unit 33 performs dequantization processing.

In step S58, the signal synthesis unit 34 performs waveform synthesis processing. Audio data is synthesized by waveform synthesis processing and output to the subsequent stage. After step S58, the processing of the receiver 12 in FIG. 7 ends.

FIG. 8 is a flowchart illustrating the decoding process of step S56 in FIG. 7.

In the decoding process shown in FIG. 8, the number of used bits (BN _Used ), the number of bits required to decode the next sample (BN _Next ), and the bit conversion value of the supplied encoded data size (BN _Total = Data _SZ *8) is compared, and processing is continued only if there is the number of bits necessary for decoding processing.

In step S71 of FIG. 8, the decoding unit 32 sets BN _Used to 0 (BN _Used = 0).

In step S72, the decoding unit 32 obtains BN _Next , which is the number of bits required for the next decoding.

Note that BN _Next may be a value predefined by the standard, or information indicating the value of BN _Next may be recorded in the encoded data, but in this embodiment, the explanation will be simplified. It is assumed that the value is predefined in the standard for the purpose of Further, even if information indicating the value of BN _Next is recorded in the encoded data, the same procedure as for reading the bits for BN _Next , which will be described later, can be performed.

In step S73, the decoding unit 32 determines whether or not a cutting unit (N byte boundary) is not crossed when reading the BN _Next bits. If it is determined in step S73 that the N byte boundary is not crossed, the process proceeds to step S74.

In step S74, the decoding unit 32 determines whether or not BN _Total will not be exceeded even if the bits corresponding to BN _Next are further read out (BN _Used +BN _Next <BN _Total ). If it is determined in step S74 that the BN _Total will not be exceeded even if BN _Next bits are further read, the process proceeds to step S75.

In step S75, the decoding unit 32 actually reads out the bits corresponding to BN _Next and decodes one value.

In step S76, the decoding unit 32 updates BN _Used by the number of bits corresponding to BN _Next (BN _Used +=BN _Next ).

After step S76, the process proceeds to step S83.

On the other hand, if it is determined in step S73 that the N byte boundary is crossed when reading the BN _Next bits, the process proceeds to step S77.

In step S77, the decoding unit 32 determines whether or not BN _Total will not be exceeded even if the 1 bit of 0 that appears every N bytes is further read out in addition to the bits for BN _Next (BN _Used +BN _Next +1< BN _Total ).

If it is determined in step S77 that the BN _Total is not exceeded, the process proceeds to step S78.

In step S78, the decoding unit 32 calculates the number of bits up to the N-byte boundary (BN _Read ).

The calculation is performed, for example, by finding the maximum value that does not exceed BN _Used + BN _Next as a multiple of N*8, and taking the difference from BN _Used , as shown in the following equation (1).

In step S79, the decoding unit 32 reads out the calculated bits of BN _Read .

In step S80, the decoding unit 32 skips 1 bit in order to skip "0" appearing on the N-byte boundary.

In step S81, the decoding unit 32 further reads the remaining necessary bits BN _Next - BN _Read .

In step S82, the decoding unit 32 updates BN _Used with the total number of read bits (BN _Used +=BN _Next +1).

In step S83, the decoding unit 32 determines whether decoding of all values has been completed. If it is determined in step S83 that the decoding of all values has not been completed yet, the process returns to step S72 and the subsequent processes are repeated.

If it is determined in step S83 that decoding of all values has been completed, the process proceeds to step S84.

Further, if it is determined in step S77 that BN _Total is exceeded, the process also proceeds to step S84.

In step S84, the decoding unit 32 zero-clears the remaining information that could not be decoded. After step S84, the decoding process in FIG. 8 ends.

As described above, in the first embodiment, bit rate adjustment (change) can be realized by cutting out encoded data.

<2. Second embodiment>
<Configuration of real-time communication playback system>
FIG. 9 is a diagram illustrating a configuration example of a real-time communication playback system according to the second embodiment of the present technology.

Note that in the case of the first embodiment, the encoded data that remained after the extraction (that is, the untransmitted encoded data) was not transmitted and remained as it was, but in the case of the second embodiment, the In this case, the encoded data remaining after the extraction is transmitted using the idle communication time.

In addition, on the receiving side of the second embodiment, the encoded data is played back in real time and recorded, so that it can be played back later, and the recorded encoded data also includes communication free space. The encoded data that remains after being cut out is added (integrated), and is transmitted using time.

The real-time communication playback system 101 in FIG. 9 is configured to include a transmitter 111 and a receiver 112.

The transmitter 111 transmits to the receiver 112 at least part of the encoded data in the format shown in FIG. 2, which is recorded in advance in the storage area. In addition, the transmitter 111 utilizes the idle communication time to transmit the remaining encoded data, which is the encoded data that remains after cutting out, if there is encoded data that has not been transmitted (hereinafter referred to as untransmitted encoded data). and transmits it to the receiver 112.

Specifically, upon receiving a playback start instruction, the transmitter 111 generates cutout setting information, and according to this information, cuts out one frame's worth of encoded data, performs termination processing, and divides the data to be transmitted at once. The information is collected into chunks and output to the transmission processing section 25. At this time, information on the encoded data that has been extracted and terminated is saved as transmission history information.

Further, the transmitter 111 refers to the transmission history information and configures the remaining encoded data based on the untransmitted encoded data, for example, while playback is stopped or during a gap between main data communication, and the receiver 112 Send to.

Furthermore, if the transmitter 111 finds that the transmission of encoded data could not be completed due to a communication error or the like, it updates the history of the corresponding frame in the transmission history information and indicates that the encoded data that could be transmitted is 0. Leave a history of what happened. This is to retransmit all the data even if only a portion of the data is successful, but it is also possible to leave a history of failures in transmission and retransmit only the portions that could not be transmitted.

The receiver 112 receives at least a portion of the encoded data and records it while playing it in real time. Further, the receiver 112 receives the remaining encoded data transmitted using the idle communication time, and records the received remaining encoded data in addition to the recorded encoded data. Thereby, the quality of the recorded content can be kept higher.

Specifically, upon receiving data transmitted from the transmitter 111 (hereinafter referred to as received data), the receiver 112 determines whether it is residual encoded data, and determines whether it is residual encoded data or not. If it is data, the encoded data is stored in the reception buffer. The receiver 112 reads out one frame's worth of encoded data from a reception buffer (temporary storage area) when a certain number of stored encoded data or a certain data size has been accumulated. Then, the receiver 112 records the encoded data in the storage area, simultaneously decodes, dequantizes, and synthesizes the audio data, and reproduces the data through a speaker or the like through an output system.

Furthermore, when the received data is residual encoded data, the receiver 112 records it in addition to the encoded data recorded in the storage area.

The storage area for recording encoded data is, for example, a long-term recording device such as a hard disk drive. That is, after that, the user can reproduce the encoded data at a desired timing without receiving the encoded data again from the transmitter 111.

The configurations of the transmitter 111 and receiver 112 will be described below. Note that in FIG. 9, parts corresponding to those in FIG. 3 are designated by corresponding symbols, and detailed explanation thereof will be omitted.

The transmitter 111 in FIG. 9 has the following points: the encoded data extraction unit 22 is replaced with an encoded data extraction unit 121, and an untransmitted encoded data information storage unit 122 and a remaining encoded data configuration unit 123 are added. However, it is different from the transmitter 11 in FIG.

That is, the transmitter 111 includes an extraction setting determining section 21, a termination processing section 23, a transmission data configuration section 24, an encoded data extraction section 121, an untransmitted encoded data information storage section 122, and a remaining encoded data configuration section 123. Constructed to include.

The encoded data cutout section 121 reads only the encoded data corresponding to Cut _SZ of the cutout setting information supplied from the cutout setting determination section 21 out of the encoded data of one frame from the storage area.

The encoded data cutting unit 121 outputs the read encoded data and the content ID, which is index information that can uniquely identify the read-out file, to the termination processing unit 23, and at the same time outputs the content ID, frame number, and Cut data. _SZ is output to the untransmitted encoded data information storage section 122. Note that one content ID corresponds to one or more frame numbers.

The untransmitted encoded data information storage unit 122 stores the supplied frame number and Cut _SZ as a pair in a storage area (not shown) as transmission history information. The transmission history information is recorded, for example, in a list format so that it can be added for each content ID. In this case, the untransmitted encoded data information storage unit 122 records the transmission history information in a format that includes at least the frame number and Cut _SZ in the elements of the list.

The remaining encoded data configuration unit 123 starts up intermittently, refers to the above-mentioned transmission history information, and when there is untransmitted encoded data and the main data communication is not being performed, for example, when playback is stopped. The remaining encoded data is generated during a gap time between main data communication and the like, and the remaining encoded data is output to the transmission processing section 25.

Specifically, the remaining encoded data configuration unit 123 first refers to, for example, the oldest transmission history information, and reads out the encoded data of the portion larger than Cut _SZ of the encoded data of the corresponding frame number. The read size is changed according to the communication situation. For example, the read size is set to N bytes in the gap time of main data communication, and is set to the size up to the end of the frame if playback is stopped.

Then, the remaining encoded data configuration unit 123 adds header information that can be distinguished from the transmission data configured by the transmission data configuration unit 24 to the read encoded data, content ID, frame number, and read position information (how many N bytes). , and the read size, configure and transmit the remaining encoded data. At the same time, the remaining encoded data configuration unit 123 also updates transmission history information.

For example, a value predetermined by the standard, such as a 1-byte data string in which the first bit is 1, such as 0xFF, is used as header information for distinguishing between transmission data and remaining encoded data.

As an example of updating the transmission history information, for example, when Cut _SZ of frame number 10 is initially 3N bytes, when the remaining encoded data configuration unit 123 reads N bytes of data, Cut _SZ of frame number 10 is updated. is updated to 4N bytes. In addition, if all the encoded information has been successfully sent, in order to easily determine this, for example, set Cut _SZ of the relevant frame number to -1 or delete the transmission history information of the relevant frame number. You may also do this. By updating the transmission history information as described above, encoded data of frames with the smallest frame number can be transmitted in order without omission or duplication.

The remaining encoded data configuration unit 123 also records transmission history information in a list format, and, for example, moves the updated history to the end of the list and transmits frames in which all untransmitted encoded data has been transmitted. The update rule is to delete history information from the list. For example, the residual encoded data configuration unit 123 transmits the 10th frame number by N bytes, the 11th frame number by N bytes, and so on in order in minimum units, so that the remaining encoded data is averaged for all frames. Data can be sent.

In the following, for the sake of simplicity, an example will be described in which N bytes of data are sent for each frame. Furthermore, since there may be less than N at the end of the frame, the size of the remaining encoded data to be transmitted is also sent together. Note that although the second embodiment has a configuration with one receiver, a configuration with a plurality of receivers may be used, and in this case, transmission history information may be saved for each receiver.

The receiver 112 in FIG. 9 differs from the receiver 12 in FIG. 4 in that a received data sorting section 131 and a bitstream writing section 132 are added.

The received data sorting unit 131 analyzes the header information of the received data, determines whether it is residual encoded data, and if it is residual encoded data, outputs it as is to the bitstream writing unit 132. If the received data is not the remaining encoded data, the received data sorting unit 131 stores it in the reception buffer.

For example, if the header information for determination is 1 byte of 0xFF described above, the received data sorting unit 131 determines whether the first 1 byte of the received data matches 0xFF. If they match, the received data sorting section 131 outputs the received data to the bitstream writing section 132 as residual encoded data.

The bitstream writing unit 132 writes one frame of encoded data supplied from the bitstream reading unit 31 and the remaining encoded data supplied from the received data sorting unit 131 to the storage area.

First, a method for writing out one frame's worth of encoded data will be explained.

One frame's worth of encoded data is extracted on the transmitter 111 side, so it is partial data. However, when writing, the bitstream writing unit 132 extracts the relevant frame at the writing destination. Secure the previous size and export.

That is, the bitstream writing unit 132 obtains the frame size before cutting out from the header information of the encoded data, and writes out one frame's worth of encoded data that is initially supplied.

Next, the bitstream writing unit 132 writes 0xFF until the written data size reaches the acquired frame size.

For example, if the frame size written in the header of encoded data is 330 bytes and the size of the supplied encoded data is 192 (6N) bytes, the bitstream writing unit 132 secures an area of 330 bytes. Then, the supplied encoded data is recorded in the first 192 bytes, and the remaining 138 bytes are filled with 0xFF.

Further, encoded data with the same content ID are written to the same file, and when the content IDs are different, encoded data are written to different files.

Note that the writing start position is the end of the file when encoded data arrives consecutively. On the other hand, if the frame numbers are not consecutive due to a transmission error or the like, the bitstream writing unit 132 secures an area of the maximum size specified by the encoding method at the position of the missing frame number, and then After that, write out the received encoded data.

The information written in the missing frame number position may, for example, be all 0xFF, or it may be possible to write dedicated header information consisting of the frame size and information indicating that frame interpolation is necessary, and then write the rest as 0xFF. It's okay.

Next, a method for writing the remaining encoded data will be explained.

To write the remaining encoded data, first detect the encoded data of the frame with the frame number corresponding to the content ID sent with the remaining encoded data, and then look at the first bit every N bytes. , 1 appears, and overwrites the last detected location with the remaining encoded data.

For example, if N bytes are supplied as the remaining encoded data for the frame written out as described above (assuming frame number 10), the bitstream writing unit 132 records the data in the storage area corresponding to the frame number. Open the content file with the specified content ID, that is, the file in which encoded data is recorded. Then, in the opened file, the bitstream writing unit 132 moves to the beginning of the encoded data of the 10th frame, reads the first bit every N bytes from this first address, detects the address where 1 appears, and detects Overwrite the rest of the encoded data starting at the position where it was, and save the file.

By doing this, for example, in addition to the first 6N bytes of data received, N bytes for the remaining encoded data are written, and 7N bytes of encoded data are written to the storage area. become. As a result, the quality of the recorded content can be improved without the user knowing.

<Transmitter processing>
FIG. 10 is a flowchart illustrating the main processing of the transmitter 111.

Note that the processes in steps S111 to S113 and S115 to S119 in FIG. 10 are the same as the processes in steps S11 to S18 in FIG. 5, so the description thereof will be omitted.

In step S113, the encoded data cutting unit 121 reads the encoded data corresponding to the calculated Cut _SZ . At this time, the encoded data cutting section 121 outputs the content ID to the termination processing section 23 and at the same time outputs the content ID, frame number, and Cut _SZ to the untransmitted encoded data information storage section 122.

In step S114, the untransmitted encoded data information storage unit 122 pairs the frame number and Cut _SZ supplied from the encoded data cutout unit 121 and records them in the transmission history information.

FIG. 11 is a flowchart illustrating sub-processing of the transmitter 111.

The sub-processing in FIG. 11 is a process that is intermittently executed by the residual encoded data configuration unit 123 in the transmitter 111 of the second embodiment in parallel with the main processing in FIG. 10.

In step S131, the remaining encoded data configuration unit 123 determines whether there is transmission history information. If it is determined in step S131 that there is transmission history information, the process proceeds to step S132.

In step S132, the remaining encoded data configuration unit 123 opens the encoded data file corresponding to the content ID of the transmission history information.

In step S133, the remaining encoded data configuration unit 123 acquires the oldest information (frame number, etc.) in the transmission history information. For example, if the information is recorded in a list format, the information at the head of the list is acquired.

In step S134, the remaining encoded data configuration unit 123 moves the file pointer to the beginning of the frame with the acquired frame number in the opened file, and acquires the frame size from the header.

In step S135, the remaining encoded data configuration unit 123 refers to the size of the remaining encoded data, which is untransmitted encoded data, and determines the read size.

For example, when acquiring the frame size by N bytes, the remaining encoded data configuration unit 123 compares it with the size of the remaining encoded data and adopts the smaller value (read size = MIN (N, frame size - transmission completed data size)). For example, if N is 32 bytes and the size of the remaining encoded data is 40 bytes, 32 bytes will be used, and if the size of the remaining encoded data is 20 bytes, 20 bytes will be used.

In step S136, the remaining encoded data configuration unit 123 moves the file pointer by the transmitted data size in the opened file. The remaining encoded data configuration unit 123 moves the file pointer to the position where it is actually read. For example, if the transmitted size obtained from the transmission history information is 6N, the remaining encoded data configuration unit 123 moves the file pointer to the position where it is actually read. Read forward 6N bytes at the position.

In step S137, the remaining encoded data configuration unit 123 reads encoded data from the moved file pointer position in the opened file by the readout size calculated in step S135.

In step S138, the remaining encoded data configuration unit 123 updates the transmission history information. For example, the contents of the data in the list referenced in step S133 are updated.

In step S139, the remaining encoded data configuration unit 123 determines whether or not the amount to be processed in the current execution has been completed. The amount to be processed in the current execution may be predetermined as, for example, one frame or five frames in one process. Also, the amount processed in this execution depends on the playback status, for example, if the playback is in progress, only N bytes of remaining encoded data of one frame will be sent in one process, and if the playback is stopped, the remaining encoded data will be sent. , may be dynamically determined, such as sending all the remaining encoded data of one frame, or sending N bytes of remaining encoded data for 10 frames.

If it is determined in step S139 that the portion to be processed in the current execution has not been completed, the process returns to step S133, and the subsequent processes are repeated.

If it is determined in step S139 that the content to be processed in the current execution has been completed, the process proceeds to step S140.

In step S140, the remaining encoded data configuration unit 123 closes the file. After step S140, the sub-processing in FIG. 11 ends.

Even if it is determined in step S131 that there is no transmission history information, the sub-processing in FIG. 11 ends.

<Receiver processing>
FIG. 12 is a flowchart illustrating the processing of the receiver 112.

Note that the processes in steps S151, S152, S154 to S156, and S159 to S161 in FIG. 12 are basically the same as the processes in steps S51 to S58 in FIG. 7, so the explanation thereof will be omitted.

Further, the process in S157 is basically the same process as step S152 (step S52 in FIG. 7), except that the order is changed in order to obtain the first byte required in step S154. , is basically the same as the content shown in FIG.

In step S153 of FIG. 12, the received data sorting unit 131 determines whether the received data is residual encoded data based on the 1 byte acquired in step S152. For example, it is determined whether the 1 byte acquired in step S152 is 0xFF indicating header information for determination.

If it is determined in step S153 that the received data is not residual encoded data, the process proceeds to step S154.

In step S154, the bitstream reading unit 31 determines whether the first bit is 0 or not. If it is determined in step S154 that the leading bit is not 0, the process proceeds to step S158. In this case, the bitstream reading section 31 outputs one frame's worth of encoded data to the bitstream writing section 132.

In step S158, the bitstream writing unit 132 writes out one frame of encoded data supplied from the bitstream reading unit 31. The process of writing encoded data will be described later with reference to FIG. 13.

After the encoded data is written out, the process proceeds to step S159, where decoding processing is performed.

On the other hand, if it is determined in step S153 that the data is residual encoded data, the process proceeds to step S162. In this case, the received data sorting unit 131 outputs the remaining encoded data to the bitstream writing unit 132 as is.

In step S162, the bitstream writing unit 132 writes out one frame's worth of residual encoded data supplied from the bitstream reading unit 31. The process of writing out the remaining encoded data will be described later with reference to FIG.

After the remaining encoded data is written out, the processing of the receiver 112 in FIG. 12 ends.

FIG. 13 is a flowchart illustrating the encoded data writing process in step S158 of FIG. 12.

In step S171, the bitstream writing unit 132 opens the file corresponding to the content ID. The file to be opened is a file in which encoded data corresponding to the sent content ID is recorded, and if a corresponding file does not exist, a new file is generated.

In step S172, the bitstream writing unit 132 obtains the frame number and frame size from the header information of the received encoded data.

In step S173, the bitstream writing unit 132 moves the file pointer to the beginning of the corresponding frame in the opened file. If encoded data is sent continuously, the destination will be the end of the file. In addition, if the frame numbers are not consecutive due to a transmission error, etc., the destination of the movement is determined by securing an area of the maximum size specified by the encoding method at the position of the missing frame number, as described above. It will be the next area.

In step S174, the bitstream writing unit 132 writes the received encoded data to a file, and sets the size of the actually written encoded data to the write size indicating the exported size (write size = received encoded data size).

In step S175, the bitstream writing unit 132 determines whether the writing size is smaller than the frame size obtained in step S172. If it is determined in step S175 that the write size is smaller than the frame size, the process proceeds to step S176.

In step S176, the bitstream writing unit 132 writes out 0xFF for each byte (writing size +=1). In addition, in the example in Figure 13, the example was written in 1-byte increments, but after checking the difference between the frame size and the export size, it is written in 8-byte or 4-byte chunks, and finally in 1-byte increments. etc.

After step S176, the process returns to step S175, and the subsequent processes are repeated.

If it is determined in step S175 that the write size is larger than the frame size, the process proceeds to step S177.

In step S177, the bitstream writing unit 132 closes the file.

After step S177, the encoded data writing process in FIG. 13 ends.

FIG. 14 is a flowchart illustrating the process of writing out the remaining encoded data in step S162 of FIG. 12.

In step S181, the bitstream writing unit 132 opens the file corresponding to the content ID. The file to be opened is the one in which the encoded data corresponding to the sent content ID is recorded.As mentioned above, if there is no corresponding file when writing the encoded data, the above-mentioned It is generated in step S171 in FIG.

In step S182, the bitstream writing unit 132 acquires the frame number of the received remaining encoded data.

In step S183, the bitstream writing unit 132 moves the file pointer to the position where the encoded data of the frame number acquired in step S182 is recorded on the file, for example, to the beginning of the frame of the frame number. For the movement, for example, a method is used in which the frame size of the header information of each frame is referred to and each frame is skipped until the corresponding frame is reached.

In step S184, the bitstream writing unit 132 reads 1 byte of encoded data that has already been written.

In step S185, the bitstream writing unit 132 determines whether the leading bit is 0 or not. If it is determined that the first bit is 0, the process advances to step S186.

In step S186, the bitstream writing unit 132 reads N-1 bytes of encoded data that has already been written. After step S186, the process returns to step S184, and the subsequent processes are repeated.

That is, in steps S184 to S186, the first bit is checked every N bytes in the encoded data that has already been written, and a non-zero position is detected.

If it is determined in step S185 that the first bit is not 0, the process proceeds to step S187.

In step S187, the bitstream writing unit 132 writes the remaining encoded data to the reading source from which 1 byte was read. That is, in step S187, the remaining encoded data is written from the 1-byte area in which it is determined that the leading bit is not 0. The size to be written can be obtained from the size information sent together.

In step S188, the bitstream writing unit 132 determines whether writing of all remaining encoded data has been completed. If it is determined in step S188 that writing of all remaining encoded data has not yet been completed, the process returns to step S182, and the subsequent processes are repeated.

If it is determined in step S188 that writing of all remaining encoded data has been completed, the process proceeds to step S189.

In step S189, the bitstream writing unit 132 closes the file opened for writing. After step S189, the remaining encoded data writing process in FIG. 14 ends.

In the second embodiment, the case where the bitstream writing unit 132 receives encoded data from the bitstream reading unit 31 has been described as an example, but the bitstream writing unit 132 receives encoded data from the received data sorting unit 131. It may be configured to always receive converted data and write it to a storage area. In this case, the functions necessary for the bitstream writing section 132 are moved from the bitstream reading section 31 and installed therein.

<3. Third embodiment>
<Configuration of real-time recording communication playback system>
FIG. 15 is a diagram illustrating a configuration example of a real-time recording communication playback system according to the third embodiment of the present technology.

The real-time recording communication playback system 201 in FIG. 15 differs from the real-time communication playback system 1 in FIG. 4 in that an encoder 211 is added.

That is, the real-time recording communication playback system 201 in FIG. 15 is configured to include an encoder 211, and the transmitter 11 and receiver 12 in FIG. Note that in FIG. 15, parts common to those in FIG. 4 are denoted by the same reference numerals, and their explanations will be omitted.

The encoder 211 communicates with the transmitter 11 via communication A. The transmitter 11 communicates with the receiver 12 via communication B.

The encoder 211 encodes the audio signal input from the input system into encoded data having the format shown in FIG. 2, and transmits it to the transmitter 11 via communication A.

Specifically, upon receiving an instruction to start communication, the encoder 211 analyzes, quantizes, and encodes the audio signal input from the input system, and then forms encoded data in the format shown in FIG. and transmits it to the transmitter 11.

The transmitter 11 transmits at least part of the encoded data to the receiver 12 depending on the status of communication B.

Note that in FIG. 15, unless otherwise specified, communication A is a communication path that has sufficient bandwidth to communicate all encoded data in real time, such as communication within the same device or sufficiently high-speed Internet communication. etc. Communication B is a communication channel that does not have sufficient bandwidth to communicate all encoded data in real time, for example, short-range communication such as Bluetooth.

Furthermore, to simplify the explanation, it is assumed that the encoded data received via communication A is temporarily stored in a storage area (not shown) of the transmitter 11. That is, the encoder 211 and the transmitter 11 may be included in the same device.

Furthermore, although the data to be transmitted may be information such as video, audio, tactile sensation, or vibration, FIG. 15 will also be described using audio data such as music as an example.

Further, in FIG. 15, it is assumed that instructions to start and stop communication are exchanged between the encoder 211, the transmitter 11, and the receiver 12.

The configuration of the encoder 211 will be described below.

The encoder 211 includes a signal analysis section 221, a quantization section 222, an encoding section 223, and an encoded data forming section 224.

The signal analysis unit 221 performs band division and time-frequency conversion on one frame of the input audio signal from the input system, further analyzes it, and outputs the result to the quantization unit 222.

The band division consists of, for example, a Quadrature Mirror Filter, a Polyphase Quadrature filter, or a one-third octave band analysis filter bank. The time-frequency transform includes, for example, a discrete Fourier transform or a modified discrete cosine transform (MDCT).

Analysis includes, for example, envelope analysis of time signal waveforms, envelope analysis of frequency spectra, gathering the obtained frequency spectra into predetermined bands, such as critical bandwidth bands, and finding the power of the spectrum for each grouped band. Consists of analysis, etc. Hereinafter, in order to simplify the explanation, the supplied signal will be a frequency spectrum, but it may be a time waveform signal or the like.

The quantization unit 222 quantizes the analyzed signal supplied from the signal analysis unit 221 and outputs it to the encoding unit 223. Quantization is performed, for example, by determining the maximum power of the frequency spectrum for each predetermined processing unit, and calculating a coefficient (Scale Factor) so that this maximum value falls within +-1.

For example, if the maximum value is 120, 128, which is a power of 2, may be used as the Scale Factor. This calculated Scale Factor is output to the encoding section 223 together with the quantized signal.

The encoding unit 223 assigns a code to the quantized signal supplied from the quantizing unit 222 according to a predetermined rule, and outputs it to the encoded data forming unit 224. Examples of codes used for encoding include Huffman codes and arithmetic codes.

The encoded data forming unit 224 forms the supplied encoded data in the format shown in FIG. 2, and outputs it to a transmission processing unit (not shown). When preparing the format, the most significant bit may be set to 0 for every N bytes.

For example, if N is 32 bytes and is to be formed while being written to a temporary storage area, the encoded data forming unit 224 first sets the first bit to 0, and then sequentially sets 32 bytes of header information, sub information, and main information. Write until bytes are reached. When 32 bytes are reached, the encoded data forming unit 224 sets the first bit to 0, and then writes out the remaining data. The above process is repeated every 32 bytes.

<Encoder processing>
FIG. 16 is a flowchart illustrating the processing of the encoder 211.

In step S211, the signal analysis unit 221 analyzes the waveform of one frame of the audio signal.

In step S212, the quantization unit 222 quantizes the waveform-analyzed audio signal.

In step S213, the encoding unit 223 encodes the quantized audio signal and generates encoded data. The generated encoded data is, for example, a bit string consisting of sub information and main information.

In step S214, the encoded data forming unit 224 generates header information according to the generated encoded data. The frame size recorded in the header information is the size of the generated encoded data plus one bit required for every (N*8-1) bits.
For example, it is calculated using the following equation (2).

Here, from step S215 onwards, the encoded data forming unit 224 writes the header information and encoded data to the temporary storage area while adding 0 bits to every (N*8-1) bits.

For this purpose, a write buffer with a size of N bytes is used. That is, the encoded data forming unit 224 first clears the N-byte write buffer with 0, copies (N*8-1) bits of encoded data excluding the first bit, and temporarily stores this. Export to area.

Note that the generated header information is assumed to be at the beginning of the encoded data. Details will be explained below.

In step S215, the encoded data forming unit 224 clears the write buffer with 0.

In step S216, the encoded data forming unit 224 determines whether the size of the remaining encoded data (BN _Data ) is 7 bits or more.

If it is determined in step S216 that BN _Data is 7 bits or more, the process proceeds to step S217.

In step S217, the encoded data forming unit 224 copies the first 7 bits of the remaining encoded data from the second bit from the beginning of the write buffer.

In step S218, the encoded data forming unit 224 subtracts and updates the 7 bits of the BN _Data .

In step S219, the encoded data forming unit 224 determines whether BN _Data is larger than (N*8-8) bits. If it is determined in step S219 that BN _Data is larger than (N*8-8) bits, the process proceeds to step S220.

In step S220, the encoded data forming unit 224 copies (N*8-8) bits of the remaining encoded data following the 7 bits copied earlier in the write buffer.

In step S221, the encoded data forming unit 224 subtracts (N*8-8) bits from the copied BN _Data and updates it.

In step S222, the encoded data forming unit 224 writes the contents of the write buffer to the temporary storage area. After step S222, the process returns to step S215, and the subsequent processes are repeated.

If it is determined in step S216 that the BN _Data is not 7 bits or more, or if it is determined in step S219 that the BN _Data is not (N*8-8) bits or more, the process proceeds to step S223. move on.

In step S223, the encoded data forming unit 224 copies all remaining encoded data to the write buffer.

In step S224, the encoded data forming unit 224 writes the contents copied to the write buffer to a temporary storage area. Note that if there is no copied content in the write buffer, nothing is done.

After step S224, the processing of the encoder 211 ends.

<4. Fourth embodiment>
<Real-time communication system configuration>
FIG. 17 is a block diagram illustrating a configuration example of a real-time communication system according to the fourth embodiment of the present technology.

The real-time communication system 251 in FIG. 17 is composed of one repeater 261 and four terminals 262-1 to 262-4 connected to the repeater 261 via the Internet or the like. Note that in FIG. 17, parts common to those in FIG. 4 or 15 are given the same reference numerals, and their explanations will be omitted. Note that the number of terminals may be 2 or 3, or may be 5 or more.

The repeater 261 communicates with the terminals 262-1 to 262-4 via communications A1 to A4, respectively. If there is no need to distinguish between the terminals 262-1 to 262-4, they will be referred to as terminals 262. If there is no need to distinguish communications A1 to A4, they will be referred to as communication A.

The repeater 261 receives encoded data from at least one of the terminals 262, forms encoded data according to the communication status of communication A with each transmitting terminal 262, and transmits the formed encoded data. , to each terminal 262.

In the real-time communication system 251 of FIG. 17, the communication status of each of communications A1 to A4 changes from moment to moment. Therefore, for example, even if the terminal 262-1 sends encoded data with a high bit rate to the repeater 261 because the communication condition of communication A1 is good, if the communication condition of communication A2 is bad, the high bit rate If encoded data is sent at the same rate, problems such as audio interruptions may occur.

In the real-time communication system 251, the repeater 261 can devise a method for transmitting encoded data in such a situation, thereby avoiding the occurrence of a failure.

Furthermore, the configuration of each device will be explained.

Similar to the transmitter 11 in FIG. 4, the repeater 261 is configured to include a cutout setting determination section 21, an encoded data cutout section 22, a termination processing section 23, and a transmission data configuration section 24.

When the cutout setting determining unit 21 receives encoded data from any of the terminals 262, it estimates the communication status of communication A with each terminal 262, analyzes the situation, and sends cutout setting information to each terminal 262. Generate each.

The encoded data cutting unit 22 cuts out and generates one frame worth of encoded data for each terminal 262.

The termination processing unit 23 performs termination processing on each of the generated encoded data.

The transmission data composition unit 24 compiles the data into chunks of data to be transmitted at once to each terminal 262 and outputs the data to the transmission processing unit 25. The data supplied to the transmission processing unit 25 is transmitted to each terminal 262, respectively.

The terminal 262 is configured to include an encoder 211, a transmitter 11, and a receiver 12.

When an audio signal is input from the input system, the encoder 211 forms encoded data in the format shown in FIG. 2 and outputs it to the transmitter 11.

The transmitter 11 collects the encoded data formed by the encoder 211 into a data chunk to be transmitted at once, and then transmits it to the repeater 261.

When the receiver 12 receives the encoded data from the repeater 261, it synthesizes the audio data and reproduces it through a speaker or the like through an output system (not shown).

Note that in FIG. 17, for the sake of simplicity, the encoder 211 and transmitter 11 are described as separate blocks, but they may be combined into one block. When the data are combined into one block, the encoded data generated by the encoder 211 may be adjusted to fit within the size to be transmitted.

In addition, in FIG. 17, the repeater 261 transmits the received encoded data to any terminal 262 without distinguishing it from the terminal 262 that sent it. You may choose not to do so.

Furthermore, when multiple encoded data are received from multiple terminals 262 almost simultaneously, the repeater 261 may mix the multiple encoded data before transmitting. In this case, for example, the cut out encoded data may be concatenated. Furthermore, in this case, a format in which encoded data may be concatenated may be defined in advance.

It should be noted that the processing of the repeater 261 and each terminal 262 can be realized by a configuration in which the processing described in the first to third embodiments is operated in a time-sharing manner, so a description thereof will be omitted. The third and fourth embodiments are combinations with the first embodiment, but the second embodiment may be combined with the second embodiment instead of the first embodiment.

<5. Others>
<Effects of this technology>
As described above, in the present technology, one frame of encoded data including header information and main information is cut out from the beginning of the header information in a cutout size that is an integral multiple of the cutout unit of N bytes, and the cutout data is A termination identifier is added to the end of the data.

According to this technology, the bit rate adjustment according to the transmission band and purpose can be done by cutting out the data without decoding the encoded data, so even if the content is stored as encoded data, the amount of processing Content can be distributed at a bit rate suitable for the communication channel without increasing the bit rate.

In addition, by using communication idle time to transmit unsent encoded data, the quality of recorded content can be made to the highest quality. In particular, the quality of recorded content can be maximized without the user knowing.

In addition, in real-time communication systems, each terminal does not send data encoded at multiple bit rates; instead, at the repeater, data is encoded at the optimal bit rate according to the communication status between each terminal and the repeater. Data can be sent.

<Computer configuration example>
The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, a program constituting the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

FIG. 18 is a block diagram showing an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

A CPU 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are interconnected by a bus 304.

An input/output interface 305 is further connected to the bus 304. Connected to the input/output interface 305 are an input section 306 consisting of a keyboard, mouse, microphone, etc., and an output section 307 consisting of a display, speakers, etc. Further, connected to the input/output interface 305 are a storage section 308 made up of a hard disk, a nonvolatile memory, etc., a communication section 309 made up of a network interface, etc., and a drive 310 that drives a removable medium 311 .

In the computer configured as described above, the CPU 301, for example, loads a program stored in the storage unit 308 into the RAM 303 via the input/output interface 305 and the bus 304 and executes it, thereby performing the series of processes described above. will be held.

A program executed by the CPU 301 is installed in the storage unit 308 by being recorded on a removable medium 311 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

Note that the program executed by the computer may be a program in which processing is performed chronologically in accordance with the order described in this specification, in parallel, or at necessary timing such as when a call is made. It may also be a program that performs processing.

Note that in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are located in the same casing. Therefore, multiple devices housed in separate casings and connected via a network, and a single device with multiple modules housed in one casing are both systems. .

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiments of the present technology are not limited to the embodiments described above, and various changes can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared and jointly processed by multiple devices via a network.

Furthermore, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices.

Further, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

<Example of configuration combinations>
The present technology can also have the following configuration.
(1)
a cutting unit that cuts out one frame of encoded data including header information and main information in a cutting size that is an integral multiple of a cutting unit of N bytes from the beginning of the header information;
A signal processing device comprising: a termination processing section that adds a termination identifier to the end of the cutout data cut out by the cutout section.
(2)
The cutout size is based on at least one of the processing load on the sending side, the communication status between the sending side and the receiving side, resource availability information on the receiving side, the CPU usage rate on the receiving side, and the remaining battery level on the receiving side. The signal processing device according to (1), further comprising: a cutout size determination unit that determines the size of the cutout size.
(3)
The signal processing device according to (1) or (2), further comprising a transmitter that transmits the cut-out data to which the termination identifier is added.
(4)
further comprising a residual encoded data configuration unit that generates residual encoded data that is the remainder of the encoded data after the cutout data is extracted by the extraction unit,
The signal processing device according to (3), wherein the transmitter transmits the remaining encoded data.
(5)
The signal processing device according to (4), wherein the transmitting unit transmits the remaining encoded data using a free time for transmitting the cut-out data.
(6)
further comprising a history information storage unit that stores history information of the cutout data cut out by the cutout unit,
The signal processing device according to (4) or (5), wherein the residual encoded data configuration unit generates the residual encoded data with reference to the history information.
(7)
The signal processing device according to (6), wherein the history information includes at least the frame number of the encoded data and the cutout size.
(8)
The cutout unit is predetermined between the sending side and the receiving side, or is shared between the sending side and the receiving side by being transmitted together with the cutout data. Any of (1) to (7) above. The signal processing device according to claim 1.
(9)
The signal processing device according to any one of (1) to (7), wherein the cutting unit is changeable.
(10)
In the cutout data, the first bit of the cutout unit takes a first value, and the first bit of the termination identifier takes a second value different from the first value. (1) to (9) The signal processing device according to any one of.
(11)
The signal processing device according to any one of (1) to (10), wherein the header information includes the size of the encoded data of the one frame.
(12)
The encoded data of the one frame includes the header information, sub information, and the main information,
The signal processing device according to any one of (1) to (11), wherein the sub-information includes general shape information of a signal to be restored.
(13)
The signal processing device according to (12), wherein the outline information includes at least one of information for restoring an envelope of a frequency spectrum and information for restoring an envelope of a time waveform.
(14)
The signal processing device according to any one of (1) to (13), wherein the information recorded in the main information is arranged in order of importance from the beginning or from the beginning.
(15)
The information recorded in the main information is at least one of audio information, video information, tactile information, and vibration information, and is arranged from the beginning in order from low range to high range, as described in (14) above. signal processing device.
(16)
The signal processing device according to (14), wherein the information recorded in the main information is hierarchically encoded information and is arranged in order from the top to the first layer.
(17)
The information recorded in the main information is at least one of audio information, video information, tactile information, vibration information, and hierarchically encoded information; The signal processing device according to (14) above, in which the signal processing device is arranged in order from the top to the first layer, and in the case of hierarchically encoded information, it is arranged in order from the top to the first layer.
(18)
The signal processing device
Cutting out one frame of encoded data including header information and main information from the beginning of the header information in a cutout size that is an integral multiple of the cutout unit of N bytes,
A signal processing method that adds a termination identifier to the end of the extracted data.
(19)
a cutting unit that cuts out one frame of encoded data including header information and main information in a cutting size that is an integral multiple of a cutting unit of N bytes from the beginning of the header information;
A program that causes a computer to function as a termination processing section that adds a termination identifier to the end of the cutout data cut out by the cutout section.
(20)
One frame of encoded data including header information and main information is cut out from the beginning of the header information in a cutout size that is an integral multiple of the cutout unit of N bytes, and is transmitted based on the first bit of the cutout data. A signal processing device comprising a decoding unit that decodes cut-out data while detecting a termination identifier added to the end of the cut-out data, and decodes information that cannot be decoded as zero at the time when the termination identifier is detected.

1. Real-time communication playback system, 11. Transmitter, 12. Receiver, 21. Extraction setting determination section, 22. Encoded data extraction section, 23. Termination processing section, 24. Transmission data composition section, 25. Transmission processing section, 31. Bit stream reading section, 32 Decoding unit, 33 Inverse quantization unit, 34 Signal synthesis unit, 101 Real-time communication reproduction system, 111 Transmitter, 112 Receiver, 121 Encoded data extraction unit, 122 Untransmitted encoded data information storage unit, 123 Remaining encoded data Component, 131 Received data sorting unit, 132 Bitstream writing unit, 201 Real-time recording communication playback system, 211 Encoder, 221 Signal analysis unit, 222 Quantization unit, 223 Encoding unit, 224 Encoded data forming unit, 251 Real-time communication system, 261 repeater, 262-1 to 262-4 terminal

Claims

a cutting unit that cuts out one frame of encoded data including header information and main information in a cutting size that is an integral multiple of a cutting unit of N bytes from the beginning of the header information;
A signal processing device comprising: a termination processing section that adds a termination identifier to the end of the cutout data cut out by the cutout section.
The cutout size is based on at least one of the processing load on the sending side, the communication status between the sending side and the receiving side, resource availability information on the receiving side, the CPU usage rate on the receiving side, and the remaining battery level on the receiving side. The signal processing device according to claim 1, further comprising: a cutout size determination unit that determines the size of the cutout size.
The signal processing device according to claim 1, further comprising a transmitter that transmits the cut-out data to which the termination identifier is added.
further comprising a residual encoded data configuration unit that generates residual encoded data that is the remainder of the encoded data after the cutout data is extracted by the extraction unit,
The signal processing device according to claim 3, wherein the transmitter transmits the residual encoded data.
The signal processing device according to claim 4, wherein the transmitting unit transmits the remaining encoded data using a free time for transmitting the cut-out data.
further comprising a history information storage unit that stores history information of the cutout data cut out by the cutout unit,
The signal processing device according to claim 4, wherein the residual encoded data configuration unit generates the residual encoded data with reference to the history information.
The signal processing device according to claim 6, wherein the history information includes at least a frame number of the encoded data and the cutout size.
The signal processing device according to claim 1, wherein the cutting unit is predetermined between the transmitting side and the receiving side, or is shared between the transmitting side and the receiving side by being transmitted together with the cutting data.
The signal processing device according to claim 1, wherein the cutting unit is changeable.
The signal processing according to claim 1, wherein in the cutout data, the first bit of the cutout unit takes a first value, and the first bit of the termination identifier takes a second value different from the first value. Device.
The signal processing device according to claim 1, wherein the header information includes a size of the encoded data of the one frame.
The encoded data of the one frame includes the header information, sub information, and the main information,
The signal processing device according to claim 1, wherein the sub-information includes outline information of a signal to be restored.
The signal processing device according to claim 12, wherein the outline information includes at least one of information for restoring an envelope of a frequency spectrum and information for restoring an envelope of a time waveform.
The signal processing device according to claim 1, wherein the information recorded in the main information is arranged in order of importance from the beginning or from the beginning to the end.
15. The information recorded in the main information is at least one of audio information, video information, tactile information, and vibration information, and is arranged from the beginning in order from low range to high range. Signal processing device.
The signal processing device according to claim 14, wherein the information recorded in the main information is hierarchically encoded information and is arranged in order from the top to the first layer.
The information recorded in the main information is at least one of audio information, video information, tactile information, vibration information, and hierarchically encoded information, and at least one of audio information, video information, tactile information, and vibration information. 15. The signal processing device according to claim 14, wherein if there is one, the signal processing device is arranged from the top in order from low band to high band, and in the case of layered encoded information, it is arranged in order from the top to the first layer.
The signal processing device
Cutting out one frame of encoded data including header information and main information from the beginning of the header information in a cutout size that is an integral multiple of the cutout unit of N bytes,
A signal processing method that adds a termination identifier to the end of the extracted data.
a cutting unit that cuts out one frame of encoded data including header information and main information in a cutting size that is an integral multiple of a cutting unit of N bytes from the beginning of the header information;
A program that causes a computer to function as a termination processing section that adds a termination identifier to the end of the cutout data cut out by the cutout section.
One frame of encoded data including header information and main information is cut out from the beginning of the header information in a cutout size that is an integral multiple of the cutout unit of N bytes, and is transmitted based on the first bit of the cutout data. A signal processing device comprising a decoding unit that decodes cut-out data while detecting a termination identifier added to the end of the cut-out data, and decodes information that cannot be decoded as zero at the time when the termination identifier is detected.