WO2018139285A1 - Image processing device, method and program - Google Patents

Abstract

The present technology pertains to an image processing device, method and program which make it possible to reduce an unnatural sensation when viewing. This image processing device is equipped with a playback control unit for controlling in a manner such that the quality of playback data changes as time passes during the playback of the higher quality playback data among first playback data and second playback data, when switching playback from playback based on first playback data to playback based on second playback data, the quality of which differs from that of the first playback data. This technology is applicable to client devices.

Description

Image processing apparatus and method, and program

The present technology relates to an image processing apparatus and method, and a program, and more particularly to an image processing apparatus and method, and a program capable of reducing discomfort when viewing.

For example, Bitrate Adaptation is a feature of MPEG-DASH (Moving Picture Experts Group phase-Dynamic Adaptive Streaming over HTTP) streaming (see, for example, Non-Patent Document 1).

That is, in the MPEG-DASH streaming, the same reproduction content is prepared for each of a plurality of bit rates in the server according to the transfer rate of the transmission path. Specifically, for example, data of each bit rate of content is defined as Representation.

Then, a list of Representations for each of the bit rates is presented to the client device by a Media Presentation Description (MPD) file.

In the client device, data of an appropriate bit rate is selected while measuring the download speed of the data of the content at a constant interval, such as at each boundary position of a segment (Segment) which is a unit in which the data of the content is handled. Thus, it is possible to reproduce the video and audio of the content without interruption while guaranteeing the supply of the reproduction content.

In this case, although the content prepared by the server changes the encoding condition depending on the bit rate, for example, in the case of video, it is general to produce data of a plurality of patterns having different resolutions as data of different bit rates. It is.

In addition, compressed audio is generally used for audio, bit rates are several tenths to hundreds of times lower than video, and are rarely subject to Bitrate Adaptation in video-based distribution. .

However, in recent years the demand for high resolution quality (high quality) audio is increasing, and network download sales have already been launched as a service. In addition, delivery of streaming viewing format has also begun experimentally, and demand is expected in the future.

Therefore, since a high bit rate (for example, about several Mbps to several tens of Mbps) comparable to that of video is required for audio, as in the case of video, it can be a target of bit rate adaptation according to the transmission band.

As described above, in MPEG-DASH streaming, the content prepared in the server is encoded under the conditions suitable for each bit rate, but the authoring work not only increases correspondingly if the number of selectable bit rates is increased The data capacity required by the server will also increase. Therefore, the number of selectable bit rates is not necessarily so large. Also, in the case of video, for example, the variation of resolution for each bit rate is generally selected from representative ones, and arbitrary resolution is not used very much.

Therefore, if the number of bit rates of the content prepared on the server side is small, when performing Bitrate Adaptation, the quality of the content such as the resolution of the video changes in stages, causing discomfort in the viewing experience. There is.

The switching of the stream (bit rate) by Bitrate Adaptation is performed at the boundary of the segment, and at the boundary position, one frame before is reproduced at the quality before switching, and one frame after that is reproduced at the quality after switching. Therefore, for example, when the resolution or the like of the displayed image changes significantly before and after switching, a sense of discomfort may be generated.

The present technology has been made in view of such a situation, and makes it possible to reduce discomfort when viewing.

The image processing device according to one aspect of the present technology switches playback from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data. The reproduction control unit controls the quality of the reproduction data to change with time at the time of reproduction of reproduction data of higher quality of the first reproduction data and the second reproduction data.

The first reproduction data and the second reproduction data may be reproduction data of different bit rates.

In the reproduction control unit, when the quality of the first reproduction data is higher than the quality of the second reproduction data, in the portion of the first reproduction data immediately before switching to the second reproduction data, The quality of the first reproduction data may be changed such that the quality of the first reproduction data is substantially the same as the quality of the second reproduction data.

In the reproduction control unit, when the quality of the second reproduction data is higher than the quality of the first reproduction data, in the part immediately after switching from the first reproduction data in the second reproduction data, The quality of the second reproduction data is changed so that the quality of the second reproduction data is substantially the same as the quality of the first reproduction data, and the quality of the second reproduction data increases with time. can do.

The first reproduction data and the second reproduction data may be video data.

The reproduction control unit may change the quality of the reproduction data by changing at least one of a resolution, a frame rate, a bit depth, and a color gamut of the reproduction data.

The first reproduction data and the second reproduction data may be audio data.

The reproduction control unit can change the quality of the reproduction data by changing at least one of the sampling rate and the bit depth of the reproduction data.

The reproduction control unit causes specification information specifying a parameter for changing the quality of the reproduction data to be acquired, and changes the parameter indicated by the specification information of the plurality of the parameters in the reproduction data. Thus, the quality of the reproduction data can be changed.

The reproduction control unit acquires order information indicating a plurality of parameters for changing the quality of the reproduction data and an adjustment order of the plurality of parameters, and the reproduction information is indicated by the order information in the reproduction data. The quality of the reproduction data can be changed by adjusting the plurality of parameters in the adjustment order indicated by the order information.

The reproduction control unit is made to acquire processing specification information for specifying adjustment processing for changing a parameter for changing the quality of the reproduction data, and the adjustment indicated by the processing specification information among a plurality of the adjustment processing The quality of the reproduction data can be changed by changing the parameter of the reproduction data by processing.

The reproduction data may be video data, and the parameters may be resolution, frame rate, bit depth, or color gamut.

The reproduction data may be audio data, and the parameter may be a sampling rate or bit depth.

The reproduction control unit can perform control such that the period in which the quality of the reproduction data is changed is the same for the video data as the reproduction data and the audio data as the reproduction data.

The image processing method or program according to one aspect of the present technology switches playback from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data. And controlling the quality of the reproduction data to change with time at the time of reproduction of reproduction data of higher quality among the first reproduction data and the second reproduction data.

In one aspect of the present technology, when switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first It is controlled such that the quality of the reproduction data changes with time at the time of reproduction of reproduction data of higher quality among the reproduction data and the second reproduction data.

According to one aspect of the present technology, it is possible to reduce discomfort when viewing.

In addition, the effect described here is not necessarily limited, and may be any effect described in the present disclosure.

It is a figure explaining Bitrate Adaptation. It is a figure explaining Bitrate Adaptation. It is a figure which shows the structural example of a client apparatus. It is a figure explaining control at the time of Bitrate Adaptation. It is a figure explaining control at the time of Bitrate Adaptation. It is a figure explaining the quality control at the time of an effect. It is a figure explaining the quality control at the time of an effect. It is a figure explaining the quality control at the time of an effect. It is a figure explaining the quality control at the time of an effect. It is a figure explaining switching of a bit rate. It is a flowchart explaining quality adjustment effect control processing. 5 is a flowchart illustrating content reproduction processing. It is a figure explaining designated parameter information. It is a flowchart explaining quality adjustment effect control processing. It is a figure explaining order information. It is a flowchart explaining quality adjustment effect control processing. It is a figure showing an example of composition of a computer.

Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

First Embodiment
<About this technology>
In the present technology, when switching from reproduction based on predetermined reproduction data to reproduction based on reproduction data other than the reproduction data, the quality of reproduction data is continuous in the time direction. By making the change into the above, it is possible to reduce the sense of incongruity in the viewing sensation.

For example, in the MPEG-DASH streaming delivery, when Bitrate Adaptation occurs due to bandwidth fluctuation of the transmission path, the quality of video such as resolution may change at a level that can be perceived visually. Since such abrupt quality changes cause viewing discomfort, in the present technology, by appropriately adjusting the quality of the content at the time of content reproduction, the viewing sensation is reduced.

Note that the present technology is not limited to MPEG-DASH streaming distribution, and any other content reproduction technology may be used as long as reproduction data of an arbitrary quality is acquired and reproduced from among reproduction data of a plurality of different qualities. It is also applicable to the case. In the following, the description will be continued with an example where the present technology is applied to MPEG-DASH streaming distribution.

In Bitrate Adaptation, one appropriate Representation is selected from a plurality of Representations, that is, bit rates of the same Adaptation Set, and segment data corresponding to the Representation is downloaded and reproduced.

Here, the video reproduced by each of the plurality of segment data of the same Adaptation Set has different contents such as the resolution itself, that is, the bit rate of the video data. This is true not only for video but also for audio.

For example, in MPEG-DASH streaming, if the number of bit rates that can be selected by Bitrate Adaptation is increased, the data capacity necessary for the server to record data will increase accordingly. Therefore, the number of bit rates that can be selected by Bitrate Adaptation is not necessarily large.

Also, in the case of video, the variation of resolution per bit rate is generally selected from representative values such as Full HD, ie 1920 × 1080, and it is almost the case that any resolution is used. Absent.

Specifically, in the case of a delivery service that provides 4K (3840 × 2160) images as the highest quality, the resolutions prepared are at most 3840 × 2160, 3200 × 1800, 2560 × 1440, 1920 × 1080, 1600 × It is considered to be about seven patterns of 900, 1280 × 720, and 720 × 480. It is also well assumed that some of these resolutions are not available.

In Bitrate Adaptation, the cause of the sense of incongruity in viewing experience is that the quality, such as the resolution, of the content prepared on the server side differs stepwise.

In Bitrate Adaptation, switching of Representation (bit rate) is performed at the boundary position of the segment, and at the boundary position, one frame before is reproduced at the quality before switching, and one frame after that is reproduced at the quality after switching.

Specifically, for example, in the situation where the viewer is viewing the content with the quality of 4K (3840 × 2160), the bandwidth of the transmission path is extremely reduced, and the high definition video of 4K to the quality of SD (720 × 480) It is assumed that a situation where switching to

In such a case, the image which was clearly seen by the viewer is clearly changed to a vague image at the next moment, and a phenomenon occurs in which an illusion is caused as if the eyes suddenly deteriorate. . This is because the video is up-converted from the resolution of SD to the resolution of 4K and displayed in the display device or playback device, but the amount of information originally in SD video is small. On the other hand, when the quality is switched from low to high, the image suddenly becomes clear, which may cause discomfort in such a case.

For example, as shown in FIG. 1, even with the current MPEG-DASH client device, it is possible to reduce the sense of incongruity at the time of viewing by gradually downloading segment data with a low bit rate. In FIG. 1, the horizontal direction indicates time, and each square indicates segment data.

In the example of FIG. 1, 4K (3840 × 2160) video data (segment data) having a bit rate of 40 Mbps, 3.2 K (3200 × 1800) video data having a bit rate of 30 Mbps, and a bit rate of 20 Mbps Video data of 2.5 K (2560 × 1440) and video data of 2 K (1920 × 1080) having a bit rate of 10 Mbps are prepared.

For example, when changing the resolution of the video as Bitrate Adaptation, it is determined (decided) that the quality (bit rate) is lowered to 2K video during reproduction of the 4K video by the control software according to the band fluctuation of the transmission path. Do.

That is, it is assumed here that when the 4K segment data SG11 is downloaded and reproduced as the segment Segment (n-1), it is determined that the quality is to be changed from 4K to 2K.

Even in such a case, the MPEG-DASH client device does not switch the video quality from 4K to 2K immediately, but switches from 4K to 3.2K and then switches from 3.2K to 2.5K. After that, it is also possible to switch to 2K finally.

In this example, at the boundary position of the segment Segment (n-1) and the next segment Segment (n), that is, at the end position of the segment data SG11 indicated by the arrow A11, an image is displayed from 4K to 3.2K by Bitrate Adaptation. The quality is switched and the segment data SG12 is downloaded.

Also, at the end position of the segment data SG12 indicated by the arrow A12, the quality of the video is switched from 3.2 K to 2.5 K by Bitrate Adaptation, and the segment data SG13 is downloaded. Furthermore, at the end position of the segment data SG13 indicated by the arrow A13, the quality of the video is switched from 2.5K to 2K by Bitrate Adaptation, and the segment data SG14 is downloaded.

Therefore, in this example, after the 4K video based on the segment data SG11 is played back and then the 3.2K video based on the segment data SG12 is played back, the 2.5K video based on the segment data SG13 is played back, and the final The 2K video is reproduced based on the segment data SG14.

As described above, if the bit rate (quality) of segment data to be downloaded is lowered stepwise, the quality difference before and after switching at each of the switching points indicated by arrows A11 to A13 can be reduced. . Therefore, compared with the case where the quality of the image is instantaneously changed from 4K to 2K, it is possible to reduce the discomfort caused by the resolution decrease.

Also, even in the situation where it is possible to increase the bit rate of the video (segment data), that is, the resolution of the video, if the resolution is rapidly changed from 2K to 4K as in the example shown in FIG. Instead, the quality change at each switching point can be suppressed to a small value by raising the resolution little by little while performing video reproduction at an intermediate resolution.

In these examples, it is possible to reduce the discomfort due to the quality change before and after switching by changing the quality of the image in steps, but it is only a gradual change in quality and whether the change is small or large Difference.

In addition, when the quality must be dropped sharply due to a sudden change in the bandwidth of the transmission line, or when it is not possible to prepare a sufficient bit rate for the Bitrate Adaptation due to the convenience of the distribution side, that is, the server side. There is also a possibility.

For example, as shown in FIG. 2, it may occur that the quality of the image must be sharply reduced from 4K (3840 × 2160) to 2K (1920 × 1080). In FIG. 2, portions corresponding to the case in FIG. 1 are denoted with the same reference numerals, and the description thereof will be appropriately omitted.

In this example, when the segment data SG11 is downloaded, a sudden band fluctuation occurs, so the quality of the video is switched from 4K to 2K by Bitrate Adaptation at the time indicated by the arrow A11.

As a result, in the segment Segment (n) following the segment Segment (n-1), the 2K segment data SG21 is downloaded and reproduced. In particular, if the quality is changed from 4K to 2K, the number of pixels of the downloaded segment data image will be reduced to a quarter.

Also, in this example, after the 2K segment data SG21 is downloaded, the 2K segment data SG22 is also downloaded and reproduced. Then, since a sudden band fluctuation occurs while the segment data SG22 is downloaded, the quality of the video is switched from 2K to 4K by Bitrate Adaptation at the time indicated by the arrow A13.

Therefore, here, after the segment data SG22 is downloaded, the 4K segment data SG23 is downloaded as Segment (n + 2) data and reproduced.

As described above, the quality can not be changed stepwise for some reason, and when the quality of the content changes rapidly and sharply, the discomfort caused when viewing can not be reduced, and the immersive feeling is lost for the viewer You

Furthermore, in the above, the occurrence of discomfort due to a change in quality has been described using a video as an example, but the same may occur for audio. Such a sudden change in quality causes a sense of discomfort in the switching moment of reproduction as a viewing experience, and the sense of immersion is lost.

In the first place, the sense of discomfort when viewing is caused by the fact that the quality of the content being played back changes momentarily and that the change in the quality is large, that is, the change in the quality is gradual.

For example, if the quality changes gradually over time, the difference between the quality before and after the change is very difficult to recognize, and in some cases the viewer (user) changes the quality Sometimes I do not notice myself.

Therefore, in the present technology, it is possible to make it difficult to notice a change in the quality of content at the time of viewing by applying the method described below to Bitrate Adaptation to the client device, thereby reducing the user's (viewer's) viewing discomfort. To eliminate the impeding factors of the sense of immersion.

That is, in the present technology, when high-quality content is played back, the quality is temporarily reduced and played back, and the video and audio playback data is processed so that the perceived quality of the content gradually changes. We decided to apply the regeneration control method to be applied.

For example, in a client apparatus to which the present technology is applied, a reproduction control unit realized by software or the like instructs appropriate timing by the above-described reproduction control method, and performs control such that video and audio effect processing is performed. This effect processing adjusts the quality of the content.

<Configuration Example of Client Device>
Next, a more specific embodiment of the client device to which the present technology is applied will be described.

FIG. 3 is a diagram showing a configuration example of an embodiment of a client apparatus to which the present technology is applied.

The client device 11 shown in FIG. 3 is a playback device that downloads segment data of content from a server (not shown) and controls playback of content composed of at least one of video and audio.

The client device 11 includes an HTTP (Hypertext Transfer Protocol) download control unit 21, an MPD parser 22, a segment parser 23, an image / audio processing unit 24, and a reproduction control unit 25.

The HTTP download control unit 21 downloads (receives) the MPD file from the server and supplies it to the MPD parser 22 or downloads (receives) segment data from the server and supplies it to the segment parser 23.

Here, the MPD file is data in which metadata for managing video (moving image) of content and segment data of audio is described.

In addition, the HTTP download control unit 21 measures the bandwidth of the transmission path (network) between the client device 11 and the server at the time of downloading segment data, that is, the transmission rate (bit rate) of segment data.

Further, the HTTP download control unit 21 appropriately determines the bit rate of the segment data to be downloaded based on the measurement result of the bandwidth of the transmission path and the like, and also determines segment information on the segment data to be downloaded such as the determined bit rate. It is supplied to the reproduction control unit 25.

The MPD parser 22 parses (decodes) the MPD file supplied from the HTTP download control unit 21 and supplies the decryption result to the HTTP download control unit 21 or the reproduction control unit 25.

The segment parser 23 parses (decodes) segment data (segment file) supplied from the HTTP download control unit 21, extracts video data and audio data to be reproduced from the segment data, and supplies the image data to the audio / video processing unit 24. Do.

Under the control of the reproduction control unit 25, the video / audio processing unit 24 generates content based on the video data and audio data supplied from the segment parser 23, that is, video data and audio data for presentation for reproducing video and audio. Generate and output.

The image / sound processing unit 24 has a video decoder 31, a video effector 32, an audio decoder 33, and an audio effector 34.

The video decoder 31 decodes the video data supplied from the segment parser 23 and supplies the video data to the video effector 32.

The video effector 32 outputs the video data supplied from the video decoder 31 as it is as video data for presentation, or performs effect processing on the video data supplied from the video decoder 31, and the video for presentation obtained as a result thereof Output data.

In addition, the video effector 32 appropriately processes the video data for presentation into data in a form to be finally output to a subsequent device such as an image monitor and outputs the data.

In a display device or the like subsequent to the video effector 32, up conversion processing of video data is performed to adjust the size of the video as necessary. That is, for example, when the output video data for presentation is switched from 4K video data to 2K video data, the display device or the like performs an upconversion process on 2K video data, and the image size is 4K. The size conversion is performed so as to be the size of. The video data up-conversion processing may be performed by the video effector 32.

The audio decoder 33 decodes the audio data supplied from the segment parser 23 and supplies the audio data to the audio effector 34.

The audio effector 34 outputs the audio data supplied from the audio decoder 33 as it is as audio data for presentation, or performs an effect process on the audio data supplied from the audio decoder 33, and the audio for presentation obtained as a result thereof Output data.

In addition, the audio effector 34 appropriately processes audio data for presentation into data in a form to be finally output to a subsequent device such as an audio DAC (Digital to Analog Converter) or an amplifier, and outputs the data.

The reproduction control unit 25 controls the overall operation of the client device 11. For example, the reproduction control unit 25 controls the operation of the image / sound processing unit 24 based on the segment information from the HTTP download control unit 21, the decoding result from the MPD parser 22, and the like.

<About the quality adjustment of content>
Subsequently, quality adjustment of content performed by the client device 11 will be described.

For example, the bit rate of the video data constituting the content, that is, the quality of the video is the resolution of the video based on the video data, the frame rate of the video based on the video data, the bit depth of the pixel value of each pixel of the video data, each pixel of the video data It changes by adjusting the color gamut of. In other words, resolution, frame rate, bit depth, color gamut and the like can be adjusted as the quality of the image.

Specifically, for example, the resolution of the video can be adjusted by filter processing using a soft filter for reducing the sense of resolution of video data, resolution conversion processing such as thinning, or the like. Further, the frame rate of the video can be adjusted by thinning processing of a frame of the video, frame rate conversion processing or the like.

For example, the reproduction control unit 25 causes the video effector 32 to execute processing for changing (adjusting) at least one of resolution, frame rate, bit depth, and color gamut of video data as effect processing. Change (adjust) the quality.

Similarly, the bit rate of audio data that constitutes content, that is, the quality of audio, changes by adjusting the sampling rate of audio data, the bit depth of the sample value of each sample of audio data, and the like. In other words, it is possible to adjust the sampling rate, bit depth and the like as the voice quality.

Specifically, for example, the sampling rate of audio data can be adjusted by thinning processing or filtering processing for audio data.

For example, the reproduction control unit 25 changes (adjusts) the quality of sound by causing the audio effector 34 to perform processing of changing (adjusting) at least one of the sampling rate and bit depth of the sound data as effect processing. Let

For example, when adjusting the quality of content, as shown in FIG. 4 and FIG. 5, of the segment data before and after the bit rate switching point by Bitrate Adaptation, the segment data with higher bit rate, that is, higher quality, Effect processing is performed to change the quality.

In the example shown in FIG. 4, switching is performed from 4K (3840 × 2160) segment data having a bit rate of 40 Mbps to 2K (1920 × 1080) segment data having a bit rate of 10 Mbps.

That is, in this example, when reproducing segment Segment (n-1) to segment Segment (n + 2) which are temporally continuous segments, first, data of segment Segment (n-1) and segment Segment (n) As 4K segment data SG31 and segment data SG32 are downloaded.

After that, the bit rate is switched from 40 Mbps to 10 Mbps at the playback time indicated by the arrow A31 by Bitrate Adaptation. That is, the quality of the video is switched from 4K to 2K.

Therefore, in the segment Segment (n + 1) where the playback time shown by the arrow A31 is the start time, the 2K segment data SG33 is downloaded, and subsequently the 2K segment data SG34 is downloaded.

As described above, in the example illustrated in FIG. 4, the bit rate adaptation of the segment changes in the direction of decreasing the bit rate (quality) of the segment. That is, at the time of bit rate switching, lower bit rate segment data is downloaded (acquired).

In such a case, the client device 11 sets a period of a predetermined length that ends at a reproduction time (hereinafter also referred to as a switching point) at which the bit rate indicated by the arrow A31 is switched. Then, the client device 11 performs effect processing on the video data in the effect period Te, and generates video data whose bit rate, that is, the quality gradually decreases, as video data for presentation.

In this example, the effect period Te is a period in which the reproduction time in the middle of the segment Segment (n) is the start time, and the switching point which is the reproduction time at the end position of the segment Segment (n) is the end time There is.

However, when the bit rate of segment data decreases at the switching point, the effect time Te may be any position (reproduction time) as long as the end time is a switching point, for example, for example It may be an arbitrary playback time of Segment (n-1). In other words, the effect period Te may be set to straddle the boundary position of the segments, that is, to straddle a plurality of segments.

In the effect period Te, the resolution of the image is 4K (3840 × 2160) at the start time of the effect period Te, and the resolution of the image is 2K (1920 × 1080) at the end time (switching point) of the effect period Te. Video data is generated by effect processing.

At this time, between the start time and the end time of the effect period Te, the video data for presentation is generated so that the resolution of the video changes continuously (smoothly) linearly or nonlinearly from 4K to 2K. Be done.

In the following, the effect processing for obtaining reproduction data whose quality is lowered with time as described above, that is, video data and audio data will be particularly referred to as quality deterioration effect processing.

If the video of the content is reproduced based on the video data for presentation in the effect period Te obtained by the quality reduction effect processing, the quality of the video starts to gradually decrease from the start time of the effect period Te.

After that, the quality of the video decreases with time, and at the switching point at which the bit rate is switched by the Bitrate Adaptation, the quality of the video becomes equal to the quality after switching, which is reproduced later.

If the bit rate of the video data is switched to a lower bit rate when switching the bit rate, that is, if the quality of the video is switched to a lower quality, the degradation effect is applied to the higher quality video data before switching. Processing is performed.

As a result, the quality of the video smoothly changes with time from the quality of the video before switching to the quality of the video after switching, and it is possible to reduce discomfort when watching content. As a result, it is possible to suppress the loss of immersion when watching content.

The effect period Te is normally set to several seconds to several tens of seconds, but the effect period Te is determined by the reproduction control unit 25 in consideration of the cache state of the content and whether the time is visually sufficient. Ru. For example, if the segment data cache at the bit rate before switching is small, the length of the effect period Te may be shortened according to the cache volume.

In addition, for example, the length of the effect period Te may be determined in advance with respect to the combination of bit rates of segment data before and after switching, that is, the combination of the bit rate before switching and the bit rate after switching. . In this case, for example, when the quality of the content greatly changes before and after switching, the effect period Te can be made longer.

With respect to the example shown in FIG. 4 as described above, when segment data of a higher bit rate is downloaded after switching as shown in FIG. 5 for example, the period after the switching point is taken as the effect period Te .

In the example shown in FIG. 5, the switching is performed from 2K (1920 × 1080) segment data with a bit rate of 10 Mbps to 4K (3840 × 2160) segment data with 40 Mbps.

That is, in this example, 2K segment data SG41 and segment data SG42 are downloaded as data of segment Segment (n-1) and segment Segment (n). After that, the bit rate is switched from 10 Mbps to 40 Mbps at the playback time indicated by the arrow A41 by Bitrate Adaptation. That is, the quality of the video is switched from 2K to 4K.

Therefore, at the playback time indicated by the arrow A41, that is, the segment Segment (n + 1) where the switching point is the start time, the 4K segment data SG43 is downloaded, and subsequently the 4K segment data SG44 is downloaded.

As described above, in the example illustrated in FIG. 5, the bit rate adaptation in the segment changes in the direction of increasing the bit rate. That is, at the time of switching the bit rate, segment data with a higher bit rate is downloaded (acquired).

In such a case, the client device 11 sets, as the effect period Te, a period of a predetermined length in which the switching point indicated by the arrow A41 is a start time. Then, the client device 11 performs effect processing on the video data in the effect period Te, and generates video data whose bit rate, that is, the quality gradually increases, as video data for presentation. In this case, the end position of the effect period Te may be any reproduction time.

In the client device 11, the period after the switching point is regarded as the effect period Te, and the effect processing is performed immediately after the switching point, that is, immediately after the Bitrate Adaptation.

At the start time of the effect period Te, the video resolution is 2K (1920 × 1080), and at the end time of the effect period Te, the video data for presentation with the video resolution of 4K (3840 × 2160) is generated by the effect processing .

At this time, between the start time and the end time of the effect period Te, the video data for presentation is generated so that the resolution of the video changes continuously (smoothly) linearly or nonlinearly from 2K to 4K. Be done.

In particular, in this example, the display is started with the quality that the resolution is reduced to 2K, which is the quality of the video before switching, with respect to the segment data SG43 of the 4K video originally, and the original quality is gradually increased. Effect processing is performed such that the quality rises (recovers) to 4K.

In the following, the reproduction data whose quality is raised (restored) to the original quality with time as described above, that is, the effect processing which can obtain video data and audio data, will be particularly referred to as quality restoration effect processing.

When the video of the content is reproduced based on the video data for presentation in the effect period Te obtained by the quality recovery effect processing, the video is gradually transmitted from the start time of the effect period Te which is a switching point at which the bit rate is switched by Bitrate Adaptation. Quality starts to rise. Thereafter, the quality of the video rises with time, and when the end time of the effect period Te comes, the quality of the video becomes equal to the quality after switching, which is reproduced later.

That is, when the bit rate of the video data is switched to a higher bit rate when switching the bit rate, that is, when the quality of the video is switched to a higher quality, the quality of the higher quality video data after switching is Recovery effect processing is performed.

As a result, the quality of the video smoothly changes with time from the quality of the video before switching to the quality of the video after switching, and it is possible to reduce discomfort when watching content. As a result, it is possible to suppress the loss of immersion when watching content.

As described above, in the client device 11, when the bit rate of segment data changes before and after the switching point, the effect process is performed on the segment data having the higher bit rate. In other words, when playback is switched from playback based on playback data to playback based on other playback data of different quality, at the time of playback based on playback data of higher quality among those playback data, The quality of reproduction data is controlled to change with time. By this effect processing, presentation reproduction data (video data or audio data) whose quality changes smoothly with time from the quality before switching to the quality after switching can be obtained.

In addition, what is necessary is just to select and use an appropriate method from the existing methods about the effect process with respect to video data and audio | voice data. For example, in the case of effect processing on video data, filtering processing using a soft filter or resolution conversion processing such as pixel thinning may be performed on video data.

Also, at each reproduction time within the effect period Te, for example, as shown in FIGS. 6 to 9, the quality of the reproduction data for presentation, that is, the video data for presentation and the audio data for presentation may be changed. . In particular, FIGS. 6 and 7 show examples of switching to a lower bit rate at the switch point, and FIGS. 8 and 9 show examples of switching to a higher bit rate at the switch point.

6 to 9, the horizontal axis indicates the reproduction time of the content, and the vertical axis indicates the reproduction quality of the reproduction data.

Hereinafter, an arbitrary reproduction time in the effect period Te is also described as a reproduction time t, a start time of the effect period Te is also described as a start time Tst, and a length of the effect period Te is also described as a period length Δ. Also, in the following, quality such as resolution of presentation reproduction data, that is, resolution of presentation video data and presentation audio data will be particularly referred to as reproduction quality.

For example, in the example shown in FIG. 6, the bit rate of segment data to be downloaded is switched to a lower bit rate at the switching point.

In this case, the client apparatus 11 sets the reproduction quality of the reproduction data at the reproduction time before the switching, that is, at the reproduction time before the start of the effect period Te to 100% and the reproduction quality of the reproduction data at the reproduction time after the switching point to Y%. Do.

Here, assuming that the original quality of segment data before switching is A and the original quality of segment data after switching is B, the value of Y is 100 × B / A. That is, the value of Y indicates what percentage of the quality of segment data after switching corresponds to the quality of segment data before switching.

The client device 11 changes the reproduction quality at each reproduction time t such that the reproduction quality of the reproduction data linearly decreases, ie, linearly decreases from 100% to Y% in the effect period Te.

In the example shown in FIG. 6, the reproduction quality is 100% until the start time Tst of the effect period Te, and then the reproduction quality decreases linearly with time, and the end of the effect period Te which is also a switching point At time (Tst + Δ), the reproduction quality is Y%. After that, the reproduction quality is kept at Y%.

Similarly, for example, in the example shown in FIG. 7, the bit rate of segment data to be downloaded is switched to a lower bit rate at the switching point.

In this case, the client apparatus 11 sets the reproduction quality of the reproduction data at the reproduction time before the switching, that is, at the reproduction time before the start of the effect period Te to 100%, and the reproduction quality of the reproduction data at the reproduction time after the switching point to Y%. Do.

Then, the client device 11 changes the reproduction quality at each reproduction time t so that the reproduction quality of the reproduction data decreases nonlinearly from 100% to Y% in the effect period Te.

In the example shown in FIG. 7, the reproduction quality is 100% until the start time Tst of the effect period Te, and thereafter, the reproduction quality gradually decreases with time, and the end of the effect period Te which is also a switching point At time (Tst + Δ), the reproduction quality is Y%. After that, the reproduction quality is kept at Y%.

On the other hand, for example, in the example shown in FIG. 8, the bit rate of segment data to be downloaded is switched to a higher bit rate at the switching point.

In this case, the client device 11 sets the reproduction quality of the reproduction data at the reproduction time before the switching point, that is, the reproduction time before the start time Tst of the effect period Te, to Y%, and exceeds the end time (Tst + Δ) of the effect period Te. The reproduction quality of reproduction data at a later reproduction time is assumed to be 100%.

Here, assuming that the original quality of segment data before switching is B and the original quality of segment data after switching is A, the value of Y is 100 × B / A. That is, the value of Y indicates what percentage of the quality of segment data before switching corresponds to the quality of segment data after switching.

The client device 11 changes the reproduction quality at each reproduction time t such that the reproduction quality of the reproduction data linearly increases (increases) linearly from Y% to 100% in the effect period Te.

In the example shown in FIG. 8, the reproduction quality is Y% until the start time Tst of the effect period Te which is also the switching point, and then the reproduction quality linearly increases with time, and the end of the effect period Te At time (Tst + Δ), the reproduction quality is 100%. After that, the reproduction quality is kept at 100%.

Similarly, for example, in the example shown in FIG. 9, the bit rate of segment data to be downloaded is switched to a higher bit rate at the switching point.

In this case, the client device 11 sets the reproduction quality of the reproduction data at the reproduction time before the switching point, that is, the reproduction time before the start time Tst of the effect period Te, to Y%, and exceeds the end time (Tst + Δ) of the effect period Te. The reproduction quality of reproduction data at a later reproduction time is assumed to be 100%.

Then, the client apparatus 11 changes the reproduction quality at each reproduction time t so that the reproduction quality of the reproduction data increases (increases) in a curved manner, that is, nonlinearly from Y% to 100% in the effect period Te.

In the example shown in FIG. 9, the reproduction quality is Y% until the start time Tst of the effect period Te which is also the switching point, and thereafter, the reproduction quality increases with time and the end of the effect period Te At time (Tst + Δ), the reproduction quality is 100%. After that, the reproduction quality is kept at 100%.

As shown in FIG. 6 to FIG. 9 above, if the reproduction quality is continuously changed, the difference in bit rate of the reproduction data before and after switching, that is, the quality is smooth even when the difference in the original quality of the reproduction data is large. Makes it possible to reduce the sense of incongruity at the time of viewing.

Although the same can be said for general network streaming, in MPEG-DASH streaming, segment data before and after the switching point by Bitrate Adaptation has a mechanism for caching downloaded segment data in operation. At present, segment data is generally cached on the client side for several seconds to several tens of seconds ahead at the time of reproduction.

Therefore, when it is decided to perform Bitrate Adaptation, that is, when effect processing is to be performed in a predetermined period before the switching point after the reproduction time at the current reproduction time, it is sufficient to perform the effect processing already. Segment data of such a large amount of data is rarely held (cached).

That is, for example, segment data of a higher bit rate is downloaded as shown in FIG. 10, and at the present time, it is assumed that a portion of reproduction time shown by arrow A51 is reproduced based on segment data SG51.

Also, at the present time, downloading is completed up to the portion of the reproduction time shown by arrow A52, that is, to the middle part of segment data SG54, and the portion from the segment data SG51 to the reproduction time shown by arrow A52 of segment SG54 is cached. It is assumed that In FIG. 10, the horizontal direction indicates time, and each square indicates segment data.

Usually, the client device downloads and caches one or more segment data in advance, and when actually reproducing, it reads out video data and audio data from the cache and supplies them to a decoder.

In such a case, for example, it is assumed that the switching of the bit rate is determined when the portion of the reproduction time indicated by the arrow A51 is being reproduced.

At the time of switching the bit rate, when downloading of the segment data SG54 of the segment data SG54 during downloading of the bit rate before switching is completed, the reproduction time at the end of the segment data SG54 is taken as the switching point. In this example, the playback time indicated by the arrow A53 is taken as the switching point.

Then, after that, the download of the lower bit rate segment data SG55 after switching is started, with the switching point being the head of the segment.

Therefore, in this example, even if the switching of the bit rate is determined during the reproduction of the portion indicated by the arrow A51, the period from the reproduction time indicated by the arrow A51 to the switching point is sufficiently long for effecting and the period The segment data of the bit rate before switching in is being cached.

Thus, in the MPEG-DASH streaming, it is possible to secure an effect period Te of a sufficient length immediately before the switching point.

Also, for the playback time after the switching point, the segment data of the switching point portion is already read ahead and decoded before the switching point reaches the playback time during playback, that is, before the playback reaches the switching point. Has been done. Therefore, it is possible to secure a sufficient time for effect processing on the reproduction data of the segment of the switching point.

As described above, even when the effect period Te is immediately after the switching point, a sufficient amount of segment data can be secured for performing the effect processing, and the effect processing is performed until the reproduction timing is reached. It is possible to secure enough time for

<Description of quality adjustment effect control processing>
Here, specific processing performed by the client device 11 described above will be described.

First, when the segment data is downloaded and the content is reproduced, the quality adjustment effect control process for determining the effect period Te and the reproduction quality during the effect period Te and controlling the execution of the effect process on the reproduction data will be described. .

In this quality adjustment effect control process, when switching of the bit rate occurs after the content reproduction on the client apparatus 11 is instructed, that is, when reproduction switching is performed to reproduction based on reproduction data having different qualities. To be done. The quality adjustment effect control process may be performed separately for video and audio, or one quality adjustment effect control process common to video and audio may be performed.

For example, when only the bit rate of segment data of video is changed and the bit rate of segment data of audio is not changed, the quality adjustment effect control process is performed only on the video. Also, for example, when the bit rates of both video and audio segment data are changed, quality adjustment effect control processing may be performed separately for video and audio, or one quality adjustment effect control for video and audio. Processing may be performed.

Hereinafter, the quality adjustment effect control processing by the client device 11 will be described with reference to the flowchart of FIG.

In step S11, the reproduction control unit 25 determines whether to reduce the reproduction quality of the content to the switching point, based on the segment information of each segment before and after switching supplied from the HTTP download control unit 21.

For example, when the bit rate of segment data before switching is higher than the bit rate of segment data after switching, it is determined that the reproduction quality is reduced.

When it is determined in step S11 that the reproduction quality is to be reduced, in step S12, the reproduction control unit 25 determines the start time Tst and the period length Δ of the effect period Te.

For example, the period length Δ may be a predetermined time, or may be determined based on the segment data of the bit rate before switching, that is, the cache amount of the segment data currently reproduced.

However, the period length Δ is determined so as to be shorter than the length of the period from the reproduction time currently being reproduced to the switching point. The playback time serving as the switching point is, for example, the playback time of the end position of the segment data of the bit rate before switching, which has been downloaded by the HTTP download control unit 21. In this case, the reproduction time of the switching point is the end time (Tst + Δ) of the effect period Te.

In addition, when effect processing is performed for both video and audio, the effect durations of video and audio should be set so that they are the same or substantially the same. Can. In this way, the durations of the effects applied to the video and the audio become substantially the same, and it is possible to make it difficult for the user to feel discomfort due to the change in reproduction quality. Of course, it is also possible to make the effect period different for video and audio.

In step S13, the reproduction control unit 25 sets the reproduction quality at the start time Tst of the effect period Te to 100%, and determines the reproduction quality Y% (where Y <100) at the end time (Tst + Δ) of the effect period Te. For example, as described with reference to FIG. 6 and FIG. 7, the ratio of the quality of the reproduction data at the reproduction time after the switching point to the original quality of the reproduction data before switching is Y%.

The value of the reproduction quality Y at the end time (Tst + Δ) may not necessarily coincide with the reproduction quality of the segment data after the switching, as long as the value is substantially the same.

In step S14, the reproduction control unit 25 performs an effect process on the reproduction data in the video effector 32 or the audio effector 34 such that the reproduction quality at the reproduction time t to be processed of the content becomes X% (where Y <X <100). Control. That is, the reproduction control unit 25 executes the quality reduction effect processing to reduce the reproduction quality at the reproduction time t within the effect period Te of the reproduction data.

For example, in the example shown in FIG. 6, X = ((Y−100) (t−Tst) / Δ) +100, and the quality reduction effect processing in the video effector 32 or the audio effector 34 originally causes the quality of reproduced data to be originally The operation of the video effector 32 or the audio effector 34 is controlled to be X% quality of the reproduction data quality.

Therefore, in this case, reproduction is performed with the original quality of the reproduction data until the start time Tst, and control is performed so that the quality of the reproduction data starts to decrease with time at the start time Tst. Then, at the end time (Tst + Δ) which is the portion immediately before the switching point, the quality of the reproduction data is made to be substantially the same as the quality of the reproduction data after switching. After the switching point, the original quality of the reproduction data is Is controlled to be played back.

In step S15, the reproduction control unit 25 determines whether to end the effect processing.

For example, when the reproduction time t of the reproduction data to be processed becomes a time after the end time (Tst + Δ) of the effect period Te, it is determined that the effect processing is to be ended.

Specifically, for example, when Tst ≦ t ≦ Tst + Δ holds for the reproduction time t, it is determined that the effect processing is not to be ended yet.

If it is determined in step S15 that the effect processing is not to be ended, the processing returns to step S14, and the above-described processing is repeatedly performed until the effect period Te ends. That is, the next reproduction time is set as the reproduction time t to be processed, and the quality adjustment of the reproduction data of the reproduction time t is controlled.

On the other hand, when it is determined in step S15 that the effect processing is to be ended, control is performed so that the effect processing is ended, and the quality adjustment effect control processing is ended.

If it is determined in step S11 that the reproduction quality is not to be reduced, that is, if the reproduction quality is to be increased after the switching point, the process proceeds to step S16.

In step S16, the reproduction control unit 25 determines the start time Tst of the effect period Te and the period length Δ.

Here, for example, the reproduction time of the end position of the segment data of the bit rate before switching, which is downloaded by the HTTP download control unit 21 is set as the switching point, and the switching point is set as the start time Tst.

Further, for example, as in the case of step S12, the period length Δ may be a predetermined time, or may be determined based on the cache amount or the like. Here, the period length Δ may be any length of time, such as a time longer than the segment length, and the reproduction data at the end time (Tst + Δ) may not be cached yet.

When effect processing is performed for both video and audio, the effect period may be determined such that the effect period for video and the effect period for audio are the same or substantially the same. Also, the effect period may be different between video and audio.

In step S17, the reproduction control unit 25 sets the reproduction quality at the end time (Tst + Δ) of the effect period Te to 100%, and determines the reproduction quality Y% (Y <100) at the start time Tst of the effect period Te. For example, as described with reference to FIGS. 8 and 9, the ratio of the reproduction quality of the reproduction data at the reproduction time before the switching point to the quality of the reproduction data after the switching is Y%.

The value of the reproduction quality Y at the start time Tst may not necessarily match the reproduction quality of the segment data before switching, as long as the value is substantially the same.

In step S18, the reproduction control unit 25 performs an effect process on the reproduction data in the video effector 32 or the audio effector 34 such that the reproduction quality at the reproduction time t to be processed of the content becomes X% (where Y <X <100). Control. That is, the reproduction control unit 25 executes the quality recovery effect processing to reduce the reproduction quality at the reproduction time t within the effect period Te of the reproduction data.

For example, in the example shown in FIG. 8, X = ((100−Y) (t−Tst) / Δ) + Y, and the quality recovery effect processing in the video effector 32 or the audio effector 34 reproduces reproduction data at reproduction time t. The operation of the video effector 32 or the audio effector 34 is controlled such that the quality of the video is originally X% of the quality of the reproduction data.

Therefore, in this case, reproduction is performed with the original quality of the reproduction data up to the switching point, and in the portion immediately after the start time Tst which is the switching point, the quality of the reproduction data after switching is the original of the reproduction data before switching Control is performed to be substantially the same as the quality. After that, the quality of the reproduction data increases with time, and at the end time (Tst + Δ), control is performed such that reproduction is performed with substantially the same quality as the original quality of the reproduction data.

In step S19, the reproduction control unit 25 determines whether to end the effect processing.

For example, when the reproduction time t of the reproduction data to be processed becomes a time after the end time (Tst + Δ) of the effect period Te, it is determined that the effect processing is to be ended.

If it is determined in step S19 that the effect process is not to be ended, the process returns to step S18, and the above-described process is repeated until the effect period Te ends. That is, the next reproduction time is set as the reproduction time t to be processed, and the quality adjustment of the reproduction data of the reproduction time t is controlled.

On the other hand, when it is determined in step S19 that the effect processing is to be ended, control is performed so that the effect processing is ended, and the quality adjustment effect control processing is ended.

As described above, when the switching of the bit rate occurs, the reproduction control unit 25 determines the effect period Te, and during the effect period Te, the reproduction processing unit performs the effect processing so that the reproduction quality of the reproduction data changes continuously with time. Control the execution.

As described above, by controlling the effect processing on the reproduction data of the video and the audio so that the perceived quality of the content gradually changes, it is possible to reduce the discomfort at the time of viewing.

In particular, when there is a sudden transmission band fluctuation or when there are few variations of bit rate change prepared on the server side, the change in the image quality and sound quality of the content at the time of stream change is large. The present technology is useful when it is easy. Further, according to the present technology, it is possible to reduce the discomfort at the time of viewing and listening only by the implementation in the client apparatus 11 without changing the mechanism of the MPEG-DASH distribution.

<Description of content reproduction processing>
Subsequently, a content reproduction process performed simultaneously with the quality adjustment effect control process described with reference to FIG. 11 will be described. That is, the content reproduction processing by the client device 11 will be described below with reference to the flowchart of FIG.

In step S41, the HTTP download control unit 21 downloads segment data for the video and audio of the content from the server and supplies the segment data to the segment parser 23. That is, the HTTP download control unit 21 requests the server to transmit segment data of a desired bit rate, and receives the segment data transmitted from the server in response to the request.

In step S42, the segment parser 23 parses the segment data supplied from the HTTP download control unit 21.

That is, the segment parser 23 reads the video data from the segment data of the video and supplies the video data to the video decoder 31, and reads the audio data from the segment data of the audio and supplies the audio data to the audio decoder 33.

In step S 43, the video decoder 31 decodes the video data supplied from the segment parser 23 and supplies the video data to the video effector 32.

In step S44, the video effector 32 applies an effect to the video data supplied from the video decoder 31 according to the control of the reproduction control unit 25.

That is, for example, the video effector 32 reduces the quality of the video data by performing the effect processing according to the control of the playback control unit 25 on the video data at the playback time within the effect period Te of the video. Video data is used as video data for presentation. In this case, the effect processing is performed, for example, in accordance with the control performed in step S14 or step S18 of FIG.

Specifically, for example, when the reproduction quality of the video data is lowered by reducing the resolution, the video effector 32 performs thinning processing or the like as the effect processing on the video data, and video data of lower resolution (reproduction quality) Generate

Also, for example, with respect to video data at a playback time outside the effect period Te for a video, the video effector 32 does not particularly perform the effect processing on the video data, and the video data supplied from the video decoder 31 is presented as it is Image data of

In step S45, the video effector 32 outputs the video data for presentation obtained in the process of step S44 to the subsequent stage. At this time, the image data is processed into data in a form to be output to a device at a later stage as necessary.

In step S46, the audio decoder 33 decodes the audio data supplied from the segment parser 23 and supplies the audio data to the audio effector 34.

In step S47, the audio effector 34 applies an effect to the audio data supplied from the audio decoder 33.

That is, for example, the audio effector 34 reduces the quality of the audio data by performing the effect processing according to the control of the reproduction control unit 25 on the audio data at the reproduction time within the effect period Te for audio. Voice data is assumed to be voice data for presentation. In this case, the effect processing is performed, for example, in accordance with the control performed in step S14 or step S18 of FIG.

Specifically, for example, when the reproduction quality of audio data is reduced by reducing the sampling rate, the audio effector 34 performs thinning processing or the like on the audio data as effect processing, and Generate voice data.

Also, for example, the audio effector 34 does not perform the effect processing on the audio data particularly for the audio data at the playback time outside the effect period Te for audio, and the audio data supplied from the audio decoder 33 is presented as it is Audio data of

In step S48, the audio effector 34 outputs the audio data for presentation obtained in the process of step S47 to the subsequent stage, and the content reproduction process ends. At this time, the audio data is processed into data in a form to be output to a device at a later stage as necessary.

Further, in more detail, the processing of step S43 to step S45 and the processing of step S46 to step S48 are performed in parallel.

As described above, the client device 11 decodes the video data and the audio data, performs an effect process on the video data and the audio data as appropriate, and generates and outputs the video data and the audio data for presentation.

By appropriately applying effects to the video data and the audio data, it is possible to reduce the sense of discomfort in the user's viewing sensation.

Second Embodiment
<Adjustment of playback quality of content>
By the way, although the adjustment of the reproduction quality described above is realized by the client device 11 alone, there is a possibility that the service providing side, that is, the content providing side may want to control the process related to the reproduction quality of the content. For example, there may be cases where the client side does not want to adjust the quality independently of the service provider's intention, or the client side may want to convey the processing procedure to be recommended when adjusting the reproduction quality of the content.

For example, it is assumed that at the time of Bitrate Adaptation, two or more parameters for adjusting the reproduction quality of reproduction data of content, that is, parameters for changing the quality (hereinafter also referred to as quality adjustment parameters) may be changed. In such a case, if it is possible for the service provider to indicate which quality adjustment parameter adjustment should be prioritized, it is possible to adjust the reproduction quality as intended by the service provider.

In addition, it may be considered that the service providing side can also present the method of effect processing itself. Furthermore, in order to change a plurality of quality adjustment parameters, it may be considered that the service provider can also present which order is effective as the order of changing the quality adjustment parameters. .

Therefore, hereinafter, an embodiment will be described in which information on quality adjustment parameters can be notified from the service providing side to the client side.

The quality adjustment parameters are, for example, the resolution, the frame rate, the bit depth of the pixel value of the pixel of the video (image), the color gamut of the video, etc. It is assumed that the bit depth of sample value etc. Further, information on quality adjustment parameters may be stored, for example, in an MPD file.

Here, first, the service providing side prepares designated parameter information indicating the quality adjustment parameter to be preferentially selected in adjusting the playback quality of the content, and the client side adjusts the playback quality of the content based on the designated parameter information Will be described as an example.

In such a case, for example, designated parameter information is stored in the MPD file.

That is, SupplementalProperty elements are provided in the Adaptation Set of the MPD file, an appropriate ID is newly added to the scheme IdUri according to the purpose, and a quality adjustment parameter to be designated is designated as value.

For example, here, the designated parameter information is information indicating which quality adjustment parameter adjustment is effective for adjustment of reproduction quality, and information (ID) stored in schemeIdUri, designated value, and the like The information consisting of is made designated parameter information.

In this case, for example, “urn: mpeg: dash: bitrate_adaptation_param: 2017” is stored in the scheme IdUri. Then, for the image, any one of a plurality of values such as “resolution”, “frame_rate”, “bit_depth”, and “color_primaries” is specified (described) as the value. In addition, for speech, any one of a plurality of values such as "sampling_rate" and "bits_per_sample" is specified (described) as the value.

Here, "resolution" indicates the resolution of the image, "frame_rate" indicates the frame rate of the image, "bit_depth" indicates the bit depth of the pixel value of the pixel of the image, and "color_primaries" indicates It shows the color gamut of the image. Also, “sampling_rate” indicates the sampling rate of speech, and “bits_per_sample” indicates the bit depth of sample values of speech samples.

Thus, when designated parameter information is stored in the MPD file, the MPD file is as shown in FIG. 13, for example.

In the example shown in FIG. 13, designated parameter information is stored for each of video and audio.

That is, for example, “urn: mpeg: dash: bitrate_adaptation_param: 2017” is stored as a schemeIdUri of a video in a portion indicated by an arrow Q11, and “resolution” is stored as a value. From these pieces of information, it can be understood that the resolution is designated as the quality adjustment parameter of the video by the designated parameter information in the client device 11.

Similarly, "urn: mpeg: dash: bitrate_adaptation_param: 2017" is stored as a schemeIdUri for voice in the part shown by the arrow Q12, and "sampling_rate" is stored as a value. From these pieces of information, it can be understood that the sampling rate is designated as the voice quality adjustment parameter by the designated parameter information in the client device 11.

When the MPD file shown in FIG. 13 is acquired from the server, the client device 11 adjusts the reproduction quality of the video data by changing the resolution as the quality adjustment parameter in the effect period. In addition, the client device 11 adjusts the reproduction quality of the audio data by changing the sampling rate as the quality adjustment parameter.

On the service providing side, among the plurality of quality adjustment parameters, the quality adjustment parameter that can most effectively reduce the reproduction quality may be designated by the designated parameter information. Note that one quality adjustment parameter may be designated for the entire content (stream) or may be designated for each segment.

<Description of quality adjustment effect control processing>
Next, quality adjustment effect control processing performed by the client device 11 when designated parameter information is stored in the MPD file will be described with reference to the flowchart in FIG.

In step S71, the reproduction control unit 25 acquires designated parameter information based on the decoding result of the MPD file supplied from the MPD parser 22. That is, the reproduction control unit 25 reads designated parameter information from the MPD file.

For example, when the MPD file shown in FIG. 13 is received, the reproduction control unit 25 uses, as designated parameter information, information indicating the resolution as the quality adjustment parameter, and information indicating the sampling rate as the quality adjustment parameter. get.

In step S72, the reproduction control unit 25 determines quality adjustment parameters to be adjusted at the time of reproduction quality adjustment of reproduction data, that is, at the time of effect processing, for video and audio based on the designated parameter information acquired at step S71.

For example, the reproduction control unit 25 sets the quality adjustment parameter indicated by the designated parameter information as the quality adjustment parameter to be adjusted at the time of reproduction quality adjustment of reproduction data. Therefore, in the example shown in FIG. 13, the resolution is selected as the image quality adjustment parameter, and the sampling rate is selected as the sound quality adjustment parameter.

After the quality adjustment parameters are determined, the processes of steps S73 to S81 are performed to end the quality adjustment effect control process, but these processes are similar to the processes of steps S11 to S19 of FIG. Therefore, the explanation is omitted.

However, in step S76 and step S80, the reproduction control unit 25 changes the quality adjustment parameter determined in step S72 to reduce the reproduction quality of the reproduction data of the content, thereby reducing the video effector 32 and the audio effector 34. Control the Therefore, in step S44 or step S47 of FIG. 12, the quality adjustment parameter designated by the reproduction control unit 25 among the plurality of quality adjustment parameters is adjusted.

As described above, the client apparatus 11 changes the reproduction quality of the reproduction data by reading the designated parameter information from the MPD file and adjusting the quality adjustment parameter indicated by the designated parameter information. This makes it possible to perform reproduction quality adjustment with more effective quality adjustment parameters, and to change the reproduction quality of the content more effectively and more smoothly.

Although the case where the designated parameter information is stored in the MPD file has been described as an example here, the designated parameter information may be supplied from the outside, or the designated parameter information may be stored in the segment data. In particular, when the designated parameter information changes dynamically for each segment, the designated parameter information may be stored in the segment data.

Also, instead of designating the quality adjustment parameter by the designated parameter information, the priorities of the plurality of quality adjustment parameters may be stored in the MPD file. In such a case, for example, the priority of each quality adjustment parameter in the client apparatus 11, the quality adjustment parameter adjustable on the client side, etc. are considered, and the quality adjustment parameter to be actually used for adjustment is selected.

Third Embodiment
<Adjustment of playback quality of content>
Furthermore, for example, when adjusting reproduction quality of reproduction data by adjusting a plurality of quality adjustment parameters, the order information indicating in what order it is more effective to adjust those plurality of quality adjustment parameters is MPD It may be stored in a file. That is, order information indicating a plurality of quality adjustment parameters for changing the quality of reproduction data and an adjustment order of the quality adjustment parameters may be acquired from the MPD file.

In such a case, information including the information (ID) stored in the schemeIdUri of the MPD file and the designated value is used as the order information. At this time, for example, “urn: mpeg: dash: bitrate_adaptation_param_processing_order: 2017” is stored in the scheme IdUri.

Then, for a video, two or more of a plurality of values such as “resolution”, “frame_rate”, “bit_depth”, and “color_primaries” are designated (descripted) as values.

In addition, with regard to speech, two or more of a plurality of values such as "sampling_rate" and "bits_per_sample" are designated (described) as values. In the description of value, each quality adjustment parameter is arranged in the order to be processed with delimiters such as comma and semicolon.

Thus, when order information is stored in the MPD file, the MPD file is as shown in FIG. 15, for example.

In the example shown in FIG. 15, the MPD file stores order information on the video.

That is, for example, "urn: mpeg: dash: bitrate_adaptation_param_processing_order: 2017" is stored as a schemeIdUri of a video in a portion indicated by an arrow Q21, and "resolution, frame_rate" is stored as a value.

From these pieces of information, it is understood that the client device 11 specifies the resolution and the frame rate as the video quality adjustment parameters by the order information, and the frame rate adjustment should be performed after the resolution adjustment. That is, according to the order information, a plurality of quality adjustment parameters to be adjusted and the order (order) of adjustment with the quality adjustment parameters can be known.

When the MPD file shown in FIG. 15 is obtained from the server, the client apparatus 11 first changes the resolution as the quality adjustment parameter in the effect period, and then the quality of the image data obtained as a result is further changed. The reproduction quality of the video data is adjusted by changing the frame rate as an adjustment parameter.

On the service provider side, among a plurality of quality adjustment parameters, two or more quality adjustment parameters that can most effectively reduce reproduction quality and an effective order in which those quality adjustment parameters should be adjusted Information indicating (order) is generated as order information. Note that one order information may be specified for the entire content (stream) or may be specified for each segment.

<Description of quality adjustment effect control processing>
Next, quality adjustment effect control processing performed by the client device 11 when order information is stored in the MPD file will be described with reference to the flowchart in FIG.

In step S121, the reproduction control unit 25 acquires order information based on the decoding result of the MPD file supplied from the MPD parser 22. That is, the reproduction control unit 25 reads out the order information from the MPD file.

For example, when the MPD file shown in FIG. 15 is received, the reproduction control unit 25 acquires, as the order information, information indicating that the frame rate should be adjusted after adjusting the resolution.

In step S122, the reproduction control unit 25 performs, on the basis of the order information acquired in step S121, quality adjustment parameters for performing adjustment at the time of reproduction quality adjustment of reproduction data, that is, at the time of effect processing with respect to video and audio. Determine the adjustment order of adjustment parameters.

For example, the reproduction control unit 25 sets the quality adjustment parameter and the adjustment order indicated by the order information as the adjustment order of the quality adjustment parameter to be adjusted and the quality adjustment parameter. Therefore, in the example shown in FIG. 15, it is decided to adjust the frame rate after adjusting the resolution as the image quality adjustment parameter.

After the quality adjustment parameters and the adjustment order are determined, the processes of steps S123 to S131 are performed to end the quality adjustment effect control process, but these processes are the same as the processes of steps S11 to S19 of FIG. The description is omitted because it is similar.

However, in step S126 or step S130, the reproduction control unit 25 changes the plurality of quality adjustment parameters in the order determined in step S122 to reduce the reproduction quality of the reproduction data of the content. And controls the audio effector 34. Therefore, in step S44 and step S47 of FIG. 12, the plurality of quality adjustment parameters designated by the reproduction control unit 25 are adjusted in the designated order.

As described above, the client apparatus 11 changes the reproduction quality of the reproduction data by reading out the order information from the MPD file and adjusting the quality adjustment parameter indicated by the order information in the order indicated by the order information. As a result, the reproduction quality adjustment can be performed more effectively, and the reproduction quality of the content can be changed more effectively and smoothly.

In addition, there may be a plurality of methods as a method of changing one quality adjustment parameter. Specifically, for example, the resolution as the quality adjustment parameter can be adjusted by filter processing such as a soft filter, or can be adjusted by resolution conversion processing. Even when the adjustment of the same quality adjustment parameter is performed, depending on what kind of process is used to perform the adjustment, the sense of incongruity that occurs when the content after adjustment is reproduced may differ.

Therefore, for example, processing for adjusting a predetermined quality adjustment parameter (hereinafter, also referred to as adjustment processing) may be specified on the service providing side.

In such a case, for example, adjustment processing specification information indicating the adjustment processing to be performed when adjusting the quality adjustment parameter is stored in the MPD file. The adjustment process specification information is information for specifying the adjustment process for changing the quality adjustment parameter for changing the quality of the reproduction data.

When the adjustment process specification information is stored in the MPD file and supplied, for example, the adjustment process specification information is read together with the specified parameter information and the order information in step S71 of FIG. 14 and step S121 of FIG. Furthermore, in step S72 of FIG. 14 and step S122 of FIG. 16, not only the quality adjustment parameter to be adjusted but also adjustment processing for the quality adjustment parameter are determined.

Therefore, in step S76 and step S80 of FIG. 14 and step S126 and step S130 of FIG. 16, adjustment of the quality adjustment parameter is performed by the adjustment process determined based on the adjustment process specification information of the plurality of adjustment processes. Control takes place. By doing this, it is possible to adjust the reproduction quality of the content more effectively so as not to cause discomfort.

Although the example in which the adjustment process specification information is stored in the MPD file together with the specified parameter information and the order information has been described here, only the adjustment process specification information may be stored in the MPD file. In such a case, the client device 11 changes the predetermined quality adjustment parameter or the arbitrarily determined quality adjustment parameter by performing the adjustment process indicated by the adjustment process specification information.

Further, although the example in which the present technology is applied to the MPEG-DASH has been described above, the present invention is not limited to this, and the present technology is also applicable to content delivery etc. that can implement the same technology as Bitrate Adaptation of the MPEG-DASH. It can apply. In this case, the present technology may be applied, for example, when performing quality control of video and audio.

<Configuration example of computer>
By the way, the series of processes described above can be executed by hardware or software. When the series of processes are performed by software, a program that configures the software is installed on a computer. Here, the computer includes, for example, a general-purpose computer capable of executing various functions by installing a computer incorporated in dedicated hardware and various programs.

FIG. 17 is a block diagram showing an example of a hardware configuration of a computer that executes the series of processes described above according to a program.

In the computer, a central processing unit (CPU) 501, a read only memory (ROM) 502, and a random access memory (RAM) 503 are mutually connected by a bus 504.

Further, an input / output interface 505 is connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

The input unit 506 includes a keyboard, a mouse, a microphone, an imaging device, and the like. The output unit 507 includes a display, a speaker, and the like. The recording unit 508 includes a hard disk, a non-volatile memory, and the like. The communication unit 509 is formed of a network interface or the like. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 501 loads, for example, the program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the above-described series. Processing is performed.

The program executed by the computer (CPU 501) can be provided by being recorded on, for example, a removable recording medium 511 as a package medium or the like. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by attaching the removable recording medium 511 to the drive 510. Also, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. In addition, the program can be installed in advance in the ROM 502 or the recording unit 508.

Note that the program executed by the computer may be a program that performs processing in chronological order according to the order described in this specification, in parallel, or when necessary, such as when a call is made. It may be a program to be processed.

Further, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.

For example, the present technology can have a cloud computing configuration in which one function is shared and processed by a plurality of devices via a network.

Further, each step described in the above-described flowchart can be executed by one device or in a shared manner by a plurality of devices.

Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step can be executed by being shared by a plurality of devices in addition to being executed by one device.

Further, the effects described in the present specification are merely examples and are not limited, and other effects may be present.

Furthermore, the present technology can also be configured as follows.

(1)
When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second An image processing apparatus comprising: a reproduction control unit configured to control the quality of the reproduction data to change with time when reproducing reproduction data of higher quality among reproduction data.
(2)
The image processing apparatus according to (1), wherein the first reproduction data and the second reproduction data are reproduction data of different bit rates.
(3)
When the quality of the first reproduction data is higher than the quality of the second reproduction data, the reproduction control unit performs the process in the portion of the first reproduction data immediately before switching to the second reproduction data. The image processing apparatus according to (1) or (2), wherein the quality of the first reproduction data is changed such that the quality of the first reproduction data becomes substantially the same as the quality of the second reproduction data.
(4)
When the quality of the second reproduction data is higher than the quality of the first reproduction data, the reproduction control unit performs the process immediately after switching from the first reproduction data in the second reproduction data. The quality of the second reproduction data is changed such that the quality of the second reproduction data becomes substantially the same as the quality of the first reproduction data, and the quality of the second reproduction data becomes higher with time (1) The image processing apparatus according to any one of (3) to (3).
(5)
The image processing apparatus according to any one of (1) to (4), wherein the first reproduction data and the second reproduction data are video data.
(6)
The image processing apparatus according to (5), wherein the reproduction control unit changes the quality of the reproduction data by changing at least one of a resolution, a frame rate, a bit depth, and a color gamut of the reproduction data.
(7)
The image processing apparatus according to any one of (1) to (4), wherein the first reproduction data and the second reproduction data are audio data.
(8)
The image processing apparatus according to (7), wherein the reproduction control unit changes the quality of the reproduction data by changing at least one of a sampling rate and a bit depth of the reproduction data.
(9)
The reproduction control unit acquires specification information specifying a parameter for changing the quality of the reproduction data, and changes the parameter indicated by the specification information of the plurality of the parameters in the reproduction data. The image processing apparatus according to any one of (1) to (4), wherein the quality of the reproduction data is changed.
(10)
The reproduction control unit acquires order information indicating a plurality of parameters for changing the quality of the reproduction data and an adjustment order of the plurality of parameters, and the reproduction information is indicated by the order information in the reproduction data. The image processing apparatus according to any one of (1) to (4), wherein the quality of the reproduction data is changed by adjusting a plurality of the parameters in an adjustment order indicated by the order information.
(11)
The reproduction control unit acquires processing specification information specifying an adjustment processing for changing a parameter for changing the quality of the reproduction data, and the adjustment processing indicated by the processing specification information among a plurality of the adjustment processing The image processing apparatus according to any one of (1) to (4), wherein the quality of the reproduction data is changed by changing the parameter of the reproduction data according to.
(12)
The image processing apparatus according to any one of (9) to (11), wherein the reproduction data is video data, and the parameter is a resolution, a frame rate, a bit depth, or a color gamut.
(13)
The image processing apparatus according to any one of (9) to (11), wherein the reproduction data is audio data, and the parameter is a sampling rate or a bit depth.
(14)
The reproduction control unit controls the video data as the reproduction data and the audio data as the reproduction data to have the same period during which the quality of the reproduction data changes (1) to (13) An image processing apparatus according to any one of the preceding claims.
(15)
When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second Controlling the quality of the reproduction data to change with time at the time of reproduction of reproduction data of higher quality among the reproduction data.
(16)
When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second A program that causes a computer to execute a process including the step of controlling the quality of the reproduction data to change with time when reproducing reproduction data of higher quality among reproduction data.

11 Client Device, 21 HTTP Download Control Unit, 25 Playback Control Unit, 31 Video Decoder, 32 Video Effector, 33 Audio Decoder, 34 Audio Effector

Claims (16)

1. When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second An image processing apparatus comprising: a reproduction control unit configured to control the quality of the reproduction data to change with time when reproducing reproduction data of higher quality among reproduction data.
2. The image processing apparatus according to claim 1, wherein the first reproduction data and the second reproduction data are reproduction data of different bit rates.
3. When the quality of the first reproduction data is higher than the quality of the second reproduction data, the reproduction control unit performs the process in the portion of the first reproduction data immediately before switching to the second reproduction data. The image processing apparatus according to claim 1, wherein the quality of the first reproduction data is changed such that the quality of the first reproduction data becomes substantially the same as the quality of the second reproduction data.
4. When the quality of the second reproduction data is higher than the quality of the first reproduction data, the reproduction control unit performs the process immediately after switching from the first reproduction data in the second reproduction data. The quality of the second reproduction data is changed such that the quality of the second reproduction data becomes substantially the same as the quality of the first reproduction data, and the quality of the second reproduction data becomes higher with time. The image processing apparatus according to claim 1.
5. The image processing apparatus according to claim 1, wherein the first reproduction data and the second reproduction data are video data.
6. The image processing apparatus according to claim 5, wherein the reproduction control unit changes the quality of the reproduction data by changing at least one of a resolution, a frame rate, a bit depth, and a color gamut of the reproduction data.
7. The image processing apparatus according to claim 1, wherein the first reproduction data and the second reproduction data are audio data.
8. The image processing apparatus according to claim 7, wherein the reproduction control unit changes the quality of the reproduction data by changing at least one of a sampling rate and a bit depth of the reproduction data.
9. The reproduction control unit acquires specification information specifying a parameter for changing the quality of the reproduction data, and changes the parameter indicated by the specification information of the plurality of the parameters in the reproduction data. The image processing apparatus according to claim 1, wherein the quality of the reproduction data is changed.
10. The reproduction control unit acquires order information indicating a plurality of parameters for changing the quality of the reproduction data and an adjustment order of the plurality of parameters, and the reproduction information is indicated by the order information in the reproduction data. The image processing apparatus according to claim 1, wherein the quality of the reproduction data is changed by adjusting a plurality of the parameters in an adjustment order indicated by the order information.
11. The reproduction control unit acquires processing specification information specifying an adjustment processing for changing a parameter for changing the quality of the reproduction data, and the adjustment processing indicated by the processing specification information among a plurality of the adjustment processing The image processing apparatus according to claim 1, wherein the quality of the reproduction data is changed by changing the parameter of the reproduction data according to.
12. The image processing apparatus according to claim 9, wherein the reproduction data is video data, and the parameter is a resolution, a frame rate, a bit depth, or a color gamut.
13. The image processing apparatus according to claim 9, wherein the reproduction data is audio data, and the parameter is a sampling rate or a bit depth.
14. The image processing according to claim 1, wherein the reproduction control unit controls the video data as the reproduction data and the audio data as the reproduction data to have the same period during which the quality of the reproduction data is changed. apparatus.
15. When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second Controlling the quality of the reproduction data to change with time at the time of reproduction of reproduction data of higher quality among the reproduction data.
16. When switching from playback based on first playback data to playback based on second playback data having a quality different from that of the first playback data, the first playback data and the second A program that causes a computer to execute a process including the step of controlling the quality of the reproduction data to change with time when reproducing reproduction data of higher quality among reproduction data.

Images