JP2006101063A - Transmission of moving image data in consideration of environment of reproduction side - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology which can smoothly reproduce with a reproducing apparatus of different throughput while the deterioration of the quality of an image is suppressing in processing which transmits moving image data. <P>SOLUTION: A moving image transmitting device which transmits the moving image data is provided in a moving image data reproducing apparatus. This moving image transmitting device includes an object extraction unit for extracts a plurality of objects including an attention object from the moving image data, a compression rate determination unit for determining a compression rate for regenerating the environment of the moving image data reproducing apparatus for a plurality of objects according to reproducing environmental information representing the reproducing environment of the moving image data reproducing apparatus, a coding unit for encoding the moving image data while the plurality of objects are compressing based on the determined each compression rate, and a transmitter for transmitting the encoded moving image data. The compression rate of the attention object is set so as to become lower than the compression rate of other objects of the plurality of objects. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for compressing the data size of moving image data.

  In recent years, provision of moving image data has spread not only to communication via a recording medium such as a DVD or a wired network, but also to mobile communication such as a mobile phone and a PDA. With such expansion of communication targets, the targets of playback devices that are targets for providing moving image data are also expanding.

  Such expansion of the reproducing apparatus is a factor in increasing variation in processing capability of the reproducing apparatus. In this way, when there is a variation in processing capability among the playback devices to be provided, for example, moving image data is provided by adjusting the image size of the moving image data.

JP 2004-56632 A

  However, the adjustment of the image size has caused a problem that the visibility of the moving image may be excessively deteriorated due to the reduction of the display image size and the resolution.

  The present invention has been made to solve the above-described problems in the prior art, and in the process of transmitting moving image data, it is possible to perform smooth reproduction with a reproducing apparatus having different processing capabilities while suppressing deterioration in image quality. It aims at providing the technology to do.

The present invention is a moving image transmitting apparatus for transmitting moving image data to a moving image data reproducing apparatus,
An object extraction unit that extracts a plurality of objects including a target object from moving image data;
A compression rate determining unit that receives playback environment information representing a playback environment of the moving image data playback device and determines a compression rate for each of the plurality of objects according to the playback environment information;
An encoding unit that encodes the moving image data while compressing the plurality of objects based on the determined compression rates;
A transmission unit for transmitting the encoded moving image data;
With
The attention object is an image area including an attention area that attracts relatively attention among image areas displayed in the moving image data,
The compression rate of the object of interest is set to be lower than the compression rate of other objects among the plurality of objects.

  According to the moving image transmitting apparatus of the present invention, the object of interest which is an image region including a region of interest that is relatively focused among a plurality of objects extracted from the moving image data is encoded with a lower compression rate than the others. Therefore, it is possible to provide moving image data that can clearly reproduce an image region including a region of interest in various reception and reproduction environments.

  Here, for the “attention area”, for example, an importance index indicating the importance of the image portion may be calculated by a predetermined calculation method, and may be detected as a region having a high importance index. It may be detected. The “playback environment information” includes various types of information such as decoding performance and communication rate of the moving image data playback device, and user setting information specified by the user. An “object” is a broad concept including a partial area in an image and an AV object.

  In the moving image transmitting apparatus, the object of interest may include a specific object that is an image region including a human face.

  In this way, it is possible to provide moving image data that can clearly reproduce an image area including a human face.

In the moving image transmitting apparatus, the reproduction environment information includes decoding performance information indicating decoding performance of the moving image data reproducing apparatus,
The compression level determination unit may determine the compression level according to the decoding performance information.

  In this way, it is possible to provide moving image data corresponding to the decoding performance to various playback devices having different decoding performance. Here, the higher compression ratio of other objects can be realized by the following method, for example. That is, it can be realized by various methods such as pixel sub-sampling (thinning), reduction of the number of gradations, further reduction of high-frequency components, and diversion of frame images.

  The pixel sub-sampling is, for example, a process of thinning out pixels at a rate of one in two in at least one of the vertical direction and the horizontal direction in the “other region”. The reduction in the number of gradations can be realized, for example, by reducing the number of gradations of YCbCr from 256 gradations to 128 gradations or 64 gradations only for CbCr. The diversion of the frame image has the effect of reducing the fluctuation of the pixel value and consequently increasing the compression rate by variable length coding.

  In the moving image transmitting apparatus, the encoding unit may have an encoding processing mode for reducing an update rate of the other object.

  Objects other than the object of interest generally have little movement, so even if the update rate is reduced, the video quality degradation is small, and it is not an area of interest, so it does not cause viewers to feel excessive image quality degradation. The rate can be greatly reduced. Here, reducing the update rate includes stopping the update for at least a predetermined time.

  In the moving image transmitting apparatus, the encoding unit may have a sprite encoding processing mode for performing sprite encoding on the other object.

  If other objects are sprite-encoded, the video playback device will have image information behind the object of interest, so no blank part will occur even if the object of interest moves while the other object is stopped. Can be.

In the moving image transmitting apparatus, the reproduction environment information includes communication environment information indicating information on a communication rate between the moving image data reproducing apparatus and the moving image transmitting apparatus,
The compression level determination unit may change the compression level according to the communication environment information.

  In this way, even in a reception environment where the changing communication rate becomes a bottleneck, it is possible to provide moving image data according to the environment.

  In the moving image transmitting apparatus, the encoding unit may execute the encoding in real time at the changed compression level.

  The present invention is embodied in a carrier wave including a moving image data reproducing apparatus, a computer program for causing a computer to implement the function of the moving image data generation method or apparatus, a recording medium storing the computer program, and the computer program. It can be realized in various forms such as a converted data signal.

Next, embodiments of the present invention will be described in the following order based on examples.
A. Configuration of a moving image data providing system in an embodiment of the present invention:
B. Moving image data provision processing in the first embodiment of the present invention:
C. Moving image data provision processing in the second embodiment of the present invention:
D. Variations:

A. Configuration of a moving image data providing system in an embodiment of the present invention:
FIG. 1 is an explanatory diagram showing the configuration of the network system in the first embodiment of the present invention. This network system includes a wired LAN (Local Area Network) and a wireless LAN. The wired LAN connects a server SV capable of providing moving image data, a personal computer 220, and a wireless LAN access point 16. The wireless LAN connects the wireless LAN access point 16, the two personal computers 200 and 210, and the projector 100. As a result, the server SV for providing moving image data is connected to the three personal computers 200, 210, and 220 and the projector 100 via the wired LAN and the wireless LAN.

  The network shown in FIG. 1 is configured assuming one scene of an in-house meeting, for example. Specifically, one of the employees is making a presentation using the projector 100 as a speaker, and a plurality of employees are listening to the presentation as an audience. One of a plurality of employees uses a personal computer 220 connected to a wired LAN, and the other employees bring their personal computers 200 and 210 accessible to the wireless LAN access point 16 into the conference room for use. ing. The personal computers 200 and 210 are playback devices whose moving image data decoding performance is unknown to the server SV for providing moving image data.

  In this network, the users of the personal computers 200, 210, and 220 can request the server SV to transmit moving image data by operating the personal computers 200, 210, and 220 during an in-house meeting. A speaker using the projector 100 can display a moving image on the screen SC by operating the server SV.

  In this embodiment, not only the personal computers 200, 210, and 220 but also the projector 100 is equipped with a decoder. Since the encoded moving image data has a smaller amount of data than image data such as RGB data and YCrCb data, the restriction due to the communication environment (for example, wireless LAN) can be relaxed to realize high-quality moving image display. Because it can.

  FIG. 2 is an explanatory diagram showing the configuration of the server SV in the first embodiment of the present invention. The server SV includes a CPU 11 that executes arithmetic processing for image processing, a ROM 12, a RAM 13 that temporarily stores image data and application programs, and a hard disk 14 that stores application software and moving image data for image processing. And an I / F circuit 15 that manages an interface between an external device such as the moving image database unit 30 and the user interface unit 18 and a wired LAN.

  The moving image database unit 30 includes a digital video camera 30a, a DVD 30b, and a hard disk 30c as a moving image data supply source. The moving image data is supplied to the server SV as a set of frame image data continuous in time series. The user interface unit 18 includes a display 18a that displays an operation display screen, and a keyboard 18b and a mouse 18c that receive input from the user.

  The server SV can provide the same image for presentation to the three personal computers 200, 210, and 220 and the projector 100. For example, the speaker can give a presentation while showing a laser pointer on the screen projected by the projector 100. In this presentation, not only still images but also moving images are displayed.

  FIG. 3 is an explanatory diagram showing the contents of the hard disk 14 provided in the server SV. The hard disk 14 stores two sets of moving image data MP1 and MP2. The moving image data MP1 includes high bit rate moving image data MP1H, medium bit rate moving image data MP1M, and low bit rate moving image data MP1L for the same image. The moving image data MP2 includes high bit rate moving image data MP2H, medium bit rate moving image data MP2M, and low bit rate moving image data MP2L for the same image. As described above, the reason for having moving image data for displaying the same moving image at a plurality of bit rates is to supply moving image data having a bit rate suitable for the reproduction environment on the receiving side. is there.

  In this moving image data providing system, a moving image data providing server SV is connected to three personal computers 200, 210, and 220 and the projector 100 via a wired LAN and a wireless LAN. The three personal computers 200, 210, and 220 and the projector 100 differ in decoding performance and communication environment. For this reason, if high bit rate moving image data is provided uniformly, for example, some playback devices may not be able to cope with decoding performance or may not be able to fully display playback device performance.

  In the moving image data providing process of each embodiment to be described later, moving image data with an appropriate bit rate is transmitted according to the decoding performance and communication environment on the receiving side. It can be fully demonstrated.

B. Moving image data provision processing in the first embodiment of the present invention:
FIG. 4 is a flowchart showing the contents of the moving image data providing process in the first embodiment of the present invention. This process includes a moving image data preparation process (step S100) and a moving image data transmission process (step S200).

  The moving image data preparation process is a process in which moving image data supplied from the moving image database unit 30 is encoded to generate moving image data of three bit rates of high, medium, and low, and then stored in the hard disk 14. . The moving image data transmission process is a process for selecting one of the three stored bit rate moving image data according to the reproduction environment on the receiving side and transmitting the selected moving image data.

  FIG. 5 is a flowchart showing the contents of the moving image data preparation process in the first embodiment of the present invention. In step S110, the server SV inputs the moving image data MP1R from the moving image database unit 30. In the present embodiment, this moving image data is supplied as a set of frame image data continuous in time series before encoding.

  In step S120, the server SV extracts a region of interest from the frame image of the moving image data MP1R (FIG. 6A). The “attention area” is typically an area that is easily noticeable for humans. In the first embodiment, an area including a human face is detected as an attention area. FIG. 6B shows a state in which two attention areas PA are detected in the moving image data MP1R. This attention area PA can be set to a predetermined geometric shape including the face area. The predetermined geometric shape may be, for example, a rectangle or a multi-value shape that matches the face area.

  As a method for detecting a face region in an image, for example, Technical Report of IEICE, PRMU2001-217, pp.87-94 (2002-1), “Independent of background and face direction. The method described in “Detection of face and estimation of face direction” can be used. Although this document describes not only the detection of the face area but also the estimation of the face direction, in the present embodiment, only the detection of the face area may be performed. Various other face area detection methods can also be used.

  In step S130, the server SV determines the compression rate of compression performed when the moving image data MP1R is encoded separately for the attention area and the other areas. In the present embodiment, encoding processing is performed for a “target region” at a preset standard compression rate, and encoding processing is performed for three types of compression rates for “other regions”. It is configured.

  For example, in the generation of the high bit rate moving image data MP1H, the “other region” is encoded at the same compression rate as that of the “region of interest”, and the medium bit rate moving image data MP1M is encoded. In the generation, the encoding process is performed at a higher compression rate than the “region of interest”, and in the generation of the low bit rate moving image data MP1L, the encoding process is performed at a higher compression ratio.

  As described above, the reason why the compression rate is increased only in the “other area” is to maintain the image quality of the “attention area” so that the viewer does not feel excessive deterioration of the image quality. Furthermore, in general, the “region of interest” is smaller in area than the other regions, so that increasing the compression ratio of the other regions can have a remarkable effect.

  High compression ratio can be realized by the following method, for example. That is, it can be realized by various methods such as pixel sub-sampling (thinning), reduction of the number of gradations, further reduction of high frequency components, and diversion of frame images. The pixel sub-sampling is, for example, a process of thinning out pixels at a rate of one in two in at least one of the vertical direction and the horizontal direction in the “other region”. The reduction in the number of gradations can be realized, for example, by reducing the number of gradations of YCbCr from 256 gradations to 128 gradations or 64 gradations only for CbCr.

  The diversion of the frame image contributes to the reduction of the bit rate as follows. First, diverting the frame image reduces the fluctuation of the pixel value, so that an effect of increasing the compression rate by variable length coding is achieved. Secondly, for example, the bit rate can be reduced by reducing the update rate of the image of the non-attention area (other object).

  Objects other than the object of interest generally have little movement, so even if the update rate is reduced, the video quality degradation is small, and it is not an area of interest, so it does not cause viewers to feel excessive image quality degradation. There is an advantage that the rate can be greatly reduced. Here, reducing the update rate includes stopping the update for at least a predetermined time.

  Furthermore, the method of reducing the update rate of other objects has a more remarkable effect when combined with sprite coding. If another object with a low update rate is sprite-encoded, the video playback device will have image information behind the object of interest, so even if the object of interest moves while the other object is stopped This is because there is no blank part.

  In step S140, the server SV performs an encoding process using the set compression method. In step S150, the server SV stores the encoded three types of moving image data in the hard disk 14.

  FIG. 7 is a flowchart showing the contents of the moving image data transmission process (FIG. 5) in the first embodiment of the present invention. In step S210, the server SV receives a moving image data request signal. In this embodiment, the moving image data request signal is received from one of the three personal computers 200, 210, and 220.

  Note that this process is omitted when the server SV controls the three personal computers 200, 210, and 220 and the projector 100 via the wired LAN and the wireless LAN to display a moving image. .

  In step S220, the server SV requests decoding performance information. The decoding performance information is information indicating the decoding performance of the transmission source of the moving image data request signal. The request for the decoding performance information is made to the transmission source of the moving image data request signal.

  In step S230, the server SV selects moving image data. Selection of moving image data is performed according to the received decoding performance information. For example, based on the decoding performance represented by the decoding performance information and two threshold values, when the decoding performance is higher than either threshold value, the high bit rate moving image data MP1H is selected, and the decoding performance is between the two threshold values. In some cases, the medium bit rate moving image data MP1M is selected, and when the decoding performance is lower than any threshold, the low bit rate moving image data MP1L is selected.

  In step S240, the server SV transmits the selected moving image data to the transmission source of the moving image data request signal. In this way, moving image data having a bit rate suitable for the decoding performance of the transmission source of the moving image data request signal is transmitted.

  As described above, in the first embodiment, it is possible to transmit moving image data having a bit rate suitable for the decoding performance of the playback device without causing the viewer to feel excessive deterioration in image quality. Can provide moving image data that can be stably reproduced.

C. Moving image data provision processing in the second embodiment of the present invention:
FIG. 8 is a flowchart showing the contents of moving image data transmission processing in the second embodiment of the present invention. In this moving image data transmission process, moving image data is selected in consideration of not only the decoding performance of the moving image data request signal source but also the communication environment with the moving image data request signal source. This is different from the processing of the first embodiment. Specifically, step S230 and step S240 are changed to step S250 to step S290.

  Furthermore, the second embodiment of the present invention differs from the first embodiment in that the server SV encodes moving image data supplied from the moving image database unit 30 in real time.

  In step S250, the server SV calculates a communication rate (transmission rate). The transmission rate can be calculated, for example, by dividing the amount of moving image data transmitted in a packet whose response has been confirmed by the transmission destination by the time required for transmission.

  In step S260, the server SV determines the compression rate. The compression rate is determined according to the decoding performance information and the transmission rate received in step S220. Specifically, the compression rate is determined so that the reproduction can be performed without stopping at the performance represented by the decoding performance information and the bit rate is sufficiently smaller than the changing transmission rate.

  In step S270, the server SV performs encoding in real time based on the determined compression rate. The moving image data encoded in real time is divided into a predetermined number of packets and transmitted to the source of the moving image data request signal (step S280). The transmission of the moving image data can be completed by repeating the processing of step S250 to step S280, for example, every 10 seconds (step S290).

  As described above, in the second embodiment, it is possible to transmit moving image data having a bit rate suitable not only for the decoding performance of the playback apparatus but also for the communication environment without causing the viewer to feel excessive deterioration in image quality. Therefore, it is possible to provide moving image data that can be stably reproduced in various reception environments and communication environments.

  For example, when a certain level of communication rate is guaranteed and known, the compression rate only needs to be determined once according to the lower of the communication rate and the decoding performance. Furthermore, the user on the receiving side may specify the bit rate.

  Also, changing the bit rate during transmission is not necessarily premised on encoding in real time. For example, the moving image data of each bit rate may be divided and stored in the hard disk 14 every 10 seconds, and the bit rate may be switched every 10 seconds according to the communication environment.

D. Variations:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

D-1. In each of the embodiments described above, a region including a human face is detected as a region of interest. However, another method, for example, an importance index MI indicating the importance of an image portion is calculated by a predetermined calculation method. You may make it detect as an area | region with high degree index MI. As a method for calculating the importance index MI, for example, a method described in Japanese Patent Application Laid-Open No. 2001-127070 can be employed. In the method described in this publication, “attraction degree” is adopted as the importance index MI. This degree of attraction is a parameter suitable for human subjectivity, and is based on the physical (characteristic color heterogeneity, texture heterogeneity, shape heterogeneity, area heterogeneity, etc.) at the image position of the target image MF. It is a value determined quantitatively. In this method, an image portion (or a rectangular region including the image portion) having an attractiveness degree equal to or greater than a threshold value is detected as the attention area PA.

D-2. In each of the above-described embodiments, the moving image data providing system is configured via the LAN. However, the Internet may be interposed. Further, as a communication protocol, the TCP / IP protocol may be used for both the wired LAN and the wireless LAN, and for example, BLUETOOTH (registered trademark) may be used only for the wireless LAN.

D-3. In each of the above-described embodiments, a personal computer or a projector is used as the moving image data reproducing device. However, the moving image data reproducing device is a device capable of displaying moving image data such as a PDA (Personal Digital Assistant), a mobile phone, or a liquid crystal viewer. I just need it. On the other hand, in each of the above-described embodiments, a server is used as a moving image transmission device, but any device capable of transmitting moving image data such as a PDA or a mobile phone may be used. As a communication mode, the present invention is applicable to various combinations such as mobile phones or mobile phones and PDAs.

D-4. In each of the embodiments described above, the original moving image data before encoding is encoded at a plurality of different compression rates. However, the encoded moving image data may be encoded again.

D-5. In each of the above-described embodiments, the object is divided into areas such as the attention area and the other areas, and the compression rate is changed for each divided object. May be used to change the compression rate for each extracted AV object. The present invention can be applied to encoding processes of various standards including MPEG-4.

  In the above embodiment, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  When some or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, and the like. An external storage device fixed to the computer is also included.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an explanatory diagram showing a configuration of a network system in a first embodiment of the present invention. Explanatory drawing which shows the structure of server SV in 1st Example of this invention. Explanatory drawing which shows the content of the hard disk 14 with which server SV is provided. The flowchart which shows the content of the moving image data provision process in 1st Example of this invention. The flowchart which shows the content of the moving image data preparation process in 1st Example of this invention. Explanatory drawing which shows the mode of the attention area detection process in 1st Example of this invention. The flowchart which shows the content of the moving image data transmission process in 1st Example of this invention. The flowchart which shows the content of the moving image data transmission process in 2nd Example of this invention.

Explanation of symbols

11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 14 ... Hard disk 15 ... I / F circuit 16 ... Wireless LAN access point 18 ... User interface part 30 ... Moving image database part 100 ... Projector 200, 210, 220 ... Personal computer SV ... Server

Claims (10)

A moving image transmitting apparatus for transmitting moving image data to a moving image data reproducing apparatus,
An object extraction unit that extracts a plurality of objects including a target object from moving image data;
A compression rate determining unit that receives playback environment information representing a playback environment of the moving image data playback device and determines a compression rate for each of the plurality of objects according to the playback environment information;
An encoding unit that encodes the moving image data while compressing the plurality of objects based on the determined compression rates;
A transmission unit for transmitting the encoded moving image data;
With
The attention object is an image area including an attention area that attracts relatively attention among image areas displayed in the moving image data,
The moving image transmitting apparatus according to claim 1, wherein a compression rate of the object of interest is set to be lower than a compression rate of another object among the plurality of objects. The moving image transmitting apparatus according to claim 1,
The moving image transmitting apparatus, wherein the target object includes a specific object that is an image region including a human face. The moving image transmitting apparatus according to claim 1, wherein:
The reproduction environment information includes decoding performance information representing decoding performance of the moving image data reproduction device,
The moving image transmitting apparatus, wherein the compression level determination unit determines the compression level according to the decoding performance information. The moving image transmitting apparatus according to any one of claims 1 to 3,
The moving image transmitting apparatus, wherein the encoding unit has an encoding processing mode for reducing an update rate of the other object. The moving image transmitting apparatus according to claim 4,
The moving image transmitting apparatus, wherein the encoding unit has a sprite encoding processing mode for performing sprite encoding on the other object. The moving image transmitting apparatus according to any one of claims 1 to 5,
The reproduction environment information includes communication environment information representing information on a communication rate between the moving image data reproducing device and the moving image transmission device,
The moving image transmitting apparatus, wherein the compression level determination unit changes the compression level according to the communication environment information. The moving image transmitting apparatus according to claim 6,
The moving image transmitting apparatus, wherein the encoding unit executes the encoding in real time at the changed compression level. A moving image data reproducing device that receives and reproduces encoded data that is moving image data encoded by a predetermined encoding from a moving image transmitting device,
A transmission unit for transmitting reproduction environment information representing the reproduction environment of the moving image data reproduction device;
An inverse encoding unit for inversely encoding the received encoded data;
A reproducing unit for reproducing the decoded encoded data;
With
The predetermined encoding is performed while compressing the plurality of objects based on each compression rate determined according to the reproduction environment information for each of the plurality of objects including the target object extracted from the moving image data. ,
The attention object is an image area including an attention area that attracts relatively attention among image areas displayed in the moving image data,
The moving image data reproducing apparatus, wherein a compression rate of the object of interest is set to be lower than a compression rate of other objects among the plurality of objects. A method of transmitting moving image data to a moving image data reproducing device,
(A) extracting a plurality of objects including the object of interest from the moving image data;
(B) receiving reproduction environment information representing the reproduction environment of the moving image data reproduction device, and determining a compression rate for each of the plurality of objects according to the reproduction environment information;
(C) encoding the moving image data while compressing the plurality of objects based on the determined compression rates;
(D) transmitting the encoded moving image data;
With
The attention object is an image area including an attention area that attracts relatively attention among image areas displayed in the moving image data,
The moving image transmission method according to claim 1, wherein a compression rate of the object of interest is set to be lower than a compression rate of another object of the plurality of objects. A computer program for causing a computer to execute processing for transmitting moving image data to a moving image data reproducing device,
(A) a function of extracting a plurality of objects including a target object from moving image data;
(B) a function of receiving reproduction environment information representing a reproduction environment of the moving image data reproduction device and determining a compression rate for each of the plurality of objects according to the reproduction environment information;
(C) a function of encoding the moving image data while compressing the plurality of objects based on the determined compression rates;
(D) a function of transmitting the encoded moving image data;
Including a program for causing the computer to realize
The attention object is an image area including an attention area that attracts relatively attention among image areas displayed in the moving image data,
The computer program according to claim 1, wherein a compression rate of the object of interest is set to be lower than a compression rate of another object of the plurality of objects.

Images