JP4305752B2 - VIDEO DISTRIBUTION SYSTEM, VIDEO DISTRIBUTION DEVICE, VIDEO DISTRIBUTION METHOD, AND VIDEO DISTRIBUTION PROGRAM - Google Patents

Description

  The present invention relates to a video distribution system, and is suitable for application to a case where video is encoded and distributed to a plurality of viewers.

  2. Description of the Related Art Conventionally, a system in which a video at a distant point is shared by a plurality of users and viewed via a network, a broadcast, or the like has become common. Examples thereof include a multipoint video conference system using ISDN (Integrated Services Digital Network) and the Internet, viewing by a large number of people in sports competitions, a surveillance camera system, and the like.

On the other hand, in a field that requires a large amount of calculation processing, so-called cluster processing, in which a plurality of computers are connected by a high-speed network and distributed processing is becoming common. By this cluster processing, processing equivalent to an expensive high-performance computer can be executed with a cheaper configuration. In general, a large amount of calculation is required for compression processing of high-quality data, and it is conceivable to apply cluster processing as means for realizing this at a low cost (for example, see Patent Document 1).
JP 2000-30047 JP

  Here, when a plurality of viewers view the same video at the same time, the regions of interest (interested portions) of each viewer in the video may be different. For example, if each viewer wants to watch a specific player more clearly in a sports broadcast, if he wants to see a specific conference participant's behavior more clearly in a video conference system, or in the surveillance image of a surveillance camera system For example, when a specific area is picked up and it is desired to monitor clearly.

  In order to view images clearly, it is generally necessary to increase the amount of data to be transmitted. However, in a system such as a network where the bandwidth is limited, it is not possible to increase the amount of transmitted data carelessly. There is a problem that it is not desirable.

  In addition, when assuming a large number of unspecified viewers, the image quality and calculation processing load may be in a trade-off relationship depending on the moving image compression method. At this time, the system load changes dynamically. However, there is a problem that it is wasteful from the viewpoint of cost and effective use of computer resources to build an excessive system specification in anticipation of such fluctuations in system load.

  It is conceivable to apply the above-described cluster processing for such purposes, however, in cluster processing, the calculation processing is managed for each computer, and a plurality of calculation processing is executed in parallel in most cases. is there. For this reason, when moving image processing is executed in real time using cluster processing, there is a problem that there is no guarantee that the processing will be completed within a predetermined aircraft time.

  The present invention has been made in consideration of the above points, and when distributing a common video to a plurality of viewers, a video distribution system capable of clearly distributing the gaze area of each viewer while suppressing the amount of transmission data. And a video distribution method.

In order to solve this problem, in the present invention, in a video distribution system including a video distribution device that distributes video and a video display device that displays the distributed video, the video distribution device receives a plurality of input video data. It is composed of video dividing means that generates a plurality of divided video data and a plurality of information processing devices, and generates a plurality of divided encoded data by encoding each of the plurality of divided video data. Encoding means for controlling, encoding control means for controlling the encoding means, combining means for combining a plurality of divided encoded data to generate encoded video data, and distributing the encoded video data to the video display device The video display device includes a receiving means for receiving the distributed encoded video data, a decoding display means for decoding the encoded video data and displaying the video, and a decoding And a region of interest detection means indicates means detects a fixation region of the viewer for the video to be displayed, the encoding control unit, code for dividing the video data of the video area watch area detection means corresponds to the gaze region detected Rutotomoni improve of quality, the processing load information of the encoding process is notified from each of the plurality of information processing apparatuses, in accordance with the information of the gaze region watch area detected by the detecting means, for each of a plurality of information processing apparatuses The encoding process of divided video data was assigned .

By selectively improving the encoding quality of the gaze area detected by the gaze area detection means,
It is possible to clearly distribute the image of the viewer's gaze area while suppressing the transmission data amount of the entire system.

In the present invention, the video data input by the video dividing means is divided corresponding to a plurality of video areas, a plurality of divided video data is generated, and the plurality of divided video data are encoded to obtain a plurality of divided codes. A plurality of information processing devices that generate encoded data are allocated and supplied with divided video data by the encoding control means, and a plurality of divided encoded data are combined by the combining means to generate encoded video data, The encoded video data is delivered to the video display device by the distribution means, and the viewer's gaze area for the video that is transmitted from the video display device by the gaze area information receiving means and decoded is displayed. Receives gaze area information and improves the encoding quality for the divided video data of the video area corresponding to the gaze area detected by the gaze area detection means. In addition, the code of the divided video data for each of the plurality of information processing devices according to the processing load information of the encoding process notified from each of the plurality of information processing devices and the information of the gaze region detected by the gaze region detection means Added the process .

  Thereby, it is possible to distribute only the viewer's gaze area with improved image quality, and it is possible to clearly distribute the image of the area of interest of each user while further reducing the amount of transmission data of the entire system.

According to the present invention as described above, achieved while suppressing the amount of data transferred across the system, clearly distribution can video distribution system image viewers stop region, the video distribution device, a video distribution method, and a video distribution program it can.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Overall Configuration of Moving Image Cluster Processing System FIG. 1 shows a moving image cluster processing system 1 according to the present invention, which includes a server PC (Personal Computer) 2, four encoder PCs 3A to 3D, and five user PCs 4A. To 4E are connected via the network 5.

  The moving image cluster processing system 1 as a video distribution system uses an information processing device and encoder PCs 3A to 3D as encoding means for processing an analog video signal S1 supplied to a server PC2 as a video distribution device. The JPEG (Joint Photographic Cording Experts Group) 2000 system performs real-time compression encoding processing and distributes it to the user PCs 4A to 4E as video display devices.

  In the server PC 2, a memory 12, a display unit 13, a hard disk drive 14, a network interface 15, and a video capture circuit 16 are connected to a CPU 11 that controls the server PC 2 through a bus 17.

  The hard disk drive 14 stores an operating system such as Windows (registered trademark) and various application programs such as a moving image cluster processing program. The CPU 11 reads the operating system from the hard disk drive 14 in response to the activation of the server PC 2, expands it in the memory 12, and executes it.

  Then, the CPU 11 executes the moving image cluster processing program under the operating system execution environment, and causes the encoders PC3A to 3D to distribute the moving image compression encoding processing.

  That is, the video capture circuit 16 of the server PC 2 converts the analog video signal S1 supplied from the outside into a digital signal, generates a digital video signal D1, and supplies this to the CPU 11. In accordance with the moving image cluster processing program, the CPU 11 causes the encoders PC3A to 3D to distributely compress and encode the digital video signal D1.

  Here, the JPEG2000 system performs compression encoding of a digital video signal for each frame (or field) as a picture, and divides each frame into a plurality of image areas, and compresses each image area individually. It is made to become possible.

  That is, as shown in FIG. 2, the CPU 11 as the video dividing means divides each frame of the digital video signal S1 composed of, for example, 720 × 480 pixels into four to generate four blocks composed of 360 × 240 pixels. Further, the CPU 11 divides each block into four to generate 16 areas each having 180 × 120 pixels. Block 1 includes areas 1 to 4, block 2 includes areas 5 to 8, block 3 includes areas 9 to 12, and block 4 includes areas 13 to 16.

  At this time, as an initial state, the CPU 11 assigns area 1 to area 4 to the encoder PC 3A, area 5 to area 8 to the encoder PC 3B, area 9 to area 12 to the encoder PC 3C, and area 13 to area 16 to the encoder PC 3D. Assign each one (this assignment is called an assignment). Then, the CPU 11 sequentially supplies the video data of the area allocated to the PC 3A to each encoder PC 3A to 3D as the divided digital video data D2A to D2D to the encoder PC 3A to 3D to the PC 3A.

  The area assignment for each of the encoders PC3A to 3D is appropriately changed by the CPU 11 serving as an encoding control unit according to the processing load of each encoder PC3 based on a region-of-interest adaptive encoding processing procedure described later.

  In the encoder PC3 (3A to 3D), a memory 22, a display unit 23, a hard disk drive 24, and a network interface 25 are connected via a bus 27 to a CPU 21 that performs overall control of the encoder PC3.

  The hard disk drive 24 stores various application programs such as an operating system and a cluster application program. The CPU 21 reads the operating system from the hard disk drive 24 in response to the activation of the encoder PC 3, expands it in the memory 22, and executes it.

  Then, the CPU 21 executes the moving image encoding program under the operating environment of the operating system, thereby compressing and encoding the divided digital video data D2 (D2A to D2D) supplied from the server PC2 by the JPEG2000 method. D3 (D3A to D3D) is generated and returned to the server PC2.

  The CPU 11 of the server PC2 serving as the synthesizing unit sequentially synthesizes the divided encoded data D3A to D3D returned from the encoders PC3A to 3D in units of frames, and encoded video data including a series of compressed and encoded frame data. Is generated. Then, the CPU 11 serving as a distribution unit distributes the combined encoded video data as encoded video data D4A to D4E to the user PCs 4A to 4E, respectively.

  In the user PC 4 (4 </ b> A to 4 </ b> E), a memory 32, a display unit 33, a hard disk drive 34, a network interface 35, and a line-of-sight detection unit 36 are connected via a bus 37 to the CPU 31 that performs overall control of the user PC 4. Yes.

  The hard disk drive 34 stores various application programs such as an operating system and a moving image display program. The CPU 31 reads the operating system from the hard disk drive 34 in response to the activation of the user PC 4, expands it in the memory 32, and executes it.

  The CPU 31 as the receiving means and the decoding display means executes the moving image display program under the operating system execution environment to decode and display the encoded video data D4 (D4A to D4E) supplied from the server PC2. Displayed on the unit 33.

  Thus, the moving image cluster processing system 1 performs real-time compression coding processing of the analog video signal S1 input to the server PC2 by cluster processing by the encoders PC3A to 3D, and distributes and displays it to the user PCs 4A to 4E. .

(2) Region-of-interest adaptive encoding according to the present invention In addition to this configuration, in the moving image cluster processing system 1, each user PC 4A to 4E is gazing at the user in the distribution video (that is, the viewer of the distribution video). Each region (region of interest) is detected, and the encoding quality of the region of interest is selectively increased and encoded, so that the amount of transmission data of the entire system is suppressed and an image of the region of interest is displayed to each user. It is designed to transmit clearly.

  As described above, the user PC 4 (4A to 4E) is provided with the line-of-sight detection unit 36 for detecting the user's line-of-sight direction. As shown in FIG. 3, the line-of-sight detection unit 36 includes two cameras 36A and 36B. The positional relationship between the cameras 36A and 36B and the display unit 33 is selected so that the face of the user who views the distribution video displayed on the display unit 33 can be photographed from the right side and the left side, respectively.

  The cameras 36A and 36B supply the right face image D6R and the left face image D6L of the photographed user to the CPU 31, respectively. The CPU 31 as the gaze area detecting means performs a template matching process for comparing the eye template image stored in advance with the right face image D6R and the left face image D6L in accordance with a predetermined line-of-sight detection program. The position of the user's eyes in the video D6R and the left face video D6L is detected.

Further, the CPU 31 calculates the orientation direction (vertical and horizontal angles) of the user's face based on the position of the user's eyes in the detected right facial image D6R and left facial image D6L. Then, the CPU 31 as the gaze area detection unit and the gaze area information transmission unit displays the user's face on the display unit 33 based on the calculated relationship between the orientation of the user's face and the installation position of the display unit 33. To which area of the distribution video is directed (that is, which area the user is gazing at), and the area number of the detected directional area (FIG. 2) is used as the attention area information D5 as the attention area information. (D5A to D5E) are transmitted to the server PC2.

  The CPU 11 of the user PC 4 performs the above-described user region-of-interest extraction process at predetermined intervals (for example, every 3 seconds).

  Here, in the JPEG 2000 system, image quality hierarchical coding (SNR scalability) can be performed in compression coding processing of a digital video signal. By this hierarchical encoding, decoding by partial extraction of encoded data is possible.

  FIG. 4 shows an example of SNR scalability, where the encoding level is set from level 1 with the highest image quality to level 4 with the lowest image quality. When the video is encoded at level 1, the encoded data includes data from level 1 to level 4. When decoding is performed using all of these encoded data, a decoded video of level 1 with the highest image quality can be obtained. Similarly, when decoding is performed using the encoded data of level 2 to level 4, the decoded video of level 2 is performed, and when decoding is performed using the encoded data of level 3 and level 4, the decoded video of level 3 is performed. If decoding is performed using only encoded data of level 4, a decoded video of level 4 with the lowest image quality can be obtained.

  When the video is encoded at level 2, the encoded data includes encoded data from level 2 to level 4, and when encoded at level 3, the encoded data of level 3 and level 4 are When encoding is performed at level 4, only encoded data at level 4 is included.

  Here, the higher the encoding level (that is, the higher the encoding bit rate), the higher the encoding quality and the higher the image quality of the decoded image, but this increases the processing time of the encoding process (in other words, in other words) The processing load increases).

  That is, as shown in FIG. 5, the encoding bit rate of level 1 with the highest image quality is 2.0 [bpp] (Bit Per Pixel), and standard encoding indicating the standard encoding processing time in level 1 is shown. The time is 48 [msec] per block. Similarly, the encoding bit rate of level 2 is 1.0 [bpp], and the standard encoding time in this case is 20 [msec]. The level 3 encoding bit rate is 0.5 [bpp], and the standard encoding time in this case is 28 [msec]. Further, the encoding bit rate of the lowest level 4 of the image is 0.25 [bpp], and the standard encoding time in this case is 20 [msec]. Note that the standard coding time per area is about ¼ of the standard coding time per block.

  At the start of cluster encoding, the CPU 11 sets the encoding level of each of the tiles 1 to 16 to the level selected in the cluster encoding preparation processing procedure described later. Then, the CPU 11 instructs each of the encoders PC3A to 3D to encode the areas 1 to 16 assigned thereto at the set encoding level.

  When the CPU 11 serving as the gaze area information receiving unit and the encoding control unit receives the region-of-interest information D5 (D5A to D5E) from each user PC 4 (4A to 4E), the CPU 11 based on the region-of-interest adaptive encoding processing procedure described later. The encoders PC3A to 3D are instructed to increase the encoding level of the area of interest indicated by the received region-of-interest information D5A to D5E (that is, to increase the encoding quality).

  At this time, the CPU 11 appropriately changes the area assignments for the encoders PC3A to 3D based on the processing load information returned from the encoders PC3A to 3D, and averages the processing loads of the encoders PC3A to 3D. The encoding processing delay is prevented.

  Further, when the CPU 11 synthesizes the divided encoded data D3A to D3D returned from the encoders PC3A to 3D and distributes them to the user PCs 4A to 4E, it is based on the region-of-interest information D5A to D5E of each user PC 4A to 4E. The level for each area of the encoded data D4A to D4D distributed to each of the user PCs 4A to 4E is individually adjusted.

  That is, all levels of encoded data are transmitted for the areas of interest of the respective user PCs 4A to 4E, while the initial value level 3 is used for areas of interest of other users PC4 and areas other than the areas of interest. Only data up to is transmitted. For this reason, only the video of the region of interest can be distributed to each of the user PCs 4A to 4E with high image quality, thereby suppressing the amount of transmission data of the entire system and the image of the region of interest to each user. Can be transmitted clearly.

(3) Processing procedure of each PC Next, the processing procedure of each of the server PC2, the encoder PC3, and the user PC4 for performing the above-described region-of-interest adaptive encoding according to the present invention will be described in detail with reference to flowcharts.

  First, the processing procedure of the server PC 2 will be described. Prior to the actual cluster encoding process, the CPU 11 of the server PC 2 determines the encoding processing capabilities of the encoders PC3A to 3D. That is, the CPU 11 of the server PC 2 enters from the start step of the cluster coding preparation processing routine RT1 shown in FIG. 6 and proceeds to the next step SP1. In step SP1, the CPU 11 reads a plurality of preset encoding levels from the hard disk drive 14, and proceeds to the next step SP2.

  In step SP2, the CPU 11 selects the highest encoding level among the plurality of encoding levels, and proceeds to the next step SP3.

  In step SP3, the CPU 11 sends to each encoder PC3A to 3D a test encoding command for executing encoding for a predetermined test image read from the hard disk drive 14 a plurality of times at the selected encoding level. Move to SP4.

  In step SP4, the CPU 11 measures the test encoding processing time of each of the encoders PC3A to 3D, stores this as the encoding processing capability value of each of the encoders PC3A to 3D, and proceeds to the next step SP5.

  In step SP5, the CPU 11 calculates a predicted process time required for the encoding process for one frame when each encoder PC3A to 3D executes the actual encoding process based on the encoding processing capability value. The process proceeds to step SP6.

  In step SP6, the CPU 11 determines whether the calculated predicted process time is within one frame period (that is, 33 [msec]). If the predicted processing time is within one frame period in step SP6, the CPU 11 moves to step SP10 and ends the cluster encoding preparation processing procedure without changing the default encoding level.

  On the other hand, if the predicted processing time exceeds one frame period in step SP6, this means that the encoding of one frame is not expected to end within one frame period with the current encoding level setting. At this time, the CPU 11 proceeds to step SP7.

  In step SP7, the CPU 11 calculates the excess rate of the predicted processing time for one frame period, selects a lower encoding level corresponding to the excess rate, and moves to the next step SP8.

  In step SP8, the CPU 11 determines whether or not the selected encoding level exists. In step SP8, if the selected encoding level exists, the CPU 11 returns to step SP3, and executes steps SP3 to SP10 again with the newly selected encoding level.

  On the other hand, if the selected encoding level does not exist in step SP8, this means that the encoding level cannot be lowered any more. At this time, the CPU 11 moves to step SP9, and each encoding is performed. By updating the encoding bit rate set for each level by a predetermined value, the processing load at each encoding level is reduced.

  Then, the CPU 11 returns to step SP2, and executes steps SP2 to SP10 again based on the updated encoding level.

  As described above, the CPU 11 of the server PC sets the encoding level corresponding to the processing capability of each of the encoders PC3A to 3D, and then starts cluster encoding processing of the distribution video.

  That is, the CPU 11 of the server PC 2 enters from the start step of the cluster encoding processing procedure routine RT2 shown in FIG. 7 and proceeds to the next step SP15. In step SP15, the CPU 11 sets an encoding level as an initial value for each of the encoders PC3A to 3D. At this time, in order to leave room for improvement in the coding level for the region of interest, the CPU 11 sets the highest coding level that is equal to or lower than the highest level and completes processing within one frame period as an initial value.

  In the next step SP16, the CPU 11 assigns areas in the initial state to the encoders PC3A to 3D. That is, area 1 to area 4 are assigned to encoder PC3A, area 5 to area 8 are assigned to encoder PC3B, area 9 to area 12 are assigned to encoder PC3C, and area 13 to area 16 are assigned to encoder PC3D.

  In the next step SP17, the CPU 11 starts supplying the divided digital video data D2A to D2D corresponding to the assignment to the encoders PC3A to 3D, and performs encoding at the encoding level set for the encoders PC3A to 3D. The start of the conversion process is instructed, and the process proceeds to the next step SP18.

  In this state, each of the encoders PC3A to 3D encodes the divided digital video data D2A to D2D, returns the divided encoded data D3A to D3D to the server PC2, and notifies the server PC2 of each processing load information.

  In step SP18, the CPU 11 synthesizes the divided encoded data D3A to D3D returned from the encoders PC3A to 3D and distributes them as encoded video data D4A to D4E to the user PCs 4A to 4E.

  In this state, each of the user PCs 4A to 4E decodes the encoded video data D4A to D4E and displays the decoded video data D4A to D4E on the display unit 33 (FIG. 1). The server PC 11 is notified as D5A to D5E.

  In the next step SP19, the CPU 11 determines whether or not the region-of-interest information D5A to D5E is notified from each user PC 4A to 4E. If the region-of-interest information D5A to D5E is not notified in step SP19, the CPU 11 moves to step SP25, and the encoding process is completed within one frame period based on the processing load information notified from each encoder PC3A to 3D. Then, after assigning areas to the encoders PC3A to 3D, the process proceeds to step SP26.

  On the other hand, when the region-of-interest information D5A to D5E is notified in step SP19, the CPU 11 proceeds to step SP20.

  In step SP20, the CPU 11 raises the encoding level for the region of interest indicated by the region-of-interest information D5A to D5E, and proceeds to the next step SP21.

  In step SP21, the CPU 11 determines that the encoding process is completed within one frame period based on the encoding level changed in step SP20 and the processing load information notified from each encoder PC3A to 3D. The assignment of the area to ˜3D is started, and the process proceeds to the next step SP22.

  In step SP22, the CPU 11 determines whether or not there is an area that cannot be assigned and whose processing time exceeds one frame period in the assignment process performed in step SP21. If there is no unassignable area in step SP22, the CPU 11 proceeds to step SP26.

  On the other hand, if there is an unassignable area in step SP22, the CPU 11 moves to step SP23, lowers the encoding level of the region of interest by one, and moves to the next step SP24.

  In step SP24, the CPU 11 compares whether or not the encoding level of the region of interest is higher than the encoding levels of other regions. If it is determined in step SP24 that the encoding level of the region of interest is not higher than the encoding level of the other region, the CPU 11 returns to step SP21 so that the encoding level of the region of interest is higher than the other region. Assign again.

  On the other hand, when it is determined in step SP24 that the encoding level of the region of interest is higher than the encoding levels of other regions, the CPU 11 proceeds to step SP26.

  In this way, the CPU 11 sets the encoding level of the region of interest higher than the other regions based on the region-of-interest information D5A to D5E notified from each user PC 4A to 4E, and the changed encoding level and The area assignment for the encoders PC3A to 3D is changed according to the processing load information.

  In step SP26, the CPU 11 starts to supply the divided digital video data D2A to D2D corresponding to the new assignment to the encoders PC3A to 3D, and starts the encoding process of the next frame at the changed encoding level. Instructed and proceeds to step SP27.

  In step SP27, the CPU 11 synthesizes the divided encoded data D3A to D3D returned from the encoders PC3A to 3D, distributes them to the user PCs 4A to 4E as encoded video data D4A to D4E, and returns to step SP19.

  At this time, the CPU 11 transmits all levels of encoded data for each user PC 4A to 4E based on the region-of-interest information D5A to D5E. For other areas, only encoded data up to the initial level 3 is transmitted.

  In this way, the CPU 11 delivers only the video of the region of interest with high image quality to each of the user PCs 4A to 4E.

  Next, the encoding process procedure in encoder PC3 (3A-3D) is demonstrated. The CPU 21 of the encoder PC3 enters from the start step of the encoding process routine RT3 shown in FIG. 8 and proceeds to step SP31. In step SP31, the CPU 21 waits for the supply of the divided digital video data D2 (D2A to D2D) supplied from the server PC 2. When the divided digital video data D2 is received, the CPU 21 proceeds to the next step SP32.

  In step SP32, the CPU 21 encodes each area of the received divided digital video data D2 at a specified encoding level to generate divided encoded data D3 (D3A to D3D), and proceeds to the next step SP33.

  In step SP33, the CPU 11 returns the divided encoded data D3 to the server PC2, notifies the server PC2 of the processing load information at this time, and returns to step SP31.

  In this way, the CPU 21 of the encoder PC3 sequentially encodes the divided digital video data D2 supplied from the server PC2 and returns the divided encoded data D3, and notifies the server PC2 of the processing load information of the encoding process. Go.

  Next, a display processing procedure in the user PC 4 (4A to 4E) will be described. The CPU 31 of the user PC 4 enters from the start step of the display processing procedure routine RT4 shown in FIG. 9 and proceeds to step SP41. In step SP41, the CPU 31 waits for the supply of the encoded video data D4 (D4A to D4E) supplied from the server PC 2. When the encoded video data D4 is received, the CPU 31 proceeds to the next step SP42.

  In step SP42, the CPU 31 decodes the received encoded video data D4 and displays it on the display unit 33, and proceeds to the next step SP43.

  In step SP43, the CPU 31 determines whether or not the user's region of interest extraction timing has come. If it is determined in step SP43 that the region of interest extraction timing has not arrived, the CPU 31 returns to step SP41, and receives, decodes, and displays the encoded video data D4 again.

  On the other hand, if it is determined in step SP43 that the region of interest extraction timing has arrived, the CPU 31 proceeds to step SP44.

  In step SP44, the CPU 31 captures the user image via the line-of-sight detection unit 36, extracts a region of interest in the display image from the captured user image, and transmits the region of interest information D5 (D5A to D5E) to the server PC2. Then, the process returns to step SP41.

  In this way, the CPU 31 of the user PC 4 sequentially decodes and displays the encoded video data D4 distributed from the server PC 2, extracts the user's region of interest at every predetermined timing, and notifies the server PC 2 of it.

(4) Example of coding level change according to region of interest transition Next, an example of the coding level change according to region of interest adaptive coding described above will be described with reference to FIG. . In FIG. 10, each user PC4A-4E extracts a region of interest every 3 seconds, and the assignment to each encoder PC3A-3D changes according to the extracted region of interest.

  That is, in the period T1 immediately after the start of processing, as an initial state, the areas 1 to 4 are the encoder PC3A, the areas 5 to 8 are the encoder PC3B, the areas 9 to 12 are the encoder PC3C, and the areas 13 to 16 are Assigned to each encoder PC3D. At this time, the encoding levels of the areas 1 to 16 are all the initial level 3, and the processing load level of each encoder PC3 is “1”.

  Then, for the video displayed during this period T1, user PC 4A uses area 12, user PC 4B uses area 4, user PC 4C uses area 2, user PC 4D uses area 5, and user PC 4E uses area 13. Extracted. The server PC2 reflects the extraction result of the region of interest and changes the encoding level and assignment in the next period T2.

  That is, in this period T2, the server PC 2 sets the areas 2, 4, 5, 12, and 13 that are the regions of interest to the encoding level 2.

  The encoding level of the display video of each user PC 4A-4E in this period T2 is shown in FIG.11 and FIG.12. In each of the user PCs 4 </ b> A to 4 </ b> E, only the individual region of interest has a high encoding level, and the region other than the region of interest has the encoding level 3.

  In addition, the encoding level 1 is set for the areas 3 and 4 in the period T4. In the initial state, both the area 3 and the area 4 are assigned to the user PC 4A, and the processing load on the user PC 4A becomes excessive as it is.

  For this reason, the server PC 2 assigns the area 4 to the user PC 4B in the period T4, and instead assigns the area 5 to the user PC 4A to distribute the encoding load.

(5) Operation and Effect In the configuration described above, the server PC2 of the moving image cluster processing system 1 divides the image data for one frame of the digital video signal D1 into 16 areas and supplies them to the four encoders PC3A to 3D. Appropriately assigned divided digital video data D2A to D2D are generated and supplied to the corresponding encoders PC3A to 3D, respectively.

  The encoders PC3A to 3D respectively encode the divided digital video data D2A to D2D to generate divided encoded data D3A to D3D, and transmit them to the server PC2 together with the processing load information at this time. The server PC2 synthesizes the divided encoded data D3A to D3D to generate encoded video data D4A to D4E consisting of data for one frame, and distributes it to the corresponding user PCs 4A to 4E.

  Each of the user PCs 4A to 4E decodes and displays the encoded video data D4A to D4E, respectively. At this time, each of the user PCs 4A to 4E detects a region of interest in which the user is gazing in the displayed distribution video, and notifies the server 2 of this as region-of-interest information D5A to D5E.

  The server 2 increases the encoding level of the region of interest based on the region-of-interest information D5A to D5E, and refers to the processing load information so that the encoding process of each encoder PC3A to 3D is completed within one frame period. The area assignments for the encoders PC3A to 3D are appropriately changed.

  Further, when the server 2 distributes the encoded video data D4A to D4E to the user PCs 4A to 4E, the encoded data of all the levels for each user PC 4A to 4E based on the region of interest information D5A to D5E. On the other hand, only the encoded data up to the initial level 3 is transmitted for the area of interest of the other user PC 4 and the area other than the area of interest.

  According to the above configuration, each user PC 4A to 4E detects the region of interest that the user is watching in the distribution video, and performs encoding by increasing the encoding quality of the region of interest, so that each user can watch. Thus, it is possible to selectively improve only the image quality of a certain area, and thereby to clearly transmit the image of the area of interest of each user while suppressing the transmission data amount of the entire system.

  In addition, when the encoded video data D4A to D4E are distributed to the user PCs 4A to 4E, the encoded data of all levels is transmitted only for the individual regions of interest of the respective user PCs 4A to 4E. Can be transmitted with standard image quality, and the image of the area of interest of each user can be transmitted clearly while further reducing the amount of transmission data of the entire system.

  In addition, by appropriately changing the area assignment for each encoder PC3A-3D based on the region-of-interest information D5A-D5E and the processing load information, the processing capability of each encoder PC3A-3D can be effectively utilized, and a moving image cluster It is possible to compress and encode a moving image with as high image quality as possible while ensuring real-time processing.

(6) Other Embodiments In the above-described embodiment, the case where the moving image cluster processing system 1 executes the moving image compression / decompression processing according to the JPEG 2000 method has been described. However, the present invention is not limited to this, and for example, Various encoding methods such as MPEG (Moving Picture Experts Group) 2 method can be applied to the present invention.

  In the above-described embodiment, the CPUs 11, 21, and 31 of the server PC 2, the encoder PC 3, and the user PC 4 have the moving image cluster processing program and the moving image encoding stored in the hard disk drives 14, 24, and 34, respectively. The processing program and the moving image display program are executed. However, the present invention is not limited to this, and the above-described processing may be executed by installing a program storage medium storing these programs. Good.

  In this case, the program storage medium for installing the program in the server PC 2, the encoder PC 3 and the user PC 4 is not only a package medium such as a CD-ROM (Compact Disk-Read Only Memory) or a DVD (Digital Versatile Disk). The program may be realized by a semiconductor memory or a magnetic disk in which the program is stored temporarily or permanently. Further, as means for storing the programs in these program storage media, wired and wireless communication media such as a local area network, the Internet, and digital satellite broadcasting may be used.

  In the above-described embodiment, as a method of extracting the user's region of interest, the user's face is photographed with two cameras, and the template matching process is performed on the right face image and the left face image. However, the present invention is not limited to this, and for example, the head is mounted by a headset worn by the user. The region of interest may be extracted by various methods, for example, by detecting the directivity direction and extracting the region of interest based on the detected directivity direction.

  The present invention can be applied when moving image processing is performed using a plurality of personal computers, workstations, personal computers, and the like.

It is a block diagram which shows the whole structure of a moving image cluster processing system. It is a basic diagram which shows the division | segmentation state of a flame | frame. It is a block diagram which shows the structure of a gaze detection part. It is a basic diagram with which it uses for description of SNR scalability. It is a table | surface which shows the relationship between an encoding level and a standard encoding process time. It is a flowchart which shows the cluster encoding preparation process procedure of server PC. It is a flowchart which shows the cluster encoding process sequence of server PC. It is a flowchart which shows the encoding process sequence of encoder PC. It is a flowchart which shows the display processing procedure of user PC. It is a basic diagram with which it uses for description of a region of interest and an encoding encoding level. It is a basic diagram which shows the encoding level with respect to each user PC. It is a basic diagram which shows the encoding level with respect to each user PC.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Moving image cluster processing system, 2 ... Server PC, 3A-3D ... Encoder PC, 4A-4E ... User PC, 5 ... Network, 11, 21, 31 ... CPU, 12, 22, 32 ... Memory, 13, 23, 33 ... Display section, 14, 24, 34 ... Hard disk drive, 15, 25, 35 ... Network interface, 16 ... Video capture circuit, 17, 27, 37 ... Bus, 36: Line-of-sight detection unit.

Claims (10)

In a video distribution system comprising a video distribution device for distributing video and a video display device for displaying the distributed video,
The video distribution device
Video dividing means for dividing the input video data corresponding to a plurality of video areas and generating a plurality of divided video data;
An encoding unit configured by a plurality of information processing devices, each encoding the plurality of divided video data to generate a plurality of divided encoded data;
Encoding control means for controlling the encoding means;
A synthesizing means for synthesizing a plurality of the divided encoded data to generate encoded video data;
Distribution means for distributing the encoded video data to the video display device;
The video display device
Receiving means for receiving the distributed encoded video data;
Decoding display means for decoding the encoded video data and displaying the video;
And a fixation region detecting means for detecting a fixation region of the viewer with respect to the image which the decoding display means displays,
The encoding control means is notified from the region of interest detector to improve the coding quality for said divided image data in the image area corresponding to the gaze area detected is Rutotomoni, each of the plurality of information processing apparatuses A video distribution system for allocating encoding processing of the divided video data to each of the plurality of information processing devices according to processing load information of encoding processing and information on the gaze region detected by the gaze region detecting unit . Video dividing means for dividing the input video data corresponding to a plurality of video areas and generating a plurality of divided video data;
A code control unit that allocates and supplies the plurality of divided video data to a plurality of information processing apparatuses that respectively encode the plurality of divided video data to generate a plurality of divided encoded data ;
A synthesizing means for synthesizing a plurality of the divided encoded data to generate encoded video data;
Distribution means for distributing the encoded video data to a video display device;
Gaze area information receiving means for receiving gaze area information indicating a viewer's gaze area for the video that is transmitted from the video display device and decoded and displayed from the distributed encoded video data , and
The encoding control means includes
The processing load information of the encoding process notified from each of the plurality of information processing devices, while improving the encoding quality of the divided video data of the video area corresponding to the gaze area indicated by the received gaze area information, A video distribution device that assigns the encoding processing of the divided video data to each of the plurality of information processing devices in accordance with the information of the gaze region detected by the gaze region detection means . The video distribution device according to claim 2, wherein the encoding control means improves the encoding quality for the divided video data of the video region corresponding to the plurality of gaze regions detected by the plurality of video display devices. The distribution means distributes only the divided video data corresponding to the gaze area of each video display device to each of the plurality of video display devices with the improved encoding quality, and displays the other video displays. The video distribution device according to claim 3, wherein the divided video data corresponding to the gaze area of the device is distributed with a standard encoding quality. The encoding control means includes
The video distribution apparatus according to claim 2, wherein encoding processing of the divided video data is assigned to each of the plurality of information processing apparatuses so as to effectively utilize the processing capabilities of the plurality of information processing apparatuses. The encoding control means includes
When the encoding quality of the divided video data of the video area corresponding to the gaze area is set to the highest level and it is determined that all the divided video data cannot be allocated based on the processing load information, the gaze 6. The video distribution apparatus according to claim 5, wherein the level of encoding quality of the divided video data in the video area corresponding to the area is lowered by one . The encoding control means includes
If the level of encoding quality of the divided video data of the video area corresponding to the gaze area is lower than that of the divided video data not corresponding to the gaze area, all the divided video data based on the processing load information The video distribution apparatus according to claim 6, wherein the allocation is performed again . The delivery means is
Delivering the encoded video data to a plurality of the video display devices;
The encoding control means includes
The plurality of information processing devices are controlled so as to assign the encoding processing of the divided video data to each of the plurality of information processing devices according to the information on the gaze area supplied from the plurality of the video display devices.
The video distribution apparatus according to claim 2 . A video dividing step for dividing the video data input by the video dividing means in correspondence with a plurality of video regions, and generating a plurality of divided video data;
An encoding control step of allocating and supplying the divided video data by an encoding control means to a plurality of information processing devices that respectively encode the plurality of divided video data to generate a plurality of divided encoded data;
A synthesizing step of synthesizing a plurality of the divided encoded data by a synthesizing unit to generate encoded video data;
A distribution step of distributing the encoded video data to a video display device by a distribution means ;
A gaze area information receiving step for receiving gaze area information indicating a gaze area of the viewer with respect to the video displayed by decoding the distributed encoded video data transmitted from the video display device by the gaze area information receiving means ;
Have
In the encoding control step,
Rutotomoni improve coding quality for said divided image data in the image area corresponding to the region of interest where the watch area detected by the detecting means, processing load information of the encoding process is notified from each of the plurality of information processing apparatuses And a video distribution method for allocating encoding processing of the divided video data to each of the plurality of information processing devices according to the gaze area information detected by the gaze area detection unit . Against the computer
A video dividing step for dividing the video data input by the video dividing means in correspondence with a plurality of video regions, and generating a plurality of divided video data;
An encoding control step of allocating and supplying the divided video data by an encoding control means to a plurality of information processing devices that respectively encode the plurality of divided video data to generate a plurality of divided encoded data;
A synthesizing step of synthesizing a plurality of the divided encoded data by a synthesizing unit to generate encoded video data;
A distribution step of distributing the encoded video data to a video display device by a distribution means ;
A gaze area information receiving step for receiving gaze area information indicating a gaze area of the viewer with respect to the video displayed by decoding the distributed encoded video data transmitted from the video display device by the gaze area information receiving means ;
And execute
In the encoding control step,
Rutotomoni improve coding quality for said divided image data in the image area corresponding to the region of interest where the watch area detected by the detecting means, processing load information of the encoding process is notified from each of the plurality of information processing apparatuses And a video distribution program for allocating encoding processing of the divided video data to each of the plurality of information processing devices according to the gaze area information detected by the gaze area detection unit .

Images