JP4795141B2 - Video coding / synthesizing apparatus, video coding / synthesizing method, and video transmission system - Google Patents

Description

  The present invention relates to a video encoding / synthesizing device, a video encoding / synthesizing method, and a video transmission system that encode an input video and synthesize a plurality of encoded video data.

  FIG. 21 is a diagram showing a configuration example of a conventional video coding / synthesizing apparatus. The video encoding / synthesizing device includes a multipoint control device 1701 and a plurality of user terminals 1702 to 1706. The user terminals 1702 to 1706 each have a video input unit and a video display unit. FIG. 22 is a diagram showing a screen of the video display unit of the user terminal. On the screen 1801 of the video display unit of the user terminal 1702, videos (user 2, user 3, user 4, user 5) respectively input to the video input units of the user terminals 1703 to 1706 are displayed.

  FIG. 23 is a diagram showing the configuration of the video input unit of the user terminal 1702. The configuration of the video input unit of the user terminals 1703 to 1706 is the same. The user terminal 1702 outputs the encoded video data to the multipoint control device 1701 in a format that can be combined, and includes an input processing unit 1901, a frame memory 1902, an encoding unit 1903, a buffer 1904, and input frame control. A unit 1905, an encoding control unit 1906, a data amount monitoring unit 1907, and a transmission unit 1908.

  The input processing unit 1901 converts a video from a camera (not shown) into a digital video signal, and further reduces it to ¼ of the whole. The frame memory 1902 stores the video data processed by the input processing unit 1901. The encoding unit 1903 encodes the video data stored in the frame memory 1902 with one row of macroblock lines as one video packet. The buffer 1904 stores the video data output from the encoding unit 1903. The data amount monitoring unit 1907 monitors the data accumulated in the buffer 1904, and causes the transmission unit 1908 to perform transmission if it is below a certain threshold. The input frame control unit 1905 selects a frame to be encoded under the control of the data amount monitoring unit 1907. The encoding control unit 1906 performs encoding termination processing in accordance with the control of the data amount monitoring unit 1907.

  On the other hand, the multipoint control device 1701 receives video data encoded from a user terminal 1702 with one column of macroblock lines as one video packet, and configures video data (see Patent Document 1). FIG. 24 is a diagram showing a video data composition operation. FIG. 24A shows video data encoded using one block of macroblock lines as one video packet 2001. FIG. 24B shows a composite screen 2002 composed of a plurality of received video data.

Japanese Patent Laid-Open No. 2005-20466

  The conventional video encoding / synthesizing apparatus has the following problems. In other words, when a multipoint control device is interposed when the video input at the user terminal is encoded after being reduced and transmitted, only the reduced video can be displayed at the user terminal on the receiving side, and the original resolution video can be displayed. Could not be displayed. In addition, when the amount of codes generated at each user terminal is not taken into consideration and the variation in the video characteristics input to the user terminals is large, the image quality is uneven among the user terminals.

  Also, in the conventional example, by combining with the image recognition function and reflecting the output result of this image recognition in the code amount control, it is possible to encode, for example, a human face area with high image quality. It is. However, in this case, since the video is encoded after the image recognition process is completed, the delay time until the video is encoded increases.

  The present invention has been made in view of the above circumstances. A video coding / synthesizing device, a video coding / synthesizing method, and video transmission capable of easily displaying a video having an original resolution in addition to a reduced video. The purpose is to provide a system.

  The present invention also provides a video coding / synthesizing device, a video coding / synthesizing method, and a video transmission system that can make the image quality uniform among the divided videos even when the variation in the inputted video characteristics is large. The purpose is to do.

  Another object of the present invention is to provide a video coding / synthesizing device, a video coding / synthesizing method, and a video transmission system capable of reducing a delay time until a video is coded.

The video coding / synthesizing device of the present invention includes a preprocessing unit that generates a slice video from an input video, a coding unit that slice-codes the slice video, and a plurality of slice-coded video data. A synthesizing unit that synthesizes the multi-screen display ; a decoding unit that decodes the plurality of video data synthesized to be the multi-screen display or the slice-encoded video data; A restoration unit that restores the video data to the original video data so as to be displayed on a single screen, and a video data that has been synthesized and decoded so as to be displayed on the multi-screen, or is restored so as to be displayed on the single screen. a selection unit for selecting a video data, and a display unit for displaying an image on a screen by the selected video data, a video coding synthesizer having a specific included in the video An object detection unit for detecting an object, and when the detection of the specific object has not been completed by the object detection unit, the encoding unit detects a past detection result for the specific object; Based on the above, the coding characteristics are changed .
This makes it possible to easily display a video having the original resolution in addition to the reduced video.

Further, the present invention is the video encoding / synthesizing apparatus described above, further comprising an object detection unit that detects a specific object included in the video, wherein the encoding unit detects the specific object. If not completed, the coding characteristic is changed based on the past detection result for the specific object .
Accordingly, it is possible to encode the region of the object with high image quality from the past information without waiting for the end of the detection of the object .

Further, the present invention is the video coding / synthesizing device described above, wherein the coding unit omits coding of the divided video when the specific object is not detected.
Thereby, since the video data in which the object is not detected is not encoded, it is possible to reduce the amount of data transmitted to the network .

Further, the present invention is the video encoding / synthesizing apparatus described above, further comprising an object detection unit that detects a specific object included in the video, and the encoding unit detects the specific object. In such a case, the encoding characteristic is changed.
As a result, it is possible to reduce the delay time until the video is encoded.

Also, the present invention is the video coding / synthesizing device described above, wherein the coding unit cuts out a corresponding area of the detected specific target object from the video and codes it.
As a result, only the detected object region can be encoded with high image quality based on high-resolution video data.

Further, the present invention is the video coding / synthesizing device described above, wherein the coding unit synthesizes the slice-coded video data so as to be a multi-screen display, and the code amount is set to a target code amount. It shall be encoded as follows.
As a result, the code amount is not wasted and the average image quality can be improved. Therefore, even when the variation in input video characteristics is large, the image quality can be made uniform between slice videos .

The video encoding / synthesizing method of the present invention includes a pre-processing step of generating a slice video from an input video, an encoding step of slice-coding the divided slice video, and the plurality of slice-coded videos A synthesis step for synthesizing data so as to be a multi-screen display; a decoding step for decoding a plurality of video data synthesized for the multi-screen display or the slice-encoded video data; The restoration step of restoring the decoded video data to the original video data so as to be displayed on one screen, and the video data synthesized and decoded so as to be the multi-screen display or the one-screen display. A selection step for selecting the restored video data, and a display step for displaying the video based on the selected video data on the screen. A than, further comprising, when said by the object detection step, the detection of a specific object is not finished, the encoding step the object detecting step of detecting a specific object included in the video Is for changing the coding characteristics based on the past detection results for the specific object .

In the video transmission system of the present invention, a video transmission device, a video synthesis device, and a video reception device are connected via a network, encodes the input video, and synthesizes and displays the plurality of encoded video data. In the video transmission system, the video transmission device includes a pre-processing unit that generates a slice video from the input video, a coding unit that slice-codes the slice video, and the slice-coded video data A first transmission unit that transmits the video data to the network, and the video synthesis device is input from a different video transmission device than the first reception unit that receives the slice-encoded video data from the network. A combination unit configured to combine the plurality of slice-coded video data so as to be displayed on a multi-screen, and the combined video data to the network. And a second transmission unit for transmitting to the workpiece, the video receiving device, first receives a plurality of video data or the slice encoded video data synthesized such that the multi-screen display from the network 2 receiving units, a decoding unit that decodes the plurality of video data synthesized so as to be displayed on the multi-screen or the slice-encoded video data, and the decoded video data on one screen A restoration unit that restores the original video data so as to be displayed, and a video data that is synthesized and decoded so as to be the multi-screen display or the video data restored so as to be the single-screen display. comprising: a selecting unit, and a display unit for displaying an image on a screen by the selected video data, the video transmission apparatus, object detection for detecting a specific object included in the video And when the detection of the specific object is not completed by the object detection unit, the encoding unit obtains an encoding characteristic based on a past detection result for the specific object. To change .

  According to the present invention, it is possible to provide a video coding / synthesizing device, a video coding / synthesizing method, and a video transmission system that can easily display a video having an original resolution in addition to a reduced video. Further, it is possible to provide a video coding / synthesizing device, a video coding / synthesizing method, and a video transmission system that can make the image quality uniform among the divided videos even when the variation in inputted video characteristics is large. Further, it is possible to provide a video coding / synthesizing device, a video coding / synthesizing method, and a video transmission system capable of reducing a delay time until a video is coded.

  The video coding / synthesizing apparatus according to the present embodiment is applied to, for example, a video transmission system that transmits a captured image of a surveillance camera. This type of video transmission system has a function of encoding video input from a plurality of cameras or the like, and synthesizing and displaying the encoded video data on one screen. An example of the configuration and operation of the video coding / synthesizing apparatus and video transmission system according to the present embodiment will be described below.

(First embodiment)
FIG. 1 is a diagram showing a configuration of a video transmission system according to the first embodiment of the present invention. This video transmission system has a configuration in which a plurality of video transmission devices 101 a to 101 d, a video composition device 102, and a plurality of video reception devices 103 a to 103 d are connected via a first network 104 and a second network 105. Here, the video transmission apparatuses 101a to 101d are collectively referred to as the video transmission apparatus 101. Similarly, the video receiving devices 103 a to 103 d are collectively referred to as the video receiving device 103.

  The video transmission apparatus 101 receives video data from a camera (not shown), encodes the input video data, and transmits the encoded video data to the first network 104. The video composition device 102 receives the video data encoded by each of the plurality of video transmission devices 101 from the first network 104, and synthesizes these video data so as to be displayed on the multi-screen. The data is transmitted to the network 105. The video receiving device 103 receives the encoded video data from the second network 105, decodes the received video data, and displays it. The first network 104 includes a bus in a device such as a LAN or a PCI, and connects the video transmission device 101 and the video composition device 102. The second network 105 includes a bus in a device such as a LAN or a PCI, and connects the video composition device 102 and the video reception device 103.

  The video transmission apparatus 101 includes a video input unit 107, a video preprocessing unit 108, a video encoding unit 109, a transmission / reception unit 110, and an encoding control unit 106. The video input unit 107 inputs video from the camera. The video pre-processing unit 108 has a function of a dividing unit, and performs slice processing on the video data captured by the video input unit 107 to generate slice video data. The video encoding unit 109 has the function of an encoding unit, and slice-slices the slice video data generated by the video preprocessing unit 108. The transmission / reception unit 110 transmits the slice-coded video data to the first network 104. The encoding control unit 106 controls generation of slice video data by the video preprocessing unit 108 and slice encoding by the video encoding unit 109.

  The video composition apparatus 102 includes a transmission / reception unit 114, a synthesis unit 113, a transmission / reception unit 112, and a composition control unit 111. The transmission / reception unit 114 receives the slice-coded video data from the first network 104. The synthesizer 113 synthesizes a plurality of slice encoded video data. The transmission / reception unit 112 transmits the slice-combined video data to the second network 105. The composition control unit 111 controls the composition by the composition unit 113.

  The video receiving apparatus 103 includes a transmission / reception unit 117 that receives the slice-combined video data from the second network 105, a video decoding unit 116 that decodes the slice-combined video data, and a video display unit that displays a decoding result image. 115 and a user interface (UI) 118 for inputting a request from the user. Here, the video decoding unit 116 has functions of a decoding unit and a restoration unit, and the user interface 118 has functions of a selection unit and a request input unit. The video display unit 115 has a function of a display unit.

  Note that the functions of the above units are realized by a processor executing a control program stored in a storage medium provided in each unit.

  The operation of the video transmission system according to the first embodiment having the above-described configuration will be described. First, the operation of the video preprocessing unit 108 will be described. FIG. 2 is a diagram showing a thinning process and a slice video creation process in the first embodiment.

  When viewed at the pixel level, the input video 201 is composed of “1, 2,..., Yx” pixels as shown in the input video 202. When performing the thinning process of the input video 201, the video pre-processing unit 108 thins the input video 201 at the pixel level and generates four slice videos 203. When viewed at the pixel level, the slice video 203 includes, for example, “1, 3, 5,...” Pixels as shown in the slice video 204. Although the present embodiment shows a case where four slice images are generated by thinning out every two pixels, the thinning method is not particularly limited, and can be set to any thinning method by the encoding control unit 106. It is.

  FIG. 3 is a diagram illustrating how macroblocks are handled in the video encoding unit 109 according to the first embodiment. The macro block is a unit for encoding video, and is composed of, for example, 16 × 16 pixels. Macroblock (MB) includes MPEG-2, MPEG-4, H.264, and so on. An international standard such as H.264 is used. As described above, the slice video 203 generated from the input video 201 by the video preprocessing unit 108 is composed of macroblocks MB (1), MB (2),..., MB ((k + 1) n). The video encoding unit 109 performs slice encoding using this macroblock as a basic unit. Details of slice encoding will be described later. Similarly, other slice videos are also composed of macroblocks.

  FIG. 4 is a diagram illustrating a slice combining operation in the combining unit 113 according to the first embodiment. The synthesizing unit 113 includes slice-encoded data 401 that has been slice-encoded by the video transmission apparatus 101a, slice-encoded data 402 that has been slice-encoded by the video transmission apparatus 101b, and slices that have been slice-encoded by the video transmission apparatus 101c The encoded data 403 and the slice encoded data 404 slice-coded by the video transmission apparatus 101d are slice-combined and output as slice-combined data 405.

  The slice encoded data 401 includes slice data 1-1, 1-2, 1-3, and 1-4. Similarly, the slice encoded data 402 is composed of slice data 2-1, 2-2, 2-3, 2-4. The slice encoded data 403 includes slice data 3-1, 3-2, 3-3, 3-4. The slice encoded data 404 is composed of slice data 4-1, 4-2, 4-3, and 4-4.

  The slice synthesis data 405 includes slice data 1-1 at the upper left of the slice encoded data 401 extracted from each slice encoded data, slice data 2-2 at the upper right of the slice encoded data 402, and slice encoding. The lower left slice data 3-3 of the completed data 403 and the lower right slice data 4-4 of the slice encoded data 404 are combined as slice data for one screen.

  For example, H. By using the H.264-compliant stream format, the head of each slice data can be easily found. Also, by constructing a macroblock as shown in FIG. 3, it is possible to synthesize the slice synthesis data 405 simply by combining the slice data extracted from the slice encoded data.

  FIG. 5 is a flowchart showing an operation processing procedure of the video preprocessing unit 108 in the first embodiment. The video pre-processing unit 108 waits until a video is input (step S1), and when the video is input, generates a slice image as shown in FIG. 2 (step S2). The generated slice image is output to the video encoding unit 109 (step S3). Thereafter, it is determined whether or not the final slice video has been output to the video encoding unit 109 (step S4). When the final slice video is output, the process returns to step S1 to wait for video input. On the other hand, if the final slice video has not been output, the process returns to step S3.

  FIG. 6 is a flowchart showing an operation processing procedure of the video encoding unit 109 in the first embodiment. The video encoding unit 109 waits until a slice video is input (step S11), and when the slice video is input, generates a macroblock (MB) from the slice video (step S12). Slice encoding is performed using the generated macroblock (step S13). Details of this slice encoding will be described later. Then, the slice-coded slice data is output to the transmission / reception unit 110 (step S14). Thereafter, the process returns to step S11 to wait for input of the next slice video.

  FIG. 7 is a diagram illustrating a slice encoding operation performed by the video encoding unit 109 in step S13 of the first embodiment. First, the video encoding unit 109 receives the MB data generated in step S12 and the MB encoding characteristics obtained from the encoding control unit 106 (T601), and starts processing the MB data (T602). For each macroblock, matching processing is performed within several frames before and after stored in the frame memory to detect a motion vector, motion compensation is performed based on the motion vector information, and a predicted image is generated (T603).

  In addition, intra prediction (intra prediction) is performed on the macroblock subjected to motion compensation prediction to generate a predicted image (T604). A prediction error is calculated for each prediction image generated at T603 and T604, and a prediction image with a smaller error is selected (T605). The selected predicted image is compared with the input slice video to generate a prediction difference signal (T606).

  After performing DCT transform which decomposes | disassembles into a two-dimensional frequency component with respect to a prediction difference signal, this DCT transform coefficient is used as the discrete representative value using the input MB encoding characteristic (for example, quantization step). And a quantization coefficient is output (T607).

  Here, a quantization step is shown as an example of MB encoding characteristics. When the quantization step is reduced, the generated code amount of the macroblock is increased. Conversely, if the quantization step is increased, the generated code amount of the macroblock is reduced. Entropy coding is performed using the quantized coefficient output in T607 (T608), and slice data is output.

  The encoding control unit 106 controls MB encoding characteristics so as to satisfy a target reference value (for example, the code amount per second) while receiving feedback of the code amount generated as a result of entropy encoding. In addition, inverse quantization and inverse DCT transform are performed using the quantization coefficient output in T607 to generate a prediction difference signal (T609).

  Then, the encoding control unit 106 adds the prediction image generated by the motion compensation prediction process or the intra prediction process selected in T605 and the prediction difference signal generated in T609 to generate a decoded image. (T610). Deblocking filter processing is performed on the macroblock boundary of the generated decoded image so that the block boundary is not noticeable (T611), and the decoded image subjected to the deblocking filter processing is stored in the frame memory (T612).

  FIG. 8 is a flowchart showing an operation processing procedure of the synthesizing unit 113 in the first embodiment. The synthesizing unit 113 waits until the slice encoded data is input (step S21). When the slice encoded data is input, the combining unit 113 holds the input slice data in the memory (step S22). And it is discriminate | determined whether it is the last slice data. Here, it is determined whether or not it is the fourth slice data.

  If it is not the last slice data, the process returns to step S21 to wait for input of the next slice data. On the other hand, if it is the last slice data, it is determined whether or not it is the multi-screen composition mode (step S24). Whether or not the multi-screen composition mode is selected is determined by a user request input to the user interface (UI) 108 of the video reception device 103. The determined mode information is notified from the video receiver 103 to the video synthesizer 102.

  If the multi-screen composition mode is selected in step S24, slice composition is performed as shown in FIG. 4 (step S25), and the synthesized slice composition data is output to the transmission / reception unit 112 (step S26). Thereafter, the process returns to step S21. On the other hand, if it is not the multi-screen composition mode in step S24, slice composition is not performed, and the slice data is output to the transmission / reception unit 112 as it is in step S26.

  FIG. 9 is a flowchart showing an operation processing procedure of the video decoding unit 116 in the first embodiment. The video decoding unit 116 waits until slice data is input (step S31). When the slice data is input, the video decoding unit 116 performs slice data decoding processing (step S32). The slice data decoding process is realized by executing a process opposite to the slice encoding process of FIG.

  And it is discriminate | determined whether it is the last slice data (step S33). If it is not the last slice data, the process returns to step S31 to wait for the next slice data. On the other hand, if it is the last slice data, it is determined whether or not it is the multi-screen composition mode (step S34).

  In the multi-screen composition mode, the decoded video data is output as it is to the video display unit 115 (step S35). Thereafter, the process returns to step S31. On the other hand, if it is not the multi-screen composition mode in step S34, the reverse process (complementary process) of the thinning process of the video preprocessing unit 108 shown in FIG. 2 is performed (step S36), and the decoded video data is converted in step S35. The image is output to the video display unit 115.

  As described above, according to the video encoding / synthesizing apparatus and the video transmission system of the first embodiment, the video pre-processing unit 108 performs the thinning process to create slice videos, and each slice video is configured by a macro block. The video encoding unit 109 performs slice encoding on the slice video, and the combining unit 112 combines the slice encoded slice data. Therefore, video data from a plurality of cameras can be easily encoded and combined. In addition, it is possible to easily select a multi-screen display in which reduced video is synthesized and a single-screen display of video having the original resolution, and according to a request from the user, Thus, it is possible to quickly switch between the one-screen display of the video having the same resolution as the input video from the camera.

(Second Embodiment)
FIG. 10 is a diagram showing a configuration of a video transmission system according to the second embodiment of the present invention. The same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted as appropriate. In the second embodiment, unlike the first embodiment, when the transmission / reception unit 110 receives encoding control information (for example, a target code amount) from the video synthesis apparatus 102, the transmission / reception unit 110 outputs the encoding control information to the encoding control unit 106. The synthesizing unit 113 holds a video transmission device management table 1001 and a slice synthesis data management table 1002 described later. Other configurations are the same as those in the first embodiment.

  The operation of the video transmission system of the second embodiment having the above configuration will be described. FIG. 11 is a flowchart illustrating an operation processing procedure of the synthesis unit 113 in the second embodiment.

  The synthesizing unit 113 waits until the slice encoded data is input (step S41). When the slice encoded data is input, the combining unit 113 holds the input slice data (step S42). Further, the code amount of the received slice data is monitored, and the video transmission device management table 1001 and the slice synthesis data management table 1002 are updated (step S43).

  FIG. 12 is a diagram showing a video transmission device management table 1001 and a slice synthesis data management table 1002 in the second embodiment. A video transmission device management table 1001 shown in FIG. 12A is a table for managing the upper limit value of the code amount generated by the video transmission device 101 and the code amount actually generated. The slice synthesis data management table 1002 shown in FIG. 12B indicates which video transmission device the slice synthesis data is composed of, and the upper limit value of the total code amount of the slice data, and It is a table for managing the amount of code actually generated.

  In step S43, the target code amount is calculated using the video transmission device management table 1001 and the slice synthesis data management table 1002, and is output to the transmission / reception unit 114. The transmission / reception unit 114 transmits the calculated target code amount to the video transmission apparatus 101 via the first network 104. The encoding control unit 106 in the video transmission apparatus 101 controls the MB encoding characteristics so as to be close to the received target code amount. Here, the calculation of the target code amount is performed, for example, according to the following procedures (a), (b), and (c).

(A) The slice synthesis data management table 1002 is searched, and slice synthesis data having the largest difference between the target code amount and the generated code amount is selected.
(B) With respect to the video transmission device constituting the selected slice synthesis data, the video transmission device management table 1001 is searched, the video transmission device MAX having the largest difference between the target code amount and the generated code amount, and the video having the smallest difference. Select the transmission device MIN.
(C) For example, a half value is obtained for the difference between the target code amount of the video transmission device MAX and the generated code amount, and is added to the target code amount of the video transmission device MIN.

  The new target code amount of the video transmission device MAX and the new target code amount of the video transmission device MIN obtained by such a procedure are transmitted to the video transmission device 101. Subsequent processes in steps S44 to S47 are the same as the processes in steps S23 to S26 in the first embodiment, and a description thereof will be omitted.

  As described above, according to the video encoding / synthesizing apparatus and the video transmission system of the second embodiment, the video transmission apparatus 101 is configured to hold the slice synthesis data management table 1002 and the video transmission apparatus management table 1001 in the synthesis unit 113. The code amount allocation can be changed between the two. Therefore, the image quality of each slice image, that is, the image quality between the sub-screens during multi-screen display can be made uniform.

(Third embodiment)
FIG. 13 is a diagram showing a configuration of a video transmission system according to the third embodiment of the present invention. The same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted as appropriate. In the third embodiment, unlike the first embodiment, an object detection unit 1101 is newly provided.

  The target detection unit 1101 receives the slice video from the video preprocessing unit 108, detects a specific target using the image recognition function, and outputs the detection result to the video encoding unit 109. The detection result of the target object is used to control the MB coding characteristics of the macroblock including the target object and realize high image quality of the target area.

  The operation of the video transmission system of the third embodiment having the above configuration will be described. FIG. 14 is a diagram showing the timing of the encoding process in the third embodiment. FIG. 14A shows processing timing by conventional frame coding. FIG. 14B shows processing timing by slice coding according to this embodiment.

  A processing timing 1201 by conventional frame encoding will be described. Numbers “1, 2, 3,...” Representing timing steps are shown in the uppermost part of FIG. The upper part shows the video input timing, the middle part shows the object detection timing, and the lower part shows the frame encoding timing. In this case, the object is detected once for two video data inputs. That is, in the conventional frame coding, the object detection process is performed at timing 2 on the video input at timing 1. In addition, using the detection result of the object, frame encoding is performed so as to improve the image quality of the area of the object at timing 3. In this way, it takes three steps from the input of video until the encoding is completed.

On the other hand, the processing timing 1202 by slice coding of this embodiment will be described. Similarly, the numbers “1, 2, 3,...” Representing the timing steps are shown at the top of FIG. The upper part shows the video input timing, the middle part shows the detection timing of the object, and the lower part shows the slice encoding timing. There is no change in the input timing of the video and the detection timing of the object, and only the timing of the slice encoding in the lower stage is changed. In the case of generating four slice videos, the slice encoding is started at the same timing as the object detection timing 2. In this case, after the object detection process a is completed, the detection result of the object can be used to improve the image quality of the object area only for the fourth slice video.

  As described above, the processing timing by the slice encoding according to the present embodiment requires two steps from the input of the video until the encoding is completed. Compared to the processing timing by the conventional frame encoding, the processing timing is one step. Delay time can be reduced.

  FIG. 15 is a flowchart showing an operation processing procedure of the object detection unit 1101 in the third embodiment. The object detection unit 110 waits until a slice video is input (step S51). When the slice video is input, an object detection process is performed (step S52). As this object detection processing method, for example, a method disclosed in Japanese Patent Application Laid-Open No. 2001-222719 is used. That is, in this target object detection processing method, an edge portion is extracted from the target image to generate an edge image, and a voting process is performed using a template at each pixel position of the edge image. The cluster is evaluated to determine the position and size of the face included in the target image. The object detection information detected in step S52 is output to the video encoding unit 109 (step S33). Thereafter, the process returns to step S51.

  FIG. 16 is a flowchart showing an operation processing procedure of the video encoding unit 109 in the third embodiment. The video encoding unit 109 waits until a slice video is input (step S61). When the slice video is input, it is determined whether or not the object detection information is obtained from the object detection unit 1101 (step S62). When the object detection information has not been obtained, the processes of steps S63 to S65 are performed in the same manner as the processes of steps S12 to S14 of the first embodiment.

  On the other hand, when the object detection information is obtained, a macro block is generated from the slice video (step 66), and the MB coding characteristics given from the coding control unit 106 are corrected based on the object detection information. Add (step S67). Specifically, the quantization step of the macro block included in the region where the target object is detected is changed to a smaller value. Thereafter, in the processes in steps S68 and S69, the same processes as in steps S64 and S65 are performed.

  As described above, according to the video encoding / synthesizing apparatus and the video transmission system of the third embodiment, by providing the object detection unit 1101, the delay time until the video is encoded can be reduced, and further, the target The area where an object is detected can be improved in image quality.

  When correcting the MB coding characteristic, a specific quantization step value (for example, value 40) is used, and information regarding the specific quantization step value is exchanged in advance between the video transmission unit 101 and the video reception unit 103. You may keep it. Thereby, it becomes possible to transmit the detection result of the object by the video stream without transmitting additional information. In addition, the video display unit 115 of the video receiving apparatus 103 can know the detection area of the object, and can perform highlighting such as adding a frame to the corresponding area.

  In the above embodiment, the object detection information is used at the time of slice encoding of the last slice video. However, the object detection information can be changed according to the object detection processing time.

  In the above embodiment, slice video is encoded in the order of slice video 1, slice video 2, slice video 3, and slice video 4. However, even if the order is changed for each video transmission device. Good. For example, in the case of the slice synthesis shown in FIG. 4, the slice video corresponding to the slice data to be synthesized is finally slice encoded. Thereby, only the slice image in which the image quality of the object region is improved using the result of the object detection can be used for the slice synthesis.

(Fourth embodiment)
The configurations of the video encoding / synthesizing apparatus and the video transmission system in the fourth embodiment are the same as those in the first embodiment. FIG. 17 is a flowchart showing an operation processing procedure of the video preprocessing unit 108 in the fourth embodiment of the present invention. Since the process of step S71-S73 of 4th Embodiment is the same as the process of step S1-S3 of the said 1st Embodiment, the description is abbreviate | omitted.

  After outputting the slice video to the video encoding unit 109 in step S73, the video preprocessing unit 108 determines whether or not it is one before the final slice video (step S74). If it is not one before the final slice video, the process returns to step S73. On the other hand, when it is one before the last slice video, the frame video input from the camera is output to the video encoding unit 109 instead of the slice video (step S75). Thereafter, the process returns to step S71.

  FIG. 18 is a flowchart illustrating an operation processing procedure of the video encoding unit 109 according to the fourth embodiment. The video encoding unit 109 is basically the same as the processing in FIG. 6 in the first embodiment and the processing in FIG. 16 in the third embodiment, but differs in the following processing. That is, steps S81 and S82 are the same as steps S61 and S62 in FIG. 16, steps S83 to S85 are the same as steps S12 to S14 in FIG. 6, and steps S87 to S90 are This is the same as the processing in steps S66 to S69 in FIG.

  When the object detection information has been obtained in step S82, the object detection area is cut out from the frame image received from the image preprocessing unit 108 using the object detection information (step S86). By cutting out the detection area of the object, the cut area from the frame video is slice-encoded instead of the last slice video.

  As described above, according to the video encoding / synthesizing device and the video transmission system of the fourth embodiment, the frame image is output in step S75, and the detection area of the object is cut out in step S86, so that the last slice is obtained. The clipped frame video is encoded instead of the video. As a result, only the detection area of the object can be slice-encoded with high image quality based on high-resolution video data.

  If the detection area of the object is large, slice coding can be performed after performing the reduction process. Further, although the frame image is used as the last slice image, the frame image is not limited to the last slice image, and can be changed according to the processing time of the object detection processing unit.

(Fifth embodiment)
The configurations of the video encoding / synthesizing apparatus and the video transmission system in the fifth embodiment are the same as those in the first embodiment. FIG. 19 is a flowchart showing an operation processing procedure of the video encoding unit 109 according to the fifth embodiment of the present invention. The process of the fifth embodiment is basically the same as the process of FIG. 16 in the third embodiment, but differs in the process of step S103. That is, the processes in steps S101 and S102 are the same as the processes in steps S61 and S62 in FIG. 16, and the processes in steps S104 to S107 are the same as the processes in steps S66 to S69 in FIG.

  If the object detection information is not obtained in step S102, the video encoding unit 109 extracts past object detection information held in the object detection unit 1101 (step S103). Using the extracted past object detection information, a region having a high object existence probability is encoded with high image quality until the processing of the object detection unit is completed.

  As described above, according to the video coding / synthesizing apparatus and the video transmission system of the fifth embodiment, the past object detection information is retained, so that a region with a high object existence probability is slice-encoded with high image quality. can do.

(Sixth embodiment)
The configurations of the video encoding / synthesizing apparatus and the video transmission system in the sixth embodiment are the same as those in the first embodiment. FIG. 20 is a flowchart showing an operation processing procedure of the video encoding unit 109 according to the sixth embodiment of the present invention. The process of the sixth embodiment is basically the same as the process of FIG. 16 in the third embodiment, but differs in the process described later. That is, the processing of steps S111 to S115 is the same as the processing of steps S61 to S65 in FIG. 16, and the processing of steps S117 to S120 is the same as the processing of steps S66 to S69 of FIG.

  When the object detection information is obtained in step S112, the video encoding unit 109 determines whether or not the object detection information is included (step S116). When the detection information of the target is not included, the process returns to step S111 and skips the slice encoding. On the other hand, when the detection information of the object is included, the process proceeds to step S117.

  As described above, according to the video encoding / synthesizing device and the video transmission system of the sixth embodiment, it is determined whether or not the target detection information includes the detection information of the target object, and the slice video of the target object does not exist. By skipping slice coding, the amount of slice data transmitted on the network can be reduced.

  It should be noted that the present invention is not limited to those shown in the above-described embodiments, and those skilled in the art can also make changes and applications based on the description in the specification and well-known techniques. Yes, included in the scope of protection.

  For example, by using a wired LAN, a wireless LAN, a public network, or the like for the network, it is possible to construct a video transmission system that can encode and transmit a remote video and synthesize and display a plurality of videos. In the above-described embodiment, the case where the number of slices is 4 has been shown. However, the present invention can be similarly applied to values “9, 16, 25,. It is.

  In the above embodiment, an example is shown in which slice data to be combined is obtained by simple thinning. For example, in the slice encoded data 401 of FIG. 4, slice data 1-2 is obtained by simple thinning out the slice data 1-2, 1-3, and 1-4. The present invention is not limited to this simple decimation, and can be similarly applied to an average value of peripheral pixel values. For example, slice data may be obtained from the value obtained by adding the pixel values of the pixels 1, 2, x + 1, and x + 2 and dividing the result by 4 in the input video 202 of FIG. When returning to one screen, the reverse operation may be performed.

  In the above embodiment, the video input from a plurality of cameras is targeted. However, the present invention is not limited to the camera, and the present invention can be similarly applied when inputting video recorded on a plurality of recording media. is there.

  The present invention has an effect that it is possible to easily display a video having the original resolution in addition to the reduced video, and even when there is a large variation in input video characteristics, the image quality is uniform between the divided videos. It has the effect of making it possible to reduce the delay time until the video is encoded, encodes the input video, and synthesizes multiple encoded video data It is useful for a video coding / synthesizing apparatus, a video coding / synthesizing method, a video transmission system, and the like.

The figure which shows the structure of the video transmission system which concerns on the 1st Embodiment of this invention. The figure which shows the thinning-out process and slice image | video production | generation process in 1st Embodiment The figure which shows the handling of the macroblock in the video encoding part of 1st Embodiment The figure which shows the slice synthetic | combination operation | movement in the synthetic | combination part of 1st Embodiment. The flowchart which shows the operation | movement process sequence of the image | video pre-processing part in 1st Embodiment. The flowchart which shows the operation | movement process procedure of the video coding part in 1st Embodiment. The figure which shows the slice encoding operation | movement performed by the video encoding part in step S13 of 1st Embodiment. The flowchart which shows the operation processing procedure of the synthetic | combination part in 1st Embodiment. The flowchart which shows the operation | movement process procedure of the video decoding part in 1st Embodiment. The figure which shows the structure of the video transmission system which concerns on the 2nd Embodiment of this invention. The flowchart which shows the operation processing procedure of the synthetic | combination part in 2nd Embodiment. The figure which shows the video transmission apparatus management table and slice synthetic | combination data management table in 2nd Embodiment. The figure which shows the structure of the video transmission system which concerns on the 3rd Embodiment of this invention. The figure which shows the timing of the encoding process in 3rd Embodiment. The flowchart which shows the operation | movement process sequence of the target object detection part in 3rd Embodiment. The flowchart which shows the operation | movement process sequence of the image | video encoding part in 3rd Embodiment. The flowchart which shows the operation | movement process sequence of the image | video pre-processing part in 4th Embodiment. The flowchart which shows the operation | movement process procedure of the video coding part in 4th Embodiment. The flowchart which shows the operation | movement processing procedure of the video encoding part in 5th Embodiment. The flowchart which shows the operation | movement processing procedure of the video encoding part in 6th Embodiment. The figure which shows the structural example of the conventional video coding and synthesizing | combining apparatus. The figure which shows the screen of the video display part of the user terminal in a prior art example The figure which shows the structure of the video input part of the user terminal in a prior art example. The figure which shows the synthetic | combination operation | movement of the video data in a prior art example

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Video transmission apparatus 102 Video composition apparatus 103 Video reception apparatus 104 1st network 105 2nd network 107 Video input part 108 Video pre-processing part 109 Video encoding part 110 Transmission / reception part 111 Compositing control part 112,114 Transmission / reception part 113 Synthesis | combination Unit 115 video display unit 116 video decoding unit 117 transmission / reception unit 118 user interface (UI)

Claims (8)

A pre-processing unit that generates a slice video from the input video;
An encoding unit for encoding the slice video;
A synthesizing unit that synthesizes the plurality of slice-encoded video data so as to be a multi-screen display;
A decoding unit that decodes the plurality of video data synthesized to be the multi-screen display or the slice-encoded video data;
A restoration unit that restores the decoded video data to the original video data so as to be displayed on a single screen;
A selection unit for selecting video data synthesized and decoded to be the multi-screen display or video data restored to be the single-screen display;
A display unit for displaying a video based on the selected video data on a screen;
A video encoding / synthesizing device comprising:
An object detection unit for detecting a specific object included in the video;
When the detection of the specific object is not completed by the object detection unit,
The encoding unit is a video encoding / synthesizing device that changes encoding characteristics based on past detection results for the specific object . The video encoding / synthesizing device according to claim 1 ,
When the specific object is not detected by the object detection unit,
The encoding unit is a video encoding / synthesizing device that omits encoding of the divided video. The video encoding / synthesizing device according to claim 2 ,
When the specific object is detected by the object detection unit,
The encoding unit is a video encoding / synthesizing apparatus that changes encoding characteristics. The video encoding / synthesizing device according to claim 3 ,
The encoding unit is a video encoding / synthesizing device that extracts and encodes a corresponding area of the detected specific object from the video. The video encoding / synthesizing device according to claim 4 ,
When synthesizing the slice-encoded video data to be a multi-screen display,
The encoding unit is a video encoding / synthesizing apparatus that performs encoding so that a code amount becomes a target code amount. A pre-processing step for generating a slice video from the input video;
An encoding step of performing slice encoding on the divided slice video;
A synthesis step of synthesizing the plurality of slice-coded video data so as to be a multi-screen display;
A decoding step of decoding a plurality of video data synthesized to be the multi-screen display or the slice-encoded video data;
A restoration step of restoring the decoded video data to the original video data so as to be displayed on a single screen;
A selection step of selecting the video data synthesized and decoded to be the multi-screen display or the video data restored to be the single-screen display;
A display step of displaying a video based on the selected video data on a screen;
A video encoding and synthesis method comprising :
An object detection step of detecting a specific object included in the video;
When the detection of the specific object is not completed by the object detection step,
The video encoding / synthesizing method in which the encoding step changes encoding characteristics based on a past detection result for the specific object . A video transmission system in which a video transmission device, a video synthesis device, and a video reception device are connected via a network, encodes input video, synthesizes and displays the plurality of encoded video data,
The video transmission device includes:
A pre-processing unit that generates a slice video from the input video;
An encoding unit for encoding the slice video;
A first transmission unit that transmits the slice-encoded video data to the network;
The video composition device
A first receiver for receiving the slice-encoded video data from the network;
A synthesizing unit that synthesizes a plurality of slice-coded video data input from different video transmission devices so as to be a multi-screen display;
A second transmission unit for transmitting the synthesized video data to the network;
The video receiver is
A second receiving unit for receiving a plurality of video data synthesized so as to be the multi-screen display from the network or the slice-encoded video data;
A decoding unit that decodes the plurality of video data synthesized to be the multi-screen display or the slice-encoded video data;
A restoration unit that restores the decoded video data to the original video data so as to be displayed on a single screen;
A selection unit for selecting video data synthesized and decoded to be the multi-screen display or video data restored to be the single-screen display;
A display unit for displaying a video based on the selected video data on a screen;
Equipped with a,
The video transmission device includes:
An object detection unit for detecting a specific object included in the video;
When the detection of the specific object is not completed by the object detection unit,
The video transmission system in which the encoding unit changes encoding characteristics based on past detection results for the specific object . A pre-processing unit that generates a slice video from the input video;
An encoding unit for encoding the slice video;
A synthesizing unit that synthesizes the plurality of slice-encoded video data so as to be a multi-screen display;
A decoding unit that decodes the plurality of video data combined to be the multi-screen display or the slice-encoded video data that is not combined;
A video encoding / synthesizing device comprising: a restoration unit that restores the video data that is not synthesized to the original video data so as to be displayed on a single screen;
An object detection unit for detecting a specific object included in the video;
The encoding unit includes:
When the specific object is detected by the object detection unit, the encoding characteristic is changed,
A video coding / synthesizing device that changes coding characteristics based on past detection results for the specific object when detection of the specific object is not completed.

Images