WO2007099637A1

WO2007099637A1 - Image transmission device, image reception device, and image transmission/reception system

Info

Publication number: WO2007099637A1
Application number: PCT/JP2006/303993
Authority: WO
Inventors: Koji Saito; Kazuo Kaneko; Kazuyuki Tanaka
Original assignee: Fujitsu Limited
Priority date: 2006-03-02
Filing date: 2006-03-02
Publication date: 2007-09-07
Also published as: JPWO2007099637A1

Abstract

It is possible to display image data transmitted from a transmission side of the image data, at a reception side having a lower resolution than the transmission side. An image transmission device includes an image data transmission unit for transmitting data specifying an important region as an indispensable display region when displaying the image at the reception side. An image reception device includes an image size judging unit for judging whether the size of the reception data exceeds a maximum size which can be displayed at the local device and an image decoding unit for decoding the image data in the important region specified in the reception data for display if the maximum size is exceeded and not displaying the entire image of the received image data.

Description

Specification

Image transmission device, image reception device, and image transmission / reception system

Technical field

[0001] The present invention relates to a transmission / reception method of image data, for example, moving image data, and more specifically, an image data transmission / reception method when image processing performance on a transmission side and a reception side, mainly an image display area and resolution are different. About.

Background art

[0002] The full-fledged image communication era such as broadband communication, multimedia, mopile communication, and fusion of communication and broadcasting has begun. In such image communication, there is a difference in communication performance between a communication device having a relatively low image processing performance, such as a portable terminal, and a communication device having a relatively high image processing performance, such as a stream data distribution server. Communication is performed according to

FIG. 1 is an explanatory diagram of a conventional example of an image communication method between communication devices having such different image processing performance. In the figure, PDA (personal 'digital' assistance) with VGA (video 'graphics' array, 640 x 480 dots) resolution, QVGA (quarter VGA, 320 x 240 dots) mobile phone with camera, The ability to broadcast images to a PC with a resolution of SXGA (super 'tendered' graphics' array, 1280 x 1024 dots) and a VGA resolution PDA. Since there is a mobile terminal with a camera with a lower resolution, a reception error (unreceivable) will occur on the mobile terminal with camera unless specially crafted by the distributor. A high-resolution personal computer or the like treats the sent image as small image data and does not cause communication problems.In this case, the distribution source matches the performance of the camera-equipped mobile device on the receiving side. In other words, there was a problem in that it was impossible to transmit or receive images, and to give up data distribution to mobile terminals equipped with performance.

[0004] FIG. 2 is a conventional example in the case where the transmission side distributes an image in accordance with a distribution destination having the lowest image processing performance. In this figure, the image data distribution source PDA sends the moving image at the QVGA resolution in accordance with the resolution QVGA of the mobile terminal with the camera. Broadcast distribution of image data becomes possible.

[0005] However, in such a case, it is necessary to know the delivery destination with the lowest performance at the delivery source and the image processing performance thereof, and it is necessary to perform image processing according to the delivery destination. In addition, there is a problem that images with too low resolution are too small and difficult to see at distribution destinations with high image processing performance, or that they can be blocked when simply enlarged.

[0006] In Patent Document 1 as a prior art relating to such image data transmission / reception, in a videophone that displays the other party's video and one's own video on two screens, the own video and the other's video are displayed. If overlapping makes it difficult to see important parts of the other video, technology is disclosed to solve the problem.

[0007] Further, in Patent Document 2, for example, in a surveillance system for an automobile, even if a surveillance image is transmitted through a narrow bandwidth transmission path, an image area including a person considered to be important by the user is included. A technique for transmitting with higher image quality by encoding and transmitting at a low compression rate is disclosed.

[0008] Next, in Patent Document 3, an image data is transmitted on the receiving side in an image data communication system by transmitting image data so that an image can be displayed so that resolution gradually increases from an image of an important part. A technique capable of displaying a high-resolution image for an important part before the entire display is completed is disclosed.

[0009] Further, in Patent Document 4, in a system that transmits an image by dividing the image into slices, encoding that can increase resistance to a transmission path error by overlapping and encoding and decoding a part of the image, And a decoding method is disclosed.

[0010] However, even when such conventional technology is used, image data is transmitted at a high resolution when the image data transmission side and the reception side have different image processing performance, particularly when the resolution is different. On the other hand, the receiving side with low resolution cannot receive images, and conversely, when broadcasting image data to the receiving side with low resolution, the received image is received on the receiving side with high resolution. However, it was difficult to solve the problems such as being difficult to see because of small size, or blocking when simple enlargement.

Patent Document 1: Japanese Patent Laid-Open No. 10-200873 “Videophone Device” Patent Document 2: JP-A-11 215482 “Monitoring System”

Patent Document 3: Japanese Patent Application Laid-Open No. 2003-333593 “Image Data Transmitting Device and Image Data Receiving Device”

Patent Document 4: Japanese Patent Application Laid-Open No. 2004-236337 “Duplicate Video Coding and Decoding Method and Apparatus”

Disclosure of the invention

[0011] In view of the above-described problems, an object of the present invention is to receive images with low resolution even when the image data transmission side transmits image data without causing the image processing performance of the reception side, particularly resolution, to be a problem. The image data transmitted on the side can be displayed.

[0012] The image transmission apparatus of the present invention includes an image encoding unit that encodes an image, and encoded image data as an essential display area when an image is displayed on the reception side of the image data. An image data transmitting means for sending data designating the important area to the image data receiving side.

[0013] Further, the image receiving device of the present invention has an image size determining means for determining whether or not the size of the received image data exceeds a maximum display size that can be displayed by the device, and when the size exceeds the maximum display size, Image decoding means for decoding and displaying the image data of the important area specified in the received data and not displaying all one screen of the received image data.

[0014] Further, the image transmission / reception system of the present invention includes such an image transmission device and one or more such image reception devices.

Brief Description of Drawings

FIG. 1 is an explanatory diagram of a conventional example of an image distribution method in accordance with image processing performance on a transmission side.

FIG. 2 is an explanatory diagram of a conventional example of an image distribution method in accordance with image processing performance on the receiving side.

FIG. 3 is a block diagram showing the principle configuration of the image transmission / reception system of the present invention.

FIG. 4 is a basic configuration block diagram of an image transmission device or an image reception device in the present embodiment.

FIG. 5 is an explanatory diagram of an error correction method using redundant slices.

FIG. 6 is a diagram showing the structure of an access unit in H.264. FIG. 7 is an explanatory diagram of a method for performing strong coding for error correction on important image areas.

[Fig. 8] An illustration of flexible macroblock ordering.

FIG. 9 is an explanatory diagram of a method for specifying an important region in an image according to the present embodiment.

FIG. 10 is an explanatory diagram of storing important area designation data in an access unit in the present embodiment.

FIG. 11 is an explanatory diagram of encoding and transmission processing centering on software processing.

FIG. 12 is an explanatory diagram of a decoding process centering on software processing.

FIG. 13 is an explanatory diagram of a decoding process centering on hardware processing.

FIG. 14 is a configuration block diagram of a video codec on the transmission side.

FIG. 15 is a configuration block diagram of a video codec on the receiving side.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 3 is a block diagram showing the principle configuration of the image transmission / reception system of the present invention. In the figure, the image transmission / reception system includes an image transmission device 1 and one or more, generally a plurality of image reception devices 2.

[0017] The image transmission device 1 includes an image encoding unit 3 that encodes an image, and an important area as an essential display area when the image is displayed on the receiving side of the image data together with the encoded image data. And an image data transmission unit 4 for sending data designating to the receiving side.

[0018] The image reception device 2 has an image size determination unit 5 that determines whether or not the size of the received image data exceeds the maximum display size that can be displayed by the device itself, and when the image display device 2 exceeds the maximum display size, An image data decoding unit 6 is provided that decodes and displays the image data of the important area specified in the received data and does not display the entire screen of the received image data. In general, the plurality of image receiving apparatuses 2 have different resolutions, that is, maximum display sizes.

In the present invention, the above-mentioned important area is an area corresponding to a redundant slice transmitted for transmission error correction at the time of image slice division transmission. It can also be an area corresponding to an important slice in which.

[0020] As described above, according to the present invention, an essential display area when an image is displayed from the image transmission side. Even when the data specifying the important area as the area is sent together with the image data, and the size of the image data received on the receiving side exceeds the maximum display size that can be displayed on the transmitting side, the image data in that important area Control is performed so that the entire screen of the received image data is not displayed while being decoded and displayed.

FIG. 4 is a basic configuration block diagram of an image transmission device and an image reception device that constitute the image communication system of the present invention. In the figure, the image transmitting device or the image receiving device is composed mainly of a video codec 10 that encodes and decodes image data, and the CPU 11 and the memory 12 are connected to the video codec 10, and the CPU 11 There is a cache 13 connected to it.

As will be described later, the image transmission apparatus having the configuration of FIG. 2 encodes image data with the video codec 10, and user data indicating an important area in the image together with encoded data. Send to the image data receiver. On the image receiving device side, corresponding to the data indicating the important area in the image, the image data of only the important area or the entire image data is decoded as necessary, and the decoded image data is displayed. To do. In this embodiment, the analysis of the data indicating the important area and the image display processing are basically processed by software by the CPU 11, and the parts that require processing speed, such as encoding and decoding, are processed. Suppose that video codec10 performs hardware processing.

[0023] Before describing the embodiment of the present invention in further detail, a technique relating to error countermeasures in bitstream transmission applied in the present embodiment will be described using the H.264 moving image compression coding standard system as an example.

[0024] The H.264 video compression and coding standard is based on ITU-T (International Nanole 'Telecommunication' Union 1 Telecommunications' Standardization 'Sector 1') and MPEG ( This is the latest image compression technology jointly standardized by the moving (picture “experts” group), and the embodiment of the present invention will be described using the bitstream transmission error countermeasure in this standardization technology.

[0025] In H.264, a slice as a small screen obtained by subdividing a picture corresponding to one screen is a basic unit of the code. Each slice is composed of a plurality of macroblocks. General Although the image data of each macroblock is encoded / decoded only once, there is a method of encoding / decoding the same macroblock multiple times in order to improve error correction capability when the transmission path quality is poor. is there. Basically, the same macroblock is not allowed to be included in one slice more than once, so create a redundant slice and perform error correction using the data of the redundant slice according to the transmission error status. Can do.

FIG. 5 is an explanatory diagram of an error correction method using such redundant slices. In the figure, redundant slice data is transmitted in addition to normal slice data, and even if an error occurs in normal transmission slice data, the macro block corresponding to the error location is included in the redundant slice. If there is no error with respect to the redundant slice, the image without error can be reproduced by decoding the image data using the macroblock data in the redundant slice. If the data of the macro block corresponding to the error location does not exist in the redundant slice, error correction cannot be performed.

[0027] Note that Η · 264 includes multiple slices. There are many types of groups, such as alternating slices of macroblocks, rectangular slices, clockwise 'counterclockwise, reverse raster order, etc. Slice 'group has a force S, the details of which are not directly related to the present invention, so the explanation is omitted.

[0028] In H.264, NAL (network substructure) is used between a VCL (video 'coding layer') that handles the video encoding process itself and a lower system that transmits and stores the encoded information. A layer called 'layer' is defined, and bitstream mapping (association) with lower systems is performed in units of NAL units as one NAL delimiter. Furthermore, in order to access the information in the bitstream on a picture-by-picture basis, several NAL units are grouped together and are called access units.

FIG. 6 shows the structure of this access unit. In the figure, the AU (access unit) delimiter at the head of the access unit is a start code indicating the head of the access unit, the next SPS (sequence 'parameter' set) is information (header) about the sequence, and P PS ( The picture 'parameter' set is information (header) of the coding mode of the entire picture, and the SEI (supplemental 'enhancement' information) is additional information. After that, the main picture corresponding to the entire image data, and then the redundant slice Redundant pictures are stored, and EOS (end-of-sequence) indicating the end of the sequence and EOS (end-of-stream) indicating the end of the stream are stored at the end. The redundant slice in the redundant picture stored after the main picture corresponding to normal image data is used for channel error correction as described in FIG.

[0030] Instead of using redundant slices in H.264, an important area of an image is used as an important slice that has been subjected to a coding scheme that is strong for error correction using a coding scheme different from the normal slice corresponding to other areas. There is a transmission method to the receiving side. Fig. 7 is an illustration of an image error correction method using such an important slice. The important region in the image is transmitted to the receiving side with a sign that is stronger in error correction than the normal slices for the other normal regions, and is transmitted to the receiving side even if an error occurs in the important slice. Can be encoded.

FIG. 8 is an explanatory diagram of the order of slices (macroblocks) when different coding schemes are used, distinguishing between important slices and normal slices as described above. Such an order is called FMO (flexible 'macroblock' ordering), and normal slices with normal coding are stored after important slices with strong error tolerance coding.

Next, in the present embodiment, image transmission applying the above transmission error countermeasure will be described. FIG. 9 is an explanatory diagram of a method for specifying an important area in the present embodiment. In the present embodiment, image data is utilized so that image display performance does not occur between an image transmission apparatus and an image reception apparatus having different image processing performances, mainly display image sizes, by utilizing the above-described techniques such as redundant slice and FMO. Shall be sent and received.

In FIG. 9, in the present embodiment, the important area in the image is defined by the center point A and the size B of the important area, that is, the number of pixels in the horizontal direction and the number of pixels in the vertical direction. The designation of such an important area is defined by ON / OFF indicating whether or not such designation is used on the image data transmission side, the position of the center point A of the important area, and the size B of the important area. The image data is sent to the receiving side, and the receiving side analyzes the data and selects, for example, whether to display all the image data or only the important area. The

[0034] When such an important area is designated, that is, when it is ON, it is assumed that the position of the center point A as the default area and the size B of the area are set. This data can be customized by the user. As a method of customization by the user, the basic force is to set the position of the center point A on the image on the screen, and the size B of the important area to be selected from general image sizes. The user can also specify a desired size. However, when the user designates it, it is necessary to devise it so that it is designated on the screen in order to prevent problems such as oversize.

[0035] In addition to the VGA, QVGA, and SXGA mentioned above, CIF (Common 'Interface' format, 352 X 288 dots), QCIF (Quota CIF, 176 X 144 dots), etc. are specified as general image sizes Shall.

[0036] In this way, the data such as the position A of the center point of the important area and the size B thereof as information related to the important area set on the image data transmission side, and the power to use the redundant slice as described above. Data such as whether to use the FMO method described in FIG. 8 is stored in the additional information SEI area in the access unit described in FIG. 6 and is sent to the image data receiving side. This SEI is supplementary information that is not directly related to the image decoding process, and since there is a user data area in this SEI, as shown in FIG. These data are stored and sent to the receiving side.

[0037] Transmission and reception of image data using the redundant slice and FMO method described above will be further described with reference to Figs. FIG. 11 is an explanatory diagram of the image data encoding / transmission process that is mainly performed using software on the transmission side.

In FIG. 11, when transmitting image data, it is first determined in step S1 whether the designation of the important area is on or off. If off, that is, if the designation of the important area is not performed, step S1 is performed. In S2, the normal code processing as hardware processing is performed, and in step S3, the image data is transmitted, and the processing ends.

[0039] If the designation of the important area is on, that is, if designation is made, the important area information is obtained in step S4. That is, the center point, the size, etc. are extracted. For this important area information, the default value can be used as described above, or the data specified by the user can be used. Then, depending on whether the redundant slice explained in Fig. 5 and Fig. 6 or the FM ○ method explained in Fig. 7 and Fig. 8 is used, the processing of steps S5 and S6 is performed, or the processing of steps S7 and S8 It is determined whether processing is performed.

[0040] When a redundant slice is used, coding parameter information is specified in step S5. That is, after the size and position of the important area (redundant slice) are specified, and the important area information is specified as user data in SEI, the code processing for specifying the redundant slice as hardware processing is performed in step S6. In step S3, the image data is transmitted and the process ends.

[0041] When using FM ○ method, the position and size of the important area (important slice) are specified in the same way in step S7, and after the important area information is specified as user data inside SEI, the step In S8, the slice division as hardware processing, that is, the coding process in which the division between the important slice and the normal slice is designated is performed. In step S3, the image data is transmitted and the processing is terminated.

FIG. 12 is an explanatory diagram of processing on the receiving side centering on software processing. When image data is received in the figure, the header decoding process of the access unit is performed as the header decoding process in step S11, and the received image size is larger than the image display size in the own device in step S12. It is determined whether or not. If it is not larger than the image display size, that is, if the entire received image can be displayed, normal decoding processing of a picture (slice) as hardware processing is performed in step S13, and the decoded image is converted in step S14. Displayed and the process is terminated. In this case, an image of the size of the received image itself is displayed.

[0043] If the received image size is greater than or equal to the image display size of the own device in step S12, it is determined in step S15 whether or not to display an image. If not, step S16 is performed. As a result, the decoding process is stopped as reception is impossible, and the process ends.

[0044] When image display is to be performed, the user data in the SEI in the access unit sent from the transmission side in step S17 is analyzed, and the position and size of the important area are extracted. In step S18, the decoding process of the slice inside the important area is performed as hardware processing. In step S19, the decoded image is displayed, that is, only the important area is displayed, and the process is terminated. Instead of displaying only the important area, it is also possible not to display the entire received image data screen.

FIG. 13 is an explanatory diagram of processing on the reception side mainly using hardware processing. When the image data is received in the figure, after the decoding key parameter setting process for giving the display size of the received image is performed as software processing in step S21, the access unit section is processed as hardware processing in step S22. In step S23, it is determined whether or not the size of the received image is equal to or larger than the image display size of the own device. If not, the normal picture (slice) is determined in step S24. Decoding processing is performed, and in step S25, the decoded image is displayed as software processing, and the processing ends.

[0046] If it is determined in step S23 that the size of the received image is equal to or larger than the image display size of the own device, the user data inside the SEI is analyzed in step S26, and the size of the important area is extracted. In step S27, the important area is decoded, and in step S25, the decoded image is displayed as the software process, and the process ends. Note that the decoding process for the important region includes decoding processes corresponding to the method described in FIGS. 5 and 6 using redundant slices and the FMO method using important slices described in FIGS. Done.

FIG. 14 is an explanatory diagram of the hardware configuration of the video codec corresponding to the processing on the transmission side described in FIG. In the figure, the video codec 10 is composed of a header encoding unit 15 that encodes the header of input image data and an image data encoding unit 16 that encodes the image data itself.

FIG. 15 is a configuration block diagram of a video codec corresponding to FIG. In the figure, when received image data is input, header decoding of the access unit is performed by the header decoding unit 17, and user data in the SEI of the access unit is analyzed by the SEI user information analysis unit 18. The position of the center point of the important area and its size are extracted, and the image data decoding unit 19 decodes the image data of only the important area, for example Is done. Note that the video codec corresponding to FIG. 12 includes only the header decoding unit 17 and the image data decoding unit 19 and does not include the SEI user information analysis unit. The SEI user information analysis unit 18 is basically configured using a logic circuit or the like.

As described above in detail, according to the present invention, even when the resolution on the image data receiving side is low and the maximum image display size is small, it is possible to display an image of the important region designated from the transmitting side. It becomes possible. In addition, when the image processing performance on the receiving side, that is, the resolution is the same as that on the transmitting side, all image data sent from the transmitting side is displayed, so that it is possible to prevent deterioration of the received display image.

Claims

The scope of the claims

[1] image encoding means for encoding an image;

Image data transmission means for sending, together with the encoded image data, data specifying an important area as an essential display area when an image is displayed on the image data receiving side to the image data receiving side. An image transmitting apparatus comprising:

2. The image transmitting apparatus according to claim 1, wherein the important area is an area corresponding to a redundant slice transmitted for transmission error correction at the time of image divided transmission.

[3] The image according to claim 1, wherein the important area is an area corresponding to an important slice that is subjected to strong encoding in a transmission error tolerance when the image is divided and transmitted in slices. Transmitter device.

[4] means for receiving image data;

When the data received by the receiving means is image data to which data specifying an important area is added, the image data in the important area is decoded and displayed, and all one screen of the received image data is displayed. Image decoding means not to allow

An image receiving apparatus comprising:

[5] Image size determination means for determining whether or not the size of the received image data exceeds a maximum display size that can be displayed by the own device;

Image decoding means for decoding and displaying the image data of the important area specified in the received data when the maximum display size is exceeded, and not displaying all one screen of the received image data. A characteristic image receiving apparatus.

6. The image receiving apparatus according to claim 4 or 5, wherein the important area is an area corresponding to a redundant slice transmitted for transmission error correction at the time of image slice transmission.

7. The important region is a region corresponding to an important slice that is subjected to strong encoding in a transmission error tolerance when the image is divided and transmitted in slice division. Image receiving device.

[8] Image encoding means for encoding an image, the encoded image data, and the image An image transmission device comprising: an image data transmission means for transmitting data for designating an important area as an essential display area when displaying an image on the data reception side to the image data reception side;

When the data specifying the weight area is added to the received image data, only the image data of the important area specified in the received data is decoded and displayed, and the entire screen of the received data is displayed. An image transmission / reception system comprising: an image receiving device including image decoding means that is not displayed.

9. The image transmission / reception system according to claim 8, wherein the important area is an area corresponding to a redundant slice transmitted for transmission error correction at the time of image slice transmission.

10. The important region according to claim 8, wherein the important region is a region corresponding to an important slice to be encoded during slice division transmission of an image, to have a strong transmission error resistance, and to be encoded. Image transmission / reception system.